Skip to content

batch large file-identity lookups before SQLite bind limit #1237

Description

@TheGreenCedar

Parent: #1202
Root: #1179
Release: v0.16.0

Failure

On dev/codestory-next at ea3cd4d5dd4c1a7841aaa5bea3f06ab147530f4a, a clean full index of a 94,523-file Linux corpus fails before publication with SQLite's too many SQL variables error.

The failing path is Indexer::get_existing_file_ids through Storage::get_node_kinds_for_files. The storage method builds one unbounded query with two placeholder lists:

SELECT id, kind
FROM node
WHERE id IN (...file_ids...)
   OR file_node_id IN (...file_ids...)

At this corpus size it supplies roughly 189,046 parameters. The same full-index command succeeds against a separate clean current-source control, so the defect is scale-dependent rather than an environment or cache failure.

Expected

Full indexing should handle a repository-sized candidate set without exceeding SQLite's bind-variable limit. Batch this lookup or use a temporary relation while preserving its existing file-identity semantics.

Acceptance criteria

  • Both file-identity lookup paths bound SQLite parameters below the runtime limit.
  • Result semantics remain identical across batches, including duplicate IDs and missing rows.
  • Cancellation and storage errors remain attributable and do not publish a partial core.
  • A regression exercises a candidate set above SQLite's default bind limit.
  • The approved 94,523-file corpus publishes successfully from a clean cache.

Release decision

This correction is required for v0.16. #1237, #1202, and #1196 remain in the milestone; schedule pressure or an inconclusive backend experiment does not defer them. This issue restores the approved large-corpus publication input and does not select a vector backend.

Non-goals

  • Selecting a vector backend.
  • Changing retrieval routing or file-identity semantics.
  • Treating a small-repository control as large-corpus proof.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingepic:performance-opsPerformance, scale, sidecar SLA, and artifact operationsphase:promotion-readinessPromotion readiness and post-contract proof wavepriority:p1Release blocker or high-risk reliability/security issuerisk:publishable-blockerBlocks publishable benchmark or product claim readinessrustPull requests that update rust codesaga:codestory-intelligenceCodeStory product-grade intelligence sagatype:contractRuntime, CLI, API, or evidence contract work

    Projects

    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions