Skip to content

Speed up initial full indexing - #5079

Merged
Widthdom merged 12 commits into
mainfrom
codex/initial-full-index-performance-20260813
Aug 13, 2026
Merged

Speed up initial full indexing#5079
Widthdom merged 12 commits into
mainfrom
codex/initial-full-index-performance-20260813

Conversation

@Widthdom

Copy link
Copy Markdown
Owner

Summary

  • bound SQLite named-parameter statement sizes for caller-owned initial-index transactions across chunks, symbols, issues, reference lines, and references
  • reduce reference-graph work with compact lower-rank guards, reusable target-key and property facts, fresh inline source attribution, and one-time C# type-family matching
  • remove duplicate C# prepass/reference-line materialization, make manual performance tests explicitly runnable, and keep persistence telemetry bounded without weakening cancellation checkpoints

Root cause and impact

The empty-database path combined very large named-parameter statements with repeated graph candidate probes and duplicate temporary materialization. Microsoft.Data.Sqlite resolves parameter names during binding, so the previous 994-parameter reference statements were particularly expensive. The graph phase then repeated persistent candidate lookups and target-key construction over roughly one million candidate rows.

Measured on this repository with a new empty database:

Metric origin/main baseline This branch Change
Full index 143.814 s 60.842 s 57.7% faster
Reference graph 68.912 s 8.595 s 87.5% faster
Peak working set 960.9 MB 804.4 MB 16.3% lower

The branch measurement indexed 1,411 files and 491,013 references, versus 1,399 files and 489,997 references in the baseline. Three preceding comparable branch runs had a 64.534 s median (64.351–86.887 s), with the slower first run reflecting cold-cache variance.

Validation

  • Debug and Release solution builds: 0 warnings, 0 errors
  • Release net8.0: 11,420 passed, 7 skipped, 0 failed
  • Release net9.0: 10,948 passed, 418 skipped, 0 failed
  • dotnet format CodeIndex.sln --verify-no-changes --no-restore
  • dotnet run --project tools/CodeIndex.Changelog --no-restore -- check (14 fragments validated)
  • fresh empty-database full-index benchmark on final HEAD
  • root index and both workspace members healthy; graph/index/fold readiness current
  • independent adversarial review: No blocking/actionable issues found.

Documentation and changelog

Updated the English/Japanese developer and testing guidance. Added these bilingual unreleased fragments:

  • changelog.d/unreleased/+initial-index-write-batches.changed.md
  • changelog.d/unreleased/+manual-performance-opt-in.changed.md
  • changelog.d/unreleased/+compact-lower-rank-candidates.changed.md
  • changelog.d/unreleased/+reference-resolution-symbol-facts.changed.md
  • changelog.d/unreleased/+csharp-prepass-segmented-workspace.changed.md
  • changelog.d/unreleased/+csharp-workspace-type-lookups.changed.md
  • changelog.d/unreleased/+reference-line-window-sizing.changed.md
  • changelog.d/unreleased/+csharp-property-receiver-facts.changed.md
  • changelog.d/unreleased/+fresh-reference-source-attribution.changed.md
  • changelog.d/unreleased/+csharp-type-family-candidates.changed.md

Follow-ups

Raw ordinal SQLite binding and candidate-schema compression remain possible higher-risk follow-ups; neither is required for this improvement set.

@Widthdom
Widthdom marked this pull request as ready for review August 13, 2026 17:53
@Widthdom
Widthdom merged commit 11300d3 into main Aug 13, 2026
11 checks passed
@Widthdom
Widthdom deleted the codex/initial-full-index-performance-20260813 branch August 13, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant