perf: bound build memory and harden bulk graph updates

[tae2089]

Summary

• stream deferred build edges through a disk-backed spool to reduce peak memory usage
• resolve all staged implements edges before staged call and reference edges, regardless of file or spool record order
• detect additions in a wholly new package or source directory and use the safe full-build path when transactional replacement is available
• keep additions to existing packages incremental, and preserve scoped behavior for partial or custom update paths
• cache parsed files by source hash and parser/query version to avoid unchanged Tree-sitter work
• traverse TraceFlow frontiers with batched graph reads instead of one query per visited node
• persist unresolved edge candidates with an explicit index version and replay only candidates affected by newly available symbols
• store hashes for unresolved lookup keys and fingerprints so large Kotlin candidates remain indexable on PostgreSQL
• batch annotation upserts, tag replacement, incremental file deletion, and node persistence
• delete full-graph search documents directly by namespace and avoid materializing large node-ID lists
• add a (namespace, file_path) resolver lookup index based on measured PostgreSQL query plans
• remove two high-volume allocation sources in import suffix matching and JVM member extraction

Why

Large repositories previously retained too much deferred edge state during graph builds. Streaming that state lowers peak memory, but staged replay also needs a global dependency order: call resolution can depend on implementations discovered in later files.

Incremental updates have a related correctness boundary. Adding a new package can introduce implementations that restore or change edges from unchanged callers outside the changed-file set. The unresolved-candidate index records that dependency explicitly, allowing updates to revisit affected callers while retaining a safe full-build boundary when required.

PostgreSQL profiling exposed additional persistence bottlenecks after parsing was improved. Annotation writes performed per-annotation lookup, transaction, upsert, and tag replacement; file updates repeated the same deletion sequence per file; full graph deletion selected large node-ID sets; and resolver file lookups lacked an index matching their namespace-scoped predicate. The added bulk paths and indexes target those measured operations without adding connection-pool tuning or write goroutines.

Correctness behavior

• staged edges are replayed in two global passes: implementations first, then all remaining edges
• a new package or source-directory addition selects full replacement before transactional graph writes
• existing-package additions continue through incremental reconciliation
• unresolved candidates are versioned and rebuilt when their index contract changes
• unresolved values retain their complete text while SHA-256 columns provide bounded PostgreSQL index keys
• newly available symbols replay only relevant unresolved candidates instead of globally re-resolving every unchanged file
• parse-cache entries are invalidated by source or parser/query version changes
• TraceFlow preserves depth, direction, cycle, and limit behavior while batching frontier reads
• annotation batches verify all node ownership before mutating any annotation or tag
• full graph deletion is namespace-scoped; file deletion remains node-scoped so unchanged files are preserved
• replace=false with include paths does not escape its requested scope
• non-transactional and custom incremental syncers retain the conservative update path
• update statistics continue reporting the detected added, modified, and deleted file counts

Performance

Build memory profile

Measured on the earlier large Kotlin corpus used to validate spool-backed edge streaming:

Metric	Before	After	Change
Peak RSS	1686.8 MiB	502.5 MiB	-70.2%
Pre-GC heap	1328 MiB	217 MiB	-83.7%
Wall time	112.42 s	118.24 s	+5.2%
Edge resolution time	baseline	baseline +24%	trade-off

Current full-corpus CPU and allocation profile

Measured on a fresh SQLite database with 22,719 processed files, 114,582 nodes, and 751,148 parsed edges. Both sides used the same cold-parse-cache condition.

Metric	Before allocation fixes	After	Change
Cumulative Go allocation	36.51 GiB	28.13 GiB	-23.0%
Observed GC cycles	1,609	1,304	-19.0%
Clean wall time	205.07 s	193.12 s	-5.8%
Edge resolution	76.62 s	65.59 s	-14.4%
Resolver core	31.96 s	24.68 s	-22.8%
Maximum process RSS	419.5 MiB	471.1 MiB	+12.3%

CommonSuffixDepth fell from two allocations per call to zero, and collectJVMMembersFromText cumulative allocation fell from 2.62 GiB to 82.93 MiB. The single-run process RSS did not improve; it includes Tree-sitter, SQLite, mappings, runtime, and process-level high-water variance, so this PR does not claim an RSS reduction from the allocation fixes.

PostgreSQL persistence and resolver profile

The full greeting corpus completed with 22,719 files, 114,582 nodes, and 751,148 parsed edges after the bulk annotation path was added. A prior same-corpus run without annotation batching was stopped after 616.58 s without completing; the bulk candidate completed in 257.54 s wall time.

Operation	Before	After	Change
500 annotation writes	2,000 SQL / 1.120 s	9 SQL / 50.9 ms	-99.55% SQL, ~22x faster
Delete 5 changed files	60 SQL	11 SQL	-81.7% SQL
Resolver lookup for 250 file paths	295.4 ms / 14,591 buffers	8.9 ms / 1,502 buffers	~33x faster
DeleteGraph for 66,000 PostgreSQL nodes	bind-limit failure risk	2.57 s, zero remaining	bounded subqueries

The qualified-name lookup already completed the representative 500-name query in 11.3 ms with the existing index, so no additional index was added for that path. A populated 93,752-node greeting namespace rebuilt in 1m44s, and subsequent empty-corpus cleanup removed its nodes, edges, and search documents in 1.8 s without the earlier multi-minute search-document deletion stall.

Incremental update matrix

Measured on a pristine 1,592-file Kotlin corpus with isolated SQLite databases. The changed files were copied into a wholly new package, so update intentionally selected the correctness-preserving full-build fallback.

Added files	Update	Build	Update delta	Graph diff
15 (1%)	15.22 s	13.98 s	+8.9%	0
159 (10%)	15.41 s	14.04 s	+9.8%	0
796 (50%)	16.78 s	15.03 s	+11.6%	0
1,592 (100%)	19.32 s	16.51 s	+17.0%	0

As a control, adding 15 files to an existing package stayed incremental: 1.46 s update vs 14.57 s build, with zero graph differences. A later 100% comment-modify benchmark using bulk incremental deletion completed in 25.15 s versus 10.77 s for a fresh build, also with zero persisted graph differences.

Schema changes

• version 11: bounded unresolved lookup/fingerprint hash columns and indexes; derived unresolved state is invalidated during migration
• version 12: PostgreSQL annotation summary/context columns use unbounded text, matching SQLite and preserving oversized annotations
• version 13: (namespace, file_path) node index for resolver and namespace-scoped file reads

Validation

Passed checks include:

• full CGO_ENABLED=1 go test -tags "fts5" ./... -count=1
• PostgreSQL graph store tests, including DeleteGraph with 66,000 nodes
• annotation bulk SQL-count, tag-replacement, and namespace-ownership tests
• unresolved long-value, hash-collision, migration up/down, and replay tests
• incremental bulk deletion, node/annotation batching, and graph-equivalence tests
• focused reference allocation, suffix behavior, JVM member extraction, and typed-receiver chain tests
• two fresh 22,719-file full-corpus builds with matching file/node/edge totals
• current-source ccg build ., ccg docs --out docs, and ccg lint
• git diff --check

ccg lint exits successfully with one pre-existing notice for the private helper graphgorm.unresolvedIndexHash. The existing Tree-sitter Lua null-character compiler warning remains non-fatal and unchanged.