fix(db): sync the FTS index from the schema; give the headless engine its own maintenance

[runyourempire]

The defect

source_items_fts is an external content FTS5 table (content='source_items'). SQLite
stores only the inverted index and reads the text back out of source_items, so it
maintains nothing automatically — every write has to be mirrored in by hand. Three
paths never were:

Path	What it did
batch_upsert_pending_source_items	Wrote title/content with no FTS statement at all — any item whose embedding failed was unsearchable.
cleanup_old_items, run_maintenance, prune_noise, trg_source_items_cascade_delete	Removed rows from source_items and left their postings behind. No FTS 'delete' existed anywhere in the repository.
upsert_source_item, batch_upsert_source_items	Used INSERT OR REPLACE after updating source_items. On an external-content table the implicit REPLACE-delete reads the old values back from the content table — which already held the NEW text — so it removed the new postings and stranded the old.

The third one was not in the original report and is the dominant cause on the live corpus:
it runs on every re-fetch of an item whose text changed.

Measured on a read-only snapshot of the live 247 MB corpus

BEFORE  integrity-check rank0=PASS  rank1=FAIL(database disk image is malformed)
        2,631 divergent terms  |  2,148 stale + 2,630 missing vocab rows  |  38 items affected
AFTER   integrity-check rank0=PASS  rank1=PASS

Rank matters, and it is why this went unnoticed. On an external-content table
('integrity-check', 0) checks only that the index's own b-trees are well formed — it
passes on the corrupted corpus. Only ('integrity-check', 1) recomputes the index
checksum from source_items. PRAGMA quick_check, which Database::new runs at every
startup, also passes.

The user-visible symptom, reproduced on real rows: searching a word that had been edited
out of an item still returns that item.

The fix

Schema 104 replaces all hand-maintenance with trg_source_items_fts_{insert,update,delete}
and rebuilds the index once. The three manual FTS statements are deleted. Measured
end-to-end on a copy of the live database, the whole Database::new path — pre-migration
backup copy of 247 MB included — took 5.9 s.

A trigger is the right home for two reasons: no future call site can forget it, and the
'delete' command needs the pre-update values that only OLD.* still has.

Two deliberate narrowings on the UPDATE trigger make this strictly less write
amplification than what it replaces:

• OF title, content — the scoring drain stamps thousands of relevance_score updates
  per run and now touches the index not at all.
• WHEN OLD.title IS NOT NEW.title OR OLD.content IS NOT NEW.content — a re-fetch that
  rewrites identical text (the common case) does no index work, where the unconditional
  INSERT OR REPLACE always did.

Migration, not startup repair. A guarded startup repair would need integrity-check
rank 1 — a full tokenizing scan of the content table — on every launch of both the GUI and
the engine, and it would grow linearly with the corpus. The migration runs exactly once,
inside the existing transactional + backed-up + migration_history-recorded framework.
fts_integrity_check() and rebuild_fts_index() are exposed for diagnostics and tests.

Also in this PR

Defect 2 — headless engine had no DB maintenance. run_scheduled_maintenance now runs
at the end of every headless cycle, at the end of a drain, and every 10 drain cycles.
Every previous caller lived in the GUI monitoring loop, so a headless-only day did all of
the writing and none of the upkeep — scheduler_state froze at 2026-08-12 13:54 while
fourda-engine wrote until 08-13 14:05 with zero checkpoints. The TRUNCATE gate drops
50 MB → 16 MB, a size the engine can actually reach: wal_autocheckpoint = 1000 at a
4 KiB page churns around 4 MB, so an un-truncated WAL sat between the two numbers
indefinitely (25.9 MB when reported, 47.7 MB a day later).

Defect 3 — backup pruning. The pruner understood only <db>.backup.vN, so two
unbounded families grew forever: hand-made .bak-pre-* snapshots and .db.corrupt-<unix>
quarantine copies (the corruption-recovery path writes one per incident). It now keeps the
newest 2 of each family independently, where the retention unit for a hand-made
snapshot is the snapshot plus its -wal/-shm siblings — a database restored without
its WAL, or a WAL without its database, is worse than neither. (The first cut of this
counted the sibling as its own slot; my own families_are_pruned_independently test caught
it splitting pairs, and there is now a test pinning the pairing rule.)

Conservative by construction: matched strictly against this database's own file name, an
unparseable suffix is left alone, a file whose mtime cannot be read sorts newest rather
than oldest, 4da.db.bakery is not mistaken for a backup, and the just-written backup is
protected. The retention rule is extracted as a pure function with tests — the previous
prune bug survived precisely because the rule was only reachable through real read_dir
output. Nothing is deleted by this PR; the pruner is merely made capable.

Defect 4 — partial. prune_orphaned_project_dependencies is restructured into
read / stat / write phases so std::fs::metadata no longer runs with a transaction open
(one dead mapped network drive pinned a WAL snapshot for a full SMB timeout, stalling
checkpointing), and its delete phase opens BEGIN IMMEDIATE so contention becomes a
bounded wait instead of an un-retryable SQLITE_BUSY_SNAPSHOT on lock upgrade.

The broader retry_on_busy + BEGIN IMMEDIATE sweep across the codebase is not done —
see "Left untouched" below.

Verification

The new tests are real. Restored the pre-fix sources.rs and migrations.rs, kept the
final set of tests, and re-ran: 8 of 9 fail, each with the symptom it names.

pending_embedding_items_are_indexed_for_search      left: 0, right: 1
reupserting_retires_the_terms_...                   a title term that was edited out must stop matching
retention_delete_removes_the_items_postings         a deleted item must leave no postings behind
batch_upsert_indexes_inserts_and_updates            the batch update path must retire replaced terms too
every_delete_path_keeps_the_index_consistent        prune_noise must not strand postings
hybrid_search_bm25_leg_tracks_the_current_text      the keyword leg must not still match text the item no longer contains
rebuild_repairs_an_index_that_diverged_...          FAILED
fresh_database_installs_the_fts_triggers            left: [], right: [3 triggers]

scoring_updates_do_not_disturb_the_index            ok   (asserts a property that already held)

The real migration was run over the real corpus. A throwaway in-crate test (not
committed) pointed Database::new at a copy of the live snapshot, so the whole Rust path
ran — sqlite-vec registration, pre-migration backup, schema 103 → 104, the new pruner:

MIGRATED 12273 items to schema 104 in 5.87s; source_vec rows=12273   -> integrity-check rank1 PASS
RETENTION deleted 12273 real rows                                    -> integrity-check rank1 PASS
BACKUPS v101=pruned v102=kept v103=kept  bak-pre-v7=kept bak-pre-v8=kept
        corrupt=kept  settings.json.bak-pre-keepme=kept

The Python-level verification could not cover this: stock sqlite3 cannot load vec0, so
nothing touching source_vec was exercised there.

Gates: cargo fmt --check clean · cargo clippy -- -D warnings clean for both default
and --features experimental · check-file-sizes, check-doc-location,
private-asset-guard, ghost-commands, compound-quality-check and
validate-translations clean.

cargo test --tests — 4,640 passed / 0 failed / 13 ignored across all 9 binaries:

binary	passed
fourda_lib (lib)	4,316 (baseline 4,300; +16 new)
stack_simulation	124
victauri_dogfood	157
pipeline_integration	13
cli	13
migration_tests	12
source_resilience	5
fourda, fourda-engine	0 (no tests)

The operator's live database was never opened for writing. All measurement was done on
snapshots taken with SQLite's online backup API (a plain file copy of a live WAL database
tears).

Two things worth knowing about the machine, not this PR

⚠️ D: hit 0.03 GB free mid-run and killed a link (os error 112) — Defect 3's exact
failure mode, arriving on its own. 14 worktrees hold ~230 GB of src-tauri/target between
them, and cleanup-orphaned-worktrees.cjs --execute reclaims none of it (0 dirs removable;
it only has 8 merged branches to delete). Space was freed and every suite re-run.

⚠️ A stale worktree base looks exactly like a code regression. This branch was cut from
c1fd348c; by commit time origin/main was 6 commits ahead, and the pre-commit ghost gate
failed with 13 new ghost commands in files this PR never touches. The cause was that
scripts/ghost-command-backlog.json had gained 13 entries upstream (#434 — "unblock Rust
commits on the ghost gate"). Rebasing onto current main cleared it. Worth a note in the
worktree docs: re-fetch before committing, because that failure reads as your fault.

Left untouched

The retry_on_busy helper + BEGIN IMMEDIATE sweep (rest of Defect 4). Applying
BEGIN IMMEDIATE and a retry wrapper to the codebase's unchecked_transaction() call
sites is a large, uniform change to locking behavior, and doing it to some write paths is
worse than doing it to none — mixed deferred/immediate writers can deadlock in ways neither
does alone. It wants its own PR with a contention benchmark. The one site fixed here was
fixed because it had a specific, independently-diagnosed bug (blocking I/O inside a
transaction) rather than as a partial sweep.

busy_timeout left at 5000 ms. Raising it is a one-line change that would reduce
SQLITE_BUSY, but it trades errors for UI stalls of up to the new timeout, and I have no
contention measurement to size it from. Deliberate non-change.

The 262 ad-hoc Connection::open call sites are out of scope as instructed.

---

Nothing on the operator's machine was deleted or mutated. The migration will run on
first launch after merge; it writes 4da.db.backup.v103 first (existing behavior) and the
whole step measured 5.9 s on a copy of the current corpus.

Note for activation: both binaries need rebuilding before the migration takes effect —
fourda.exe and fourda-engine.exe each run Database::new, and whichever starts first
performs the migration.