Skip to content

fix(db): a database newer than the binary is not a corrupt database (destroys the corpus) - #464

Merged
runyourempire merged 1 commit into
mainfrom
fix/newer-schema-is-not-corruption
Aug 15, 2026
Merged

fix(db): a database newer than the binary is not a corrupt database (destroys the corpus)#464
runyourempire merged 1 commit into
mainfrom
fix/newer-schema-is-not-corruption

Conversation

@runyourempire

Copy link
Copy Markdown
Collaborator

Opening a newer database with an older 4DA build destroys the user's corpus

Found while rehearsing the schema-104 activation from #462 on a copy of the founder's
live database. Measured, not inferred.

The 2026-08-14 build opened a schema-104 database. The migration guard in migrations.rs
correctly refused it. get_database()'s last-resort fallback then read that refusal as
corruption:

WARN  4da::db: Database open failed after preemptive recovery — last-resort fallback
      error=Database schema version 104 is newer than this version of 4DA supports (max 103).
INFO  4da::db: Corrupt database preserved, creating fresh database
      corrupt="…\4da.db.corrupt"
INFO  4da::db: Running Phase 1: multi-format files (schema version 1 -> 2)

Before / after, same directory:

file size schema source_items
4da.db (what the app now uses) 1.3 MB 103 0
4da.db.corrupt (the real corpus) 283 MB 104 15,659

The app comes up empty and starts re-fetching from zero. One log line is the only trace.
Every rollback to a previous release does this — and on this fleet it needs no rollback
at all, because the scheduled background refresh runs whatever was last compiled into
target/debug/fourda.exe.

The guard itself has been correct since 2026-03-29. The bug is entirely in how the caller
classifies its error.

The fix

1. A schema-too-new error is routed away from the corrupt-db fallback. state.rs now
bails out and returns the error, mirroring the is_database_lock_contention bail-out
directly above it — that precedent already existed for exactly this shape of problem.

The detector keys on both SQLITE_MISMATCH and a phrase shared with the producer via
SCHEMA_TOO_NEW_PHRASE, so:

  • an unrelated SQLITE_MISMATCH cannot suppress genuine corruption recovery, and
  • producer and detector cannot drift apart.

Getting this wrong in either direction is expensive: too narrow and the corpus is
destroyed; too broad and a genuinely corrupt database never heals. Both directions are
tested.

2. Quarantine copies are no longer auto-pruned. #462 added *.db.corrupt /
*.db.corrupt-<unix> to the backup pruner to reclaim disk. That was my change and it was
unsafe: a quarantined database is the user's only copy of that data, and — per the bug
above — can be their entire live corpus. Reclaiming 338 MB is not worth a chance of
deleting 15,659 items.

They stay classified so the pruner can report the disk they hold; only *.db.backup.vN
and hand-made *.bak-* snapshots are collected.

3. The guard gets tests. It had none in ~5 months. A future schema is refused with an
error that says why; a database at the current schema still reopens cleanly with a
consistent FTS index (so the guard cannot pass by being indiscriminate); and one test
asserts end-to-end that the error the guard actually produces is the one the detector
recognises
— testing them apart would let them drift and silently re-arm the
corpus-destroying path.

Also

Documents the two skew traps in CLAUDE.md's gotchas. Both cost real time this week and both
present as your own bug:

  • Old binary vs. newer database (above) — migrate and rebuild together.
  • Stale worktree basemain moved 6 commits during one agent session, after which
    the pre-commit ghost gate failed citing 13 "NEW" ghost commands in files the branch never
    touched. They had simply been allowlisted upstream in fix(osv): drop the expired REMOVE BY field; unblock Rust commits on the ghost gate #434. Re-fetch and rebase before
    committing; if a gate blames code you did not write, check your base before you touch an
    allowlist.

Verification

cargo fmt --check clean · cargo clippy -- -D warnings clean for both default and
--features experimental · cargo test --lib 4,378 passed / 0 failed / 8 ignored.

The founder's live database was not migrated and not written to. All of the above
was measured on copies taken with SQLite's online backup API. That decision is the point of
this PR: activating #462 before the binaries are rebuilt would have destroyed the corpus on
the next scheduled refresh.

Activation, in the right order

With this merged, activation is safe and is a single ordered operation:

  1. git pull in D:\4DA
  2. cd src-tauri && cargo build --bin fourda --bin fourda-engine
  3. launch — the migration runs, rebuilding the FTS index (447 ms on the 15,659-item corpus)

Step 2 before step 3 is the whole rule. Doing 3 with a stale step 2 is what this PR makes
survivable.

Opening a newer database with an older 4DA build destroys the user's corpus.

Measured on a copy of the founder's live database, not inferred. The 2026-08-14
build opened a schema-104 database; the migration guard in `migrations.rs`
correctly refused it; and `get_database()`'s last-resort fallback then read that
refusal as corruption. It renamed 296 MB / 15,659 items to `4da.db.corrupt` and
created a fresh 1.3 MB database with **0 items**:

    WARN  Database open failed after preemptive recovery — last-resort fallback
          error=Database schema version 104 is newer than this version of 4DA supports
    INFO  Corrupt database preserved, creating fresh database

The app comes up empty and starts re-fetching from zero. One log line is the only
trace. Every rollback to a previous release does this, and on this fleet the
scheduled background refresh runs whatever was last compiled into
`target/debug/fourda.exe` — so it needs no rollback at all, just a stale build.

Three changes:

- `state.rs` now recognises the guard's refusal and returns it instead of falling
  into the fallback, mirroring the existing `is_database_lock_contention` bail-out
  directly above it. The detector keys on BOTH `SQLITE_MISMATCH` and a phrase
  shared with the producer via `SCHEMA_TOO_NEW_PHRASE`, so an unrelated type
  mismatch cannot suppress genuine corruption recovery, and the two cannot drift.

- Quarantine copies (`*.db.corrupt`, `*.db.corrupt-<unix>`) are no longer
  auto-pruned. #462 added them to the backup pruner to reclaim disk; that is
  unsafe. A quarantined database is the user's only copy of that data, and — per
  the bug above — can be their entire live corpus. They stay classified so the
  pruner can report the disk they hold, and only `*.db.backup.vN` and hand-made
  `*.bak-*` snapshots are collected.

- The schema-downgrade guard, which has existed since 2026-03-29 with zero test
  coverage, gets tests: a future schema is refused with an error that says why,
  and a database at the current schema still reopens cleanly with a consistent
  FTS index (so the guard cannot pass by being indiscriminate). One test asserts
  end-to-end that the error the guard ACTUALLY produces is the one the detector
  recognises — testing them apart would let them drift and silently re-arm the
  corpus-destroying path.

Also documents in CLAUDE.md the two skew traps that cost real time this week, both
of which present as your own bug: an old binary against a new database (above),
and a stale worktree base — `main` moved 6 commits during one agent session, after
which the pre-commit ghost gate failed citing 13 "NEW" ghost commands in files the
branch never touched. They had simply been allowlisted upstream in #434. Re-fetch
and rebase before committing; if a gate blames code you did not write, check your
base before you touch an allowlist.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2
@runyourempire
runyourempire enabled auto-merge (squash) August 15, 2026 17:59
@runyourempire
runyourempire merged commit 34cf69c into main Aug 15, 2026
13 checks passed
@runyourempire
runyourempire deleted the fix/newer-schema-is-not-corruption branch August 15, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant