fix(db): a database newer than the binary is not a corrupt database (destroys the corpus) - #464
Merged
Merged
Conversation
Opening a newer database with an older 4DA build destroys the user's corpus.
Measured on a copy of the founder's live database, not inferred. The 2026-08-14
build opened a schema-104 database; the migration guard in `migrations.rs`
correctly refused it; and `get_database()`'s last-resort fallback then read that
refusal as corruption. It renamed 296 MB / 15,659 items to `4da.db.corrupt` and
created a fresh 1.3 MB database with **0 items**:
WARN Database open failed after preemptive recovery — last-resort fallback
error=Database schema version 104 is newer than this version of 4DA supports
INFO Corrupt database preserved, creating fresh database
The app comes up empty and starts re-fetching from zero. One log line is the only
trace. Every rollback to a previous release does this, and on this fleet the
scheduled background refresh runs whatever was last compiled into
`target/debug/fourda.exe` — so it needs no rollback at all, just a stale build.
Three changes:
- `state.rs` now recognises the guard's refusal and returns it instead of falling
into the fallback, mirroring the existing `is_database_lock_contention` bail-out
directly above it. The detector keys on BOTH `SQLITE_MISMATCH` and a phrase
shared with the producer via `SCHEMA_TOO_NEW_PHRASE`, so an unrelated type
mismatch cannot suppress genuine corruption recovery, and the two cannot drift.
- Quarantine copies (`*.db.corrupt`, `*.db.corrupt-<unix>`) are no longer
auto-pruned. #462 added them to the backup pruner to reclaim disk; that is
unsafe. A quarantined database is the user's only copy of that data, and — per
the bug above — can be their entire live corpus. They stay classified so the
pruner can report the disk they hold, and only `*.db.backup.vN` and hand-made
`*.bak-*` snapshots are collected.
- The schema-downgrade guard, which has existed since 2026-03-29 with zero test
coverage, gets tests: a future schema is refused with an error that says why,
and a database at the current schema still reopens cleanly with a consistent
FTS index (so the guard cannot pass by being indiscriminate). One test asserts
end-to-end that the error the guard ACTUALLY produces is the one the detector
recognises — testing them apart would let them drift and silently re-arm the
corpus-destroying path.
Also documents in CLAUDE.md the two skew traps that cost real time this week, both
of which present as your own bug: an old binary against a new database (above),
and a stale worktree base — `main` moved 6 commits during one agent session, after
which the pre-commit ghost gate failed citing 13 "NEW" ghost commands in files the
branch never touched. They had simply been allowlisted upstream in #434. Re-fetch
and rebase before committing; if a gate blames code you did not write, check your
base before you touch an allowlist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUeKTKwNmdow8yUk3q8RB2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opening a newer database with an older 4DA build destroys the user's corpus
Found while rehearsing the schema-104 activation from #462 on a copy of the founder's
live database. Measured, not inferred.
The 2026-08-14 build opened a schema-104 database. The migration guard in
migrations.rscorrectly refused it.
get_database()'s last-resort fallback then read that refusal ascorruption:
Before / after, same directory:
4da.db(what the app now uses)4da.db.corrupt(the real corpus)The app comes up empty and starts re-fetching from zero. One log line is the only trace.
Every rollback to a previous release does this — and on this fleet it needs no rollback
at all, because the scheduled background refresh runs whatever was last compiled into
target/debug/fourda.exe.The guard itself has been correct since 2026-03-29. The bug is entirely in how the caller
classifies its error.
The fix
1. A schema-too-new error is routed away from the corrupt-db fallback.
state.rsnowbails out and returns the error, mirroring the
is_database_lock_contentionbail-outdirectly above it — that precedent already existed for exactly this shape of problem.
The detector keys on both
SQLITE_MISMATCHand a phrase shared with the producer viaSCHEMA_TOO_NEW_PHRASE, so:SQLITE_MISMATCHcannot suppress genuine corruption recovery, andGetting this wrong in either direction is expensive: too narrow and the corpus is
destroyed; too broad and a genuinely corrupt database never heals. Both directions are
tested.
2. Quarantine copies are no longer auto-pruned. #462 added
*.db.corrupt/*.db.corrupt-<unix>to the backup pruner to reclaim disk. That was my change and it wasunsafe: a quarantined database is the user's only copy of that data, and — per the bug
above — can be their entire live corpus. Reclaiming 338 MB is not worth a chance of
deleting 15,659 items.
They stay classified so the pruner can report the disk they hold; only
*.db.backup.vNand hand-made
*.bak-*snapshots are collected.3. The guard gets tests. It had none in ~5 months. A future schema is refused with an
error that says why; a database at the current schema still reopens cleanly with a
consistent FTS index (so the guard cannot pass by being indiscriminate); and one test
asserts end-to-end that the error the guard actually produces is the one the detector
recognises — testing them apart would let them drift and silently re-arm the
corpus-destroying path.
Also
Documents the two skew traps in CLAUDE.md's gotchas. Both cost real time this week and both
present as your own bug:
mainmoved 6 commits during one agent session, after whichthe pre-commit ghost gate failed citing 13 "NEW" ghost commands in files the branch never
touched. They had simply been allowlisted upstream in fix(osv): drop the expired REMOVE BY field; unblock Rust commits on the ghost gate #434. Re-fetch and rebase before
committing; if a gate blames code you did not write, check your base before you touch an
allowlist.
Verification
cargo fmt --checkclean ·cargo clippy -- -D warningsclean for both default and--features experimental·cargo test --lib4,378 passed / 0 failed / 8 ignored.The founder's live database was not migrated and not written to. All of the above
was measured on copies taken with SQLite's online backup API. That decision is the point of
this PR: activating #462 before the binaries are rebuilt would have destroyed the corpus on
the next scheduled refresh.
Activation, in the right order
With this merged, activation is safe and is a single ordered operation:
git pullinD:\4DAcd src-tauri && cargo build --bin fourda --bin fourda-engineStep 2 before step 3 is the whole rule. Doing 3 with a stale step 2 is what this PR makes
survivable.