fix(headless): maintenance ran every cycle and left no trace, so its own detector still read "frozen" - #475
Merged
Conversation
…own detector still read "frozen" #462 gave the headless engine its own DB maintenance, and it works — the WAL checkpoint and `PRAGMA optimize` genuinely run at the end of every cycle. But `run_cycle_maintenance()` never recorded the run. `scheduler_state::persist_run(DB_MAINTENANCE, ..)` was reachable only from the GUI monitoring loop (`monitoring.rs:510`), so on a machine that runs the engine via the scheduled task and rarely opens the GUI, that row stays frozen at whenever a GUI last ran. Measured on the live corpus 2026-08-17: db_maintenance runs=25 last=3001 min ago (~50 hours) vacuum runs= 1 last=6449 min ago while the engine had checkpointed on every cycle throughout — observed directly in the 2026-08-16 run log ("Scheduled maintenance: WAL checkpoint + optimize complete wal_mb=9"). That row is not incidental telemetry. It is the exact signal the 2026-08-16 forensic audit used to *find* the missing headless maintenance in the first place: "scheduler_state froze at 2026-08-12 13:54:19 while fourda-engine kept writing until 08-13 14:05:11 — a 24-hour window with zero checkpoint". Leaving it unwritten converts that detector into a permanent false negative, so the next person to check concludes maintenance is broken when it is running fine — or, worse, concludes it is broken when it genuinely is, and cannot tell the two apart. The stamp is written AFTER the work, not before it. The GUI path stamps first and then runs, which is defensible there because it is also using the stamp as an interval lock. Here the stamp has one job — to say maintenance ran — so it is only written when it did. An unreadable clock skips the write rather than defaulting to `0`. A zero timestamp renders as "never ran", which is the same lie in the other direction; losing one stamp is recoverable, a false one is not. Not covered by a unit test: `persist_run` opens its own connection through the global database path, so there is no seam to point at a temp DB without a refactor larger than the fix. The falsifier is direct and cheap instead — `scheduler_state.db_maintenance` must advance within one engine cycle of the rebuilt binary running. I will verify that live after merge rather than assert it here. Leaves open, deliberately: nothing runs VACUUM on a headless-only machine (`run_scheduled_maintenance` is checkpoint + optimize; VACUUM is a separate, heavier GUI-side job — `vacuum` above has run exactly once in the corpus's lifetime). That needs its own decision about cadence and lock behaviour, not a line in this function. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#462 gave the headless engine its own DB maintenance, and it works — the WAL checkpoint and
PRAGMA optimizegenuinely run at the end of every cycle.But
run_cycle_maintenance()never recorded the run.scheduler_state::persist_run(DB_MAINTENANCE, ..)was reachable only from the GUI monitoring loop (monitoring.rs:510), so on a machine that runs the engine via the scheduled task and rarely opens the GUI, that row stays frozen at whenever a GUI last ran.Measured on the live corpus today:
…while the engine had checkpointed on every cycle throughout — observed directly in yesterday's run log (
Scheduled maintenance: WAL checkpoint + optimize complete wal_mb=9).Why this is worth its own PR
That row is not incidental telemetry. It is the exact signal the forensic audit used to find the missing headless maintenance in the first place — "scheduler_state froze at 2026-08-12 13:54:19 while fourda-engine kept writing until 08-13 14:05:11 — a 24-hour window with zero checkpoint".
Leaving it unwritten converts that detector into a permanent false negative. The next person to check concludes maintenance is broken when it is running fine — or concludes it is broken when it genuinely is, and cannot tell the two apart. That is the audit's own thesis reintroduced by the fix for it.
Two deliberate choices
The stamp is written after the work, not before. The GUI path stamps first and then runs, which is defensible there because it also uses the stamp as an interval lock. Here the stamp has one job — to say maintenance ran — so it is only written when it did.
An unreadable clock skips the write rather than defaulting to
0. A zero timestamp renders as "never ran", which is the same lie in the other direction. Losing one stamp is recoverable; a false one is not.Verification
cargo checkandcargo clippy --bin fourda-engine -- -D warningsclean;cargo fmt --checkclean.Not covered by a unit test, stated plainly:
persist_runopens its own connection through the global database path, so there is no seam to point it at a temp DB without a refactor larger than the fix. The falsifier is direct and cheap instead —scheduler_state.db_maintenancemust advance within one engine cycle of the rebuilt binary running. I will verify that live after merge rather than assert it here.Left open deliberately
Nothing runs VACUUM on a headless-only machine.
run_scheduled_maintenanceis checkpoint + optimize; VACUUM is a separate, heavier GUI-side job — andvacuumabove has run exactly once in this corpus's lifetime. That needs its own decision about cadence and lock behaviour, not a line in this function.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fq96xWyPQjx2bCCzWtsnC9