Skip to content

fix(headless): maintenance ran every cycle and left no trace, so its own detector still read "frozen" - #475

Merged
runyourempire merged 1 commit into
mainfrom
worktree-headless-maintenance-telemetry
Aug 16, 2026
Merged

fix(headless): maintenance ran every cycle and left no trace, so its own detector still read "frozen"#475
runyourempire merged 1 commit into
mainfrom
worktree-headless-maintenance-telemetry

Conversation

@runyourempire

Copy link
Copy Markdown
Collaborator

#462 gave the headless engine its own DB maintenance, and it works — the WAL checkpoint and PRAGMA optimize genuinely run at the end of every cycle.

But run_cycle_maintenance() never recorded the run. scheduler_state::persist_run(DB_MAINTENANCE, ..) was reachable only from the GUI monitoring loop (monitoring.rs:510), so on a machine that runs the engine via the scheduled task and rarely opens the GUI, that row stays frozen at whenever a GUI last ran.

Measured on the live corpus today:

db_maintenance   runs=25   last=3001 min ago   (~50 hours)
vacuum           runs= 1   last=6449 min ago

…while the engine had checkpointed on every cycle throughout — observed directly in yesterday's run log (Scheduled maintenance: WAL checkpoint + optimize complete wal_mb=9).

Why this is worth its own PR

That row is not incidental telemetry. It is the exact signal the forensic audit used to find the missing headless maintenance in the first place — "scheduler_state froze at 2026-08-12 13:54:19 while fourda-engine kept writing until 08-13 14:05:11 — a 24-hour window with zero checkpoint".

Leaving it unwritten converts that detector into a permanent false negative. The next person to check concludes maintenance is broken when it is running fine — or concludes it is broken when it genuinely is, and cannot tell the two apart. That is the audit's own thesis reintroduced by the fix for it.

Two deliberate choices

The stamp is written after the work, not before. The GUI path stamps first and then runs, which is defensible there because it also uses the stamp as an interval lock. Here the stamp has one job — to say maintenance ran — so it is only written when it did.

An unreadable clock skips the write rather than defaulting to 0. A zero timestamp renders as "never ran", which is the same lie in the other direction. Losing one stamp is recoverable; a false one is not.

Verification

cargo check and cargo clippy --bin fourda-engine -- -D warnings clean; cargo fmt --check clean.

Not covered by a unit test, stated plainly: persist_run opens its own connection through the global database path, so there is no seam to point it at a temp DB without a refactor larger than the fix. The falsifier is direct and cheap instead — scheduler_state.db_maintenance must advance within one engine cycle of the rebuilt binary running. I will verify that live after merge rather than assert it here.

Left open deliberately

Nothing runs VACUUM on a headless-only machine. run_scheduled_maintenance is checkpoint + optimize; VACUUM is a separate, heavier GUI-side job — and vacuum above has run exactly once in this corpus's lifetime. That needs its own decision about cadence and lock behaviour, not a line in this function.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fq96xWyPQjx2bCCzWtsnC9

…own detector still read "frozen"

#462 gave the headless engine its own DB maintenance, and it works — the WAL
checkpoint and `PRAGMA optimize` genuinely run at the end of every cycle.

But `run_cycle_maintenance()` never recorded the run.
`scheduler_state::persist_run(DB_MAINTENANCE, ..)` was reachable only from the
GUI monitoring loop (`monitoring.rs:510`), so on a machine that runs the engine
via the scheduled task and rarely opens the GUI, that row stays frozen at
whenever a GUI last ran.

Measured on the live corpus 2026-08-17:

    db_maintenance   runs=25   last=3001 min ago   (~50 hours)
    vacuum           runs= 1   last=6449 min ago

while the engine had checkpointed on every cycle throughout — observed directly
in the 2026-08-16 run log ("Scheduled maintenance: WAL checkpoint + optimize
complete wal_mb=9").

That row is not incidental telemetry. It is the exact signal the 2026-08-16
forensic audit used to *find* the missing headless maintenance in the first
place: "scheduler_state froze at 2026-08-12 13:54:19 while fourda-engine kept
writing until 08-13 14:05:11 — a 24-hour window with zero checkpoint". Leaving
it unwritten converts that detector into a permanent false negative, so the next
person to check concludes maintenance is broken when it is running fine — or,
worse, concludes it is broken when it genuinely is, and cannot tell the two
apart.

The stamp is written AFTER the work, not before it. The GUI path stamps first
and then runs, which is defensible there because it is also using the stamp as
an interval lock. Here the stamp has one job — to say maintenance ran — so it is
only written when it did.

An unreadable clock skips the write rather than defaulting to `0`. A zero
timestamp renders as "never ran", which is the same lie in the other direction;
losing one stamp is recoverable, a false one is not.

Not covered by a unit test: `persist_run` opens its own connection through the
global database path, so there is no seam to point at a temp DB without a
refactor larger than the fix. The falsifier is direct and cheap instead —
`scheduler_state.db_maintenance` must advance within one engine cycle of the
rebuilt binary running. I will verify that live after merge rather than assert
it here.

Leaves open, deliberately: nothing runs VACUUM on a headless-only machine
(`run_scheduled_maintenance` is checkpoint + optimize; VACUUM is a separate,
heavier GUI-side job — `vacuum` above has run exactly once in the corpus's
lifetime). That needs its own decision about cadence and lock behaviour, not a
line in this function.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@runyourempire
runyourempire merged commit 017787d into main Aug 16, 2026
15 checks passed
@runyourempire
runyourempire deleted the worktree-headless-maintenance-telemetry branch August 16, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant