Skip to content

fix: CLI session-restore hardening — TERM-25 wrong-thread guard, compound-restart coverage, flake/open-item closure#522

Merged
danshapiro merged 4 commits into
mainfrom
fix/cli-restore-hardening
Jul 24, 2026
Merged

fix: CLI session-restore hardening — TERM-25 wrong-thread guard, compound-restart coverage, flake/open-item closure#522
danshapiro merged 4 commits into
mainfrom
fix/cli-restore-hardening

Conversation

@danshapiro

Copy link
Copy Markdown
Owner

CLI session-restore hardening (parallel-wave track 2)

Four commits, each TDD'd and gate-verified, closing restore-hardening gaps against origin/main (ec9970c).

1. TERM-25 — reject wrong-thread Codex resume loudly (1b2aba60)

thread/resume answering with a DIFFERENT thread id than requested was silently tolerated at all three Rust resume sites in crates/freshell-freshagent/src/codex.rs (debug_assert_eq! — a no-op in release — or no check at all), leaving the sidecar on the wrong thread while the session stayed registered under the requested id. All three sites (create-with-resume, not-tracked attach, crash recovery) now reject the mismatch loudly: sidecar torn down + reaped, explicit error frame, tracing::error!, wrong id never adopted.

Unit pins (RED before fix, GREEN after):

  • handle_create_with_resume_wrong_thread_id_fails_create_and_never_adopts
  • handle_attach_unknown_session_wrong_thread_id_is_rejected_not_adopted
  • crash_recovery_resume_wrong_thread_id_is_rejected_not_silently_recovered

Root cause found in the pinned e2e itself: restore-matrix SCENARIO 7's blanket test.fail() was an artifact of a TEST BUG, not product behavior — originalSessionId was assigned the void result of expect.poll(...).toBe(...), so the final assertion compared the leaf id against undefined and could never pass on either server kind. With the capture fixed, the pane-level contract (thread-B never rendered/adopted) holds on BOTH kinds; pin removed, frozen-legacy internal gap (no adapter-level guard, unobservable at this UI level) documented in the spec header. Checklist TERM-25 evidence updated accordingly.

2. Compound-restart coverage (bd94023c)

Closes the two never-tested disruption modes from docs/plans/2026-07-19-state-sync-resilience-assessment.md §7 (new test/e2e-browser/specs/compound-restart-rust.spec.ts, rust-chromium):

  • MODE A — abrupt SIGKILL death + revival (WSL-restart-like), no page reload: sidebar-resumed codex terminal pane recovers via the client's own reconnect — same durable sessionRef, resume <id> argv re-applied (argv-log delta), pane re-anchored + non-blank, sidebar row linked (data-has-tab="true"), terminal_identity_unresolved never fires.
  • MODE B — server and browser restarting together: abrupt death + revival, then an immediate page.reload() interrupting the in-flight restore; same contract plus an idempotency re-check (second reload doesn't revert recovery).

New fixture capability: RustServer.restartAbrupt() (SIGKILL + reboot on same home/port/token; fixture-side ownership-safe descendant sweep is the primary PTY reap in this mode). Both modes are GREEN — the current product handles both compounds; the specs now pin that permanently. Assertion shapes inherit codex-terminal-bounce-rust.spec.ts's proven-discriminating pattern (RED against the real 2026-07-22 incident).

3. restore-matrix scenario 3 + multi-client flake (4b992f8d, docs)

  • Scenario 3 (campaign open item 3): already fixed on main by 8fd9233a; re-verified GREEN at this base (opening a seeded historical session…, rust-chromium). Doc annotated RESOLVED with evidence.
  • Multi-client reconnect flake (campaign open item 4): NOT reproducible at this base — deflaked by 031a7c12 (over-constrained exact-attach-intent assertion, relaxed with in-spec evidence). multi-client.spec.ts green 3x on BOTH kinds, including one run under concurrent cargo-build load. Doc annotated with evidence.

4. OpenCode restore gap check (verification record — no gaps found)

Compared docs/plans/2026-07-18-opencode-terminal-restore-spec.md slices A–D against the implementation:

  • Slice A (crates/freshell-sessions/src/opencode_locator.rs): present, 18 unit tests covering spec test-list items 1–16 including BOTH row-timing shapes (row_created_at_spawn_before_any_enter_resolves_via_spawn_window, row_created_lazily_at_first_enter_resolves_via_enter_window), ambiguity refuse, idle short-circuit (tick_while_unarmed_performs_zero_db_scans), missing-DB tolerance.
  • Slice B (crates/freshell-ws/src/opencode_association.rs): present, 6 tests covering spec items 17–20 (submit recognition, arm/skip rules, drain+broadcast on a real seeded DB row).
  • Slice C (wiring): WsState.opencode_locator (lib.rs:219,669), submit/arm hooks in terminal.rs, construction + sweep in main.rs:335-508, list_sessions_since in parse/opencode.rs:348.
  • Slice D (client): no client changes, as mandated.
  • e2e (spec item 21): opencode-terminal-restore-rust.spec.ts proves restart restore via opencode --session <id> argv AND fresh restore for a never-submitted pane.

Conclusion: all slices implemented with the spec's full test list; no real gaps to close.

Verification

  • cargo fmt --all --check GREEN (incl. one pre-existing whitespace fix, 6f4aa9fb); cargo clippy --workspace --exclude freshell-tauri --all-targets exit 0 (warnings pre-existing at base, none in changed code); cargo test --workspace --exclude freshell-tauri all green (freshell-freshagent 240/240).
  • Playwright: restore-matrix full suite 20/20 (both projects) 2x; compound-restart-rust 2/2 3x; multi-client 12/12 3x (both kinds, incl. under load); codex-terminal-bounce-rust green (touched-helper sanity).
  • Coordinated gate: FRESHELL_TEST_SUMMARY="track2 restore hardening pre-PR" npm testfull-suite success exit=0 at 6f4aa9fb (dirty:0).

🤖 Generated with Amplifier

codex and others added 4 commits July 24, 2026 00:18
thread/resume answering with a different thread id than requested was
silently tolerated at all three Rust resume sites (create-with-resume,
not-tracked attach, crash recovery): a debug_assert_eq! (no-op in
release) or no check at all, after which the sidecar sat on the wrong
thread while the session stayed registered under the requested id --
routing every subsequent send to an unrelated conversation.

Now all three sites verify the returned thread id and reject a mismatch
loudly: sidecar torn down and reaped, an explicit error emitted
(freshAgent.create.failed / CODEX_ATTACH_RESUME_FAILED /
CODEX_ATTACH_RESPAWN_FAILED), a tracing::error! record, and the wrong
id never adopted. Unit pins (RED before the fix, GREEN after):
- handle_create_with_resume_wrong_thread_id_fails_create_and_never_adopts
- handle_attach_unknown_session_wrong_thread_id_is_rejected_not_adopted
- crash_recovery_resume_wrong_thread_id_is_rejected_not_silently_recovered

restore-matrix SCENARIO 7 (TERM-25): the blanket test.fail() pin turns
out to have been an artifact of a test bug, not product behavior --
originalSessionId was assigned the void result of
expect.poll(...).toBe(...), so the final assertion compared the leaf id
against undefined and could never pass on either server kind. With the
capture fixed, the pane-level contract (thread-B never rendered or
adopted) empirically holds on BOTH kinds, so the pin is removed; the
frozen legacy adapter's missing internal guard (not observable at this
UI level) is documented in the spec header. Green both projects.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
…owser together

Close the two never-tested disruption modes the state-sync resilience
assessment (docs/plans/2026-07-19-state-sync-resilience-assessment.md
§7) flags as 'where the next incident most likely lives':

MODE A -- abrupt server death + revival (WSL-restart-like): SIGKILL the
server (no graceful shutdown, no PTY reaping, no clean WS closes), boot
a fresh process on the same home/port/token, and prove a sidebar-
resumed codex terminal pane recovers WITHOUT a page reload: same
durable sessionRef, resume argv re-applied (argv-log delta), pane
re-anchored + non-blank, sidebar row linked (data-has-tab), and the
terminal_identity_unresolved invariant WARN never fires.

MODE B -- server and browser restarting together: abrupt server death +
revival, then reload the page immediately (before the old page's
reconnect/restore settles), interrupting the in-flight restore with a
full client restart. Same recovery contract, plus an idempotency
re-check (a second reload must not revert the recovery).

New fixture capability: RustServer.restartAbrupt() -- SIGKILL + reboot
on the same home/port/token; because the server's graceful reap never
runs, the fixture's ownership-safe descendant sweep is the primary
PTY-reap mechanism for this mode.

Both modes green on rust-chromium. Assertion shapes inherit
codex-terminal-bounce-rust.spec.ts's proven-discriminating pattern
(RED against the real 2026-07-22 incident).

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
- Checklist TERM-25: record the Rust wrong-thread guard (three resume
  sites, unit pins) and the scenario-7 pin removal (the expected-fail
  was a test-bug artifact -- void expect.poll result compared against
  undefined); re-scope the remaining unproven clauses.
- Campaign status open item 3 (restore-matrix scenario 3): resolved by
  8fd9233, re-verified green at ec9970c on rust-chromium.
- Campaign status open item 4 (multi-client reconnect flake): not
  reproducible at ec9970c -- deflaked by 031a7c1; green 3x on both
  kinds including one run under concurrent cargo-build load.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Verified formatting-only: `cargo fmt --all --check` flagged this at the
ec9970c base before any change on this branch touched the file.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@danshapiro
danshapiro merged commit eac1434 into main Jul 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants