fix: CLI session-restore hardening — TERM-25 wrong-thread guard, compound-restart coverage, flake/open-item closure#522
Merged
Conversation
thread/resume answering with a different thread id than requested was silently tolerated at all three Rust resume sites (create-with-resume, not-tracked attach, crash recovery): a debug_assert_eq! (no-op in release) or no check at all, after which the sidecar sat on the wrong thread while the session stayed registered under the requested id -- routing every subsequent send to an unrelated conversation. Now all three sites verify the returned thread id and reject a mismatch loudly: sidecar torn down and reaped, an explicit error emitted (freshAgent.create.failed / CODEX_ATTACH_RESUME_FAILED / CODEX_ATTACH_RESPAWN_FAILED), a tracing::error! record, and the wrong id never adopted. Unit pins (RED before the fix, GREEN after): - handle_create_with_resume_wrong_thread_id_fails_create_and_never_adopts - handle_attach_unknown_session_wrong_thread_id_is_rejected_not_adopted - crash_recovery_resume_wrong_thread_id_is_rejected_not_silently_recovered restore-matrix SCENARIO 7 (TERM-25): the blanket test.fail() pin turns out to have been an artifact of a test bug, not product behavior -- originalSessionId was assigned the void result of expect.poll(...).toBe(...), so the final assertion compared the leaf id against undefined and could never pass on either server kind. With the capture fixed, the pane-level contract (thread-B never rendered or adopted) empirically holds on BOTH kinds, so the pin is removed; the frozen legacy adapter's missing internal guard (not observable at this UI level) is documented in the spec header. Green both projects. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
…owser together Close the two never-tested disruption modes the state-sync resilience assessment (docs/plans/2026-07-19-state-sync-resilience-assessment.md §7) flags as 'where the next incident most likely lives': MODE A -- abrupt server death + revival (WSL-restart-like): SIGKILL the server (no graceful shutdown, no PTY reaping, no clean WS closes), boot a fresh process on the same home/port/token, and prove a sidebar- resumed codex terminal pane recovers WITHOUT a page reload: same durable sessionRef, resume argv re-applied (argv-log delta), pane re-anchored + non-blank, sidebar row linked (data-has-tab), and the terminal_identity_unresolved invariant WARN never fires. MODE B -- server and browser restarting together: abrupt server death + revival, then reload the page immediately (before the old page's reconnect/restore settles), interrupting the in-flight restore with a full client restart. Same recovery contract, plus an idempotency re-check (a second reload must not revert the recovery). New fixture capability: RustServer.restartAbrupt() -- SIGKILL + reboot on the same home/port/token; because the server's graceful reap never runs, the fixture's ownership-safe descendant sweep is the primary PTY-reap mechanism for this mode. Both modes green on rust-chromium. Assertion shapes inherit codex-terminal-bounce-rust.spec.ts's proven-discriminating pattern (RED against the real 2026-07-22 incident). 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
- Checklist TERM-25: record the Rust wrong-thread guard (three resume sites, unit pins) and the scenario-7 pin removal (the expected-fail was a test-bug artifact -- void expect.poll result compared against undefined); re-scope the remaining unproven clauses. - Campaign status open item 3 (restore-matrix scenario 3): resolved by 8fd9233, re-verified green at ec9970c on rust-chromium. - Campaign status open item 4 (multi-client reconnect flake): not reproducible at ec9970c -- deflaked by 031a7c1; green 3x on both kinds including one run under concurrent cargo-build load. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
Verified formatting-only: `cargo fmt --all --check` flagged this at the ec9970c base before any change on this branch touched the file. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CLI session-restore hardening (parallel-wave track 2)
Four commits, each TDD'd and gate-verified, closing restore-hardening gaps against
origin/main(ec9970c).1. TERM-25 — reject wrong-thread Codex resume loudly (
1b2aba60)thread/resumeanswering with a DIFFERENT thread id than requested was silently tolerated at all three Rust resume sites incrates/freshell-freshagent/src/codex.rs(debug_assert_eq!— a no-op in release — or no check at all), leaving the sidecar on the wrong thread while the session stayed registered under the requested id. All three sites (create-with-resume, not-tracked attach, crash recovery) now reject the mismatch loudly: sidecar torn down + reaped, explicit error frame,tracing::error!, wrong id never adopted.Unit pins (RED before fix, GREEN after):
handle_create_with_resume_wrong_thread_id_fails_create_and_never_adoptshandle_attach_unknown_session_wrong_thread_id_is_rejected_not_adoptedcrash_recovery_resume_wrong_thread_id_is_rejected_not_silently_recoveredRoot cause found in the pinned e2e itself: restore-matrix SCENARIO 7's blanket
test.fail()was an artifact of a TEST BUG, not product behavior —originalSessionIdwas assigned the void result ofexpect.poll(...).toBe(...), so the final assertion compared the leaf id againstundefinedand could never pass on either server kind. With the capture fixed, the pane-level contract (thread-B never rendered/adopted) holds on BOTH kinds; pin removed, frozen-legacy internal gap (no adapter-level guard, unobservable at this UI level) documented in the spec header. Checklist TERM-25 evidence updated accordingly.2. Compound-restart coverage (
bd94023c)Closes the two never-tested disruption modes from
docs/plans/2026-07-19-state-sync-resilience-assessment.md§7 (newtest/e2e-browser/specs/compound-restart-rust.spec.ts, rust-chromium):sessionRef,resume <id>argv re-applied (argv-log delta), pane re-anchored + non-blank, sidebar row linked (data-has-tab="true"),terminal_identity_unresolvednever fires.page.reload()interrupting the in-flight restore; same contract plus an idempotency re-check (second reload doesn't revert recovery).New fixture capability:
RustServer.restartAbrupt()(SIGKILL + reboot on same home/port/token; fixture-side ownership-safe descendant sweep is the primary PTY reap in this mode). Both modes are GREEN — the current product handles both compounds; the specs now pin that permanently. Assertion shapes inheritcodex-terminal-bounce-rust.spec.ts's proven-discriminating pattern (RED against the real 2026-07-22 incident).3. restore-matrix scenario 3 + multi-client flake (
4b992f8d, docs)8fd9233a; re-verified GREEN at this base (opening a seeded historical session…, rust-chromium). Doc annotated RESOLVED with evidence.031a7c12(over-constrained exact-attach-intent assertion, relaxed with in-spec evidence).multi-client.spec.tsgreen 3x on BOTH kinds, including one run under concurrent cargo-build load. Doc annotated with evidence.4. OpenCode restore gap check (verification record — no gaps found)
Compared
docs/plans/2026-07-18-opencode-terminal-restore-spec.mdslices A–D against the implementation:crates/freshell-sessions/src/opencode_locator.rs): present, 18 unit tests covering spec test-list items 1–16 including BOTH row-timing shapes (row_created_at_spawn_before_any_enter_resolves_via_spawn_window,row_created_lazily_at_first_enter_resolves_via_enter_window), ambiguity refuse, idle short-circuit (tick_while_unarmed_performs_zero_db_scans), missing-DB tolerance.crates/freshell-ws/src/opencode_association.rs): present, 6 tests covering spec items 17–20 (submit recognition, arm/skip rules, drain+broadcast on a real seeded DB row).WsState.opencode_locator(lib.rs:219,669), submit/arm hooks interminal.rs, construction + sweep inmain.rs:335-508,list_sessions_sinceinparse/opencode.rs:348.opencode-terminal-restore-rust.spec.tsproves restart restore viaopencode --session <id>argv AND fresh restore for a never-submitted pane.Conclusion: all slices implemented with the spec's full test list; no real gaps to close.
Verification
cargo fmt --all --checkGREEN (incl. one pre-existing whitespace fix,6f4aa9fb);cargo clippy --workspace --exclude freshell-tauri --all-targetsexit 0 (warnings pre-existing at base, none in changed code);cargo test --workspace --exclude freshell-tauriall green (freshell-freshagent 240/240).restore-matrixfull suite 20/20 (both projects) 2x;compound-restart-rust2/2 3x;multi-client12/12 3x (both kinds, incl. under load);codex-terminal-bounce-rustgreen (touched-helper sanity).FRESHELL_TEST_SUMMARY="track2 restore hardening pre-PR" npm test→full-suite success exit=0at6f4aa9fb(dirty:0).🤖 Generated with Amplifier