Fix terminal run publication atomicity

docs/logs/engineering-log.md

@@ -66,6 +66,148 @@
   `./scripts/test-regression.sh` passed at 85.6% total coverage with zero
   uncovered functions.
 
+## 2026-07-31 (Terminal Status/Event Atomicity — Issue #1067)
+
+- Symptom: aggregate race load exposed `RunStatusFailed` from `GetRun` while
+  the same run's immediate replay ended at `llm.turn.requested` without
+  `run.failed`; code inspection found the same status-first window on completed
+  and cancelled paths.
+- Cause: every terminal helper called `setStatus` before `emit`, splitting the
+  public run record from the event journal's ledger, bounded store append,
+  subscriber fanout, and recorder drain.
+- Deterministic red: a no-sleep transition barrier reproduced all three states.
+  Completed replay lacked `run.completed`, failed replay contained the required
+  `error.context` but lacked `run.failed`, and cancelled replay lacked
+  `run.cancelled` while `GetRun` already returned each terminal status.
+- Fix: one `transitionTerminal` seam now lets the winning terminal emit seal and
+  append the matching event, completes bounded store append and ordered recorder
+  dispatch/drain, conditionally persists the matching status, commits in-memory
+  status, then fans out. Every status transition shares a per-run mutex, so a
+  delayed running/waiting snapshot cannot overwrite terminal state.
+- Preserved reliability: terminal store I/O remains outside `Runner.mu`, and
+  status-store I/O remains outside the global conversation journal lock;
+  a refcounted per-conversation sequence guard prevents same-conversation
+  overtaking while unrelated `GetRun` and unrelated event journals stay
+  responsive, then reclaims idle keys. Terminal redaction sealing, event IDs,
+  causal/error snapshot order, recorder order, backup, and pruning contracts
+  remain intact.
+- Failure policy: retained terminal status persistence is never attempted when
+  terminal append reports failure. If append succeeds but final status update
+  errors or reaches its context deadline, durable status may remain
+  non-terminal while the durable event, in-memory terminal state, and subscriber
+  fanout proceed. This is the strongest one-way guarantee available without a
+  store transaction and does not claim two-way atomicity.
+- Explicit exception: existing terminal `StorageModeNone` configurations still
+  suppress the matching replay event while sealing and publishing status, as
+  pinned by terminal-redaction tests. The stronger replay implication applies
+  to terminal events retained by policy.
+- Review regressions: append failure cannot persist terminal status; status
+  update failure/timeout still completes live publication; unrelated
+  conversations progress during terminal I/O while target events cannot
+  overtake; delayed non-terminal status cannot overwrite terminal; explicit
+  terminal redaction waits for recorder drain; and contended/distinct keyed
+  locks reclaim to zero.
+- Exact-head retention cause: the terminal event success marker made a run
+  eligible for pruning even when the matching final `UpdateRun` failed, because
+  that return value was discarded. The truthful in-memory terminal state could
+  be evicted while the durable row remained running. Both append- and
+  status-failure exceptions could also accumulate without an admission bound.
+- Retention/admission fix: terminal event resolution and terminal status
+  persistence are tracked separately. Store-backed pruning requires both;
+  `StorageModeNone` is explicit event suppression plus durable status, while
+  no-store runs remain process-local. Both unresolved append and status states
+  count toward `MaxCompletedRetention`. At the cap, Start/Continue retry only
+  status gaps under one shared deadline of at most 250 ms and otherwise return
+  `TerminalDurabilityBackpressureError`; their HTTP routes map it to 503
+  `terminal_durability_unavailable`.
+- Recovery and boundedness: no recovery store I/O holds Runner, status,
+  event-journal, or conversation locks. Status overwrite retries are safe and
+  immediately restore the completed-retention bound before reopening admission
+  once acknowledged. Ambiguous failed appends are protected but never retried
+  because a third-party store may have applied the append.
+  Already-admitted work finishes and remains visible; admission closes at the
+  cap so permanent failure growth stops at that finite admitted population.
+- Exact-head regressions: retention 1 preserves several already-admitted
+  UpdateRun-failed completions while durable rows stay non-terminal; concurrent
+  admissions reject during outage and recover under race; one blocked retry
+  proves the shared deadline and unlocked state/journal access; append failure,
+  StorageModeNone, no-store, Continue error precedence, and both HTTP 503 routes
+  are pinned.
+- Concurrent review red: a phase hook paused Continue immediately after its
+  completed-source validation. A concurrent Start recovered three pending
+  statuses, pruned the oldest validated source at retention 1, and the resumed
+  Continue deterministically failed with `run not found`.
+- Concurrent review fix: validation now increments an in-state continuation
+  reservation under `Runner.mu`. The one shared completed-run prune candidate
+  filter excludes reserved sources for every caller. A defer releases on every
+  success/error path and immediately re-prunes; the existing later write-lock
+  `continued` check still chooses exactly one continuation winner. Recovery
+  store I/O remains outside Runner/status/journal/conversation locks.
+- Gate-test correction: full race exposed one test treating terminal replay as
+  proof status had already committed. The one-way contract is the reverse:
+  terminal status implies replay, while replay may lead status. The test now
+  waits independently for failed status and retains both replay/status checks.
+- Verification: the concurrent red is green with cleanup and single-winner
+  controls normal/race at `-count=100`; the expanded durability harness and HTTP
+  suites pass normal/race at `-count=100`; complete `internal/harness` plus
+  `internal/server` normal/race and affected `go vet` pass. The final direct
+  foreground non-TTY `./scripts/test-regression.sh` passes normal, full race,
+  and `coveragegate: PASS (total=85.7%, min=80.0%, zero-functions=0)`.
+- Hosted settled-helper symptom: exact rebased race run `30656467482` failed
+  `TestRunnerHookErrorFailOpen` because `collectRunEvents` returned terminal
+  replay history before the valid later status commit; the immediate `GetRun`
+  still reported `running`.
+- Audit/cause: 215 `collectRunEvents` references, its sole configurable-timeout
+  caller, and 79 `collectEvents` snapshot references were reviewed. No shared
+  collector caller intentionally observes the event-leading-status window.
+  Exact ordering regressions use direct phase hooks/subscriptions; snapshot
+  consumers either wait separately or intentionally inspect nonterminal state.
+- Deterministic red/fix: a pre-status terminal barrier made replay visible and
+  failed the old collector immediately with `returned before terminal status
+  commit`. Both collector variants now preserve event assertions and then poll
+  for any terminal status within the same total deadline. A missing status is a
+  timeout failure rather than a false settled result; production order is
+  unchanged.
+- Hosted-equivalent verification: the collector, all hook scenarios, and the
+  configurable-timeout caller pass normal/race at `-count=100`; `make
+  test-race` passes. The final outside-sandbox foreground
+  `TMPDIR=/private/tmp GOCACHE=/private/tmp/gocode-go-cache
+  ./scripts/test-regression.sh` passes normal, full race, and
+  `coveragegate: PASS (total=85.7%, min=80.0%, zero-functions=0)`. GitHub
+  comment publication was blocked by external-write safety review and was not
+  retried; this repository evidence records the run.
+- Exact-head helper review: commit `8757e8a3` still let settlement succeed when
+  a closed stream had no terminal event, or when the sole terminal event did
+  not match the later terminal status. This was a P1 test-integrity defect, not
+  a production-path defect: it could mask the exact #1067 invariant across the
+  shared collector callers.
+- Review TDD reds: a completed run plus closed stream containing only
+  `run.started` returned success, and failed status plus `run.completed`
+  returned success. Both failures were immediate and deterministic.
+- Review fix: the test-only subscribed-stream core now requires exactly one
+  terminal event and requires its completed/failed/cancelled meaning to match
+  the observed terminal status. Event slices remain unchanged on success and
+  error, and both event collection and status settlement consume the original
+  single deadline. The phase regression now waits for an explicit settlement
+  entry signal before proving the collector cannot return ahead of status.
+- Review verification: the two regressions, explicit settlement barrier, hook
+  family, and configurable-timeout caller pass normal/race at `-count=100`.
+  Outside-sandbox `make test-race` passes. The authoritative foreground
+  `TMPDIR=/private/tmp GOCACHE=/private/tmp/gocode-go-cache
+  ./scripts/test-regression.sh` passes normal, full race, and
+  `coveragegate: PASS (total=85.7%, min=80.0%, zero-functions=0)` on this exact
+  follow-up diff.
+- Promotion integration regression: the semantic merge with #1054 initially
+  persisted every non-terminal status before committing it in memory. A failed
+  best-effort `UpdateRun` therefore left an executing run visibly stale; the
+  AskUser broker could own a live pending question while API, TUI, and GUI
+  still saw `running` instead of `waiting_for_user`. A deterministic failing
+  store test reproduced `queued` after a requested transition to `running`.
+  The unified per-run status lock now commits live non-terminal state first,
+  retaining the persistence attempt and false return so strict
+  waiting-status/event publication can retry without making the pending prompt
+  invisible. Terminal transitions retain their event-before-status contract.
+
 ## 2026-07-31 (Provider-Key Matrix Health Wait — Issue #1062)
 
 - Symptom: hosted race run `30583930460` failed

docs/logs/long-term-thinking-log.md

@@ -56,6 +56,64 @@
   repository regression gate pass. The local commit is ready for parent
   promotion; PR #1069 remains open and unmerged pending hosted reruns.
 
+## 2026-07-31 (Terminal Status/Event Atomicity — Issue #1067)
+
+- Command intent: repair the aggregate-load Runner race where `GetRun` exposes
+  completed, failed, or cancelled before the matching terminal event and
+  required causal evidence are replayable.
+- User intent: external monitoring must never report a terminal result from an
+  incomplete transcript, and this engine repair must remain isolated from PR
+  #1060/#1055.
+- Success definition: deterministic no-sleep red evidence covers all terminal
+  statuses; one shared lifecycle seam makes terminal ledger/store publication
+  precede status visibility while preserving recorder order; immediate
+  Subscribe and HTTP SSE replay agree; focused stress, affected race/vet, full
+  regression, and hosted checks pass on one unmerged closing PR. Concurrent
+  Start recovery/pruning must not evict a completed source after Continue has
+  validated it but before that continuation performs its single-winner commit.
+  Test helpers that promise a collected terminal run must preserve event
+  assertions while independently requiring the later terminal status, without
+  weakening the production event-first order or direct barrier regressions.
+- Non-goals: conversation cursor redesign, cron/callback behavior, client UI,
+  provider routing, schemas, or workflow timing changes.
+- Guardrails: preserve out-of-lock bounded store writes, terminal sealing,
+  recorder drain order, cleanup order, causal/error snapshots, SSE IDs, and
+  unrelated conversation responsiveness. Do not claim a cross-record store
+  transaction the interface cannot provide. Never evict truthful terminal state
+  without both event and status durability, and never trade that protection for
+  unbounded memory growth during permanent store failure.
+- Outcome: one winner-only transition now publishes terminal ledger history
+  before matching in-memory status and subscriber fanout. Retained terminal
+  status persistence is attempted only after event append reports success;
+  append/status failures have explicit bounded live-availability behavior.
+  Shared per-run status serialization prevents a delayed non-terminal overwrite,
+  and refcounted per-conversation sequencing avoids both overtaking and lock-map
+  growth. Exact-head retention hardening now tracks event and status durability
+  separately, requires both before pruning, and closes new admission when the
+  combined unresolved backlog reaches `MaxCompletedRetention`. Status recovery
+  uses one unlocked deadline capped at 250 ms; Start/Continue expose typed HTTP
+  503 while no-store and intentional StorageModeNone policies remain distinct.
+  Successful status recovery immediately restores the retention bound before
+  reopening admission. A temporary in-state reservation now protects a
+  validated Continue source from every prune caller across unlocked recovery;
+  release on all exits restores the retention policy without weakening the
+  existing single-winner check. Focused normal/race and real HTTP mapping tests
+  plus affected normal/race/vet are green. The unchanged foreground repository
+  gate passes normal, full race, and 85.7% coverage with zero uncovered
+  production functions on the prior follow-up diff. After hosted race run
+  `30656467482`, the shared test collector now treats terminal history plus the
+  later terminal status as its settled boundary under one total deadline. The
+  event-first production contract and direct phase tests remain unchanged;
+  affected normal/race x100 and hosted-equivalent `make test-race` pass. The
+  final outside-sandbox foreground repository gate passes normal, full race,
+  and 85.7% coverage with zero uncovered production functions. Exact-head
+  review then hardened the helper boundary again: settlement now requires
+  exactly one terminal event whose meaning matches terminal status, and the
+  phase regression proves it entered settlement before asserting non-return.
+  Focused normal/race x100, hosted-equivalent race, and the final foreground
+  repository gate pass; coverage remains 85.7% with zero uncovered production
+  functions on this follow-up diff.
+
 ## 2026-07-31 (Provider-Key Matrix Health Wait — Issue #1062)
 
 - Command intent: isolate and clear the hosted race blocker first observed on

docs/logs/observational-log.md

@@ -45,6 +45,70 @@ Use this file for observations about system behavior without immediately prescri
   calls before invoking `Shutdown`; otherwise cleanup can wait on the very run
   the fixture still holds blocked.
 
+## 2026-07-31 (Terminal Run Publication Window)
+
+- Concurrency observation: a terminal event can be prepared under the Runner
+  lock and persisted outside it without blocking unrelated run queries, but the
+  matching status needs one explicit commit point between persistence and
+  subscriber fanout. Status before preparation yields incomplete replay;
+  status after fanout lets terminal-event consumers briefly read `running`.
+- Testing observation: a phase channel at the terminal transition boundary
+  deterministically exposes the forbidden state for completed, failed, and
+  cancelled paths without relying on aggregate load or fixed sleeps.
+- Replay observation: after an HTTP status poll returns terminal, reconnecting
+  run SSE from the first event ID must replay exactly one matching terminal
+  event and no terminal event of another status.
+- Conversation-stream observation: terminal persistence and terminal fanout
+  must share the same per-conversation sequence; the global journal lock can be
+  released for slow recorder/status persistence only if later events and
+  subscriptions on that conversation cannot overtake the terminal event.
+- Durability observation: the current store interface has separate
+  `AppendEvent` and `UpdateRun` calls, so it cannot promise two-way atomicity.
+  The enforceable direction is: never attempt retained terminal status
+  persistence after terminal append reports failure. If status update then
+  fails or times out, durable event may lead durable status while bounded
+  in-memory status and fanout still complete.
+- Resource observation: a keyed sequence lock needs waiter-inclusive reference
+  accounting. Deleting only after owners and queued waiters release prevents a
+  second lock generation for the same key and avoids an unbounded conversation
+  map.
+- Retention observation: event append success alone is insufficient proof that
+  a terminal state can fall back to the store. Pruning also needs acknowledged
+  terminal status persistence; otherwise fallback resurrects a non-terminal
+  durable row after evicting the only truthful process-local result.
+- Outage observation: protecting unpersisted truth and bounding memory require
+  one admission boundary. Both ambiguous event appends and failed status
+  updates consume it; already-admitted runs may finish above the numeric cap,
+  but later admissions stop growth once the outage is observed.
+- Recovery observation: `UpdateRun` is an idempotent overwrite and can be
+  retried under one shared short context. `AppendEvent` is not safe to retry
+  after an ambiguous third-party error because the append may already exist.
+  Once status retries succeed, pruning newly durable candidates immediately
+  restores the retention bound before another admission is accepted.
+- Policy observation: no-store and `StorageModeNone` are distinct. No-store has
+  no durable fallback and stays process-local; StorageModeNone intentionally
+  resolves the event side while its final status can still make safe pruning
+  possible.
+- Continuation observation: preserving a source only in Continue's own recovery
+  prune is insufficient because concurrent Start recovery calls the same prune
+  policy without that local argument. A reservation stored on the source and
+  checked by the shared candidate filter protects it across every prune caller.
+- Contract observation: terminal replay can lead the later status commit during
+  event-first publication. Tests that assert both must wait independently for
+  status; only terminal status is guaranteed to imply matching replay.
+- Helper-audit observation: aggregate race load repeatedly finds stale tests
+  when a helper named as event collection is treated implicitly as run
+  settlement. Shared callers all want settlement, while intentional window
+  probes use direct `Subscribe`; encoding the distinction once in the test
+  helper prevents the next immediate-`GetRun` variant without changing replay.
+- Settlement observation: waiting for any terminal status is insufficient if
+  the collected transcript is absent or contradictory. A settled test result
+  requires exactly one terminal event whose completed/failed/cancelled meaning
+  matches status; stream closure is not evidence of transcript completeness.
+- Synchronization observation: a timed non-return assertion is meaningful only
+  after the tested goroutine proves it reached the intended blocking phase. An
+  explicit settlement-entry handshake removes scheduler delay as a false pass.
+
 ## 2026-07-31 (Source-Workflow Dual-Error Arbitration)
 
 - Process observation: a child can exit non-zero while closing its stdin also