[Bug]: TUI drops top-level run_id and cannot answer waiting runs

[dennisonbertram]

Work type

Bug / regression

Observed behavior

On exact main commit fedcf6073135deb7cce1fa49921aa698a9cc7cd7, a real multi-message native TUI run emitted run.waiting_for_user. GET /v1/runs/{id}/input returned the correct pending AskUserQuestion, but the TUI did not open its visible answer overlay. Posting an answer through the documented input API unblocked the same run, persisted the assistant continuation, and allowed a later chat turn.

Independent disposable reproduction through the production TUI SSE bridge/model:

TMPDIR=/private/tmp GOCACHE=/private/tmp/gocode-repro-cache go run /private/tmp/gocode_waiting_repro.go
event=run.waiting_for_user raw={"call_id":"call-1"} overlay_active=false

The input event used the live wire shape:

{"id":"run-live-shape:7","run_id":"run-live-shape","type":"run.waiting_for_user","payload":{"call_id":"call-1"}}

Reproduction rate: deterministic.

Expected behavior

When a current or replayed run.waiting_for_user event reaches the TUI bridge, the TUI must retain the envelope's top-level run_id, fetch the pending question from GET /v1/runs/{id}/input, visibly render the AskUserQuestion overlay, submit the selected answer to POST /v1/runs/{id}/input, dismiss on resume, and continue rendering the same conversation through a later assistant response/terminal event. Reconnect/resume using Last-Event-ID must preserve the same behavior without duplicate run-ID normalization.

Reproduction

Preconditions:

1. Checkout exact fedcf6073135deb7cce1fa49921aa698a9cc7cd7.
2. Deliver a normal run-scoped SSE envelope whose run_id is top-level and whose payload contains call_id only.
3. Have /v1/runs/{id}/input return a valid pending AskUserQuestion.

Exact path:

1. StartSSEBridge decodes the SSE block in cmd/harnesscli/tui/bridge.go.
2. Pass the emitted SSEEventMsg through tui.Model.Update while the matching run is active.
3. Inspect AskUserActive() / rendered view and observed /input requests.

Actual result: decodeSSE discards the envelope's top-level run_id, forwards only env.Payload, the run.waiting_for_user handler fails p.RunID != "", no GET occurs, and no overlay appears.

User and operational impact

Severity: high product-flow failure for any TUI conversation that invokes AskUserQuestion. The server remains correctly paused and durable, but the primary TUI cannot visibly continue it. Users see a stalled conversation and need an undocumented/manual API workaround. No data-loss or credential exposure is known; the answer submitted through the API proves the server broker and persistence remain intact. Fix is urgent because waiting conversations are otherwise unusable from the TUI.

Suspected seam and search evidence

Owning boundary:

• cmd/harnesscli/tui/bridge.go: sseEnvelope, decodeSSE, SSEEventMsg
• cmd/harnesscli/tui/model.go: case "run.waiting_for_user"
• cmd/harnesscli/tui/askuser.go: pending fetch, answer submission, overlay state/rendering

Callers/consumers searched:

• StartSSEBridge*, reconnect/Last-Event-ID paths in bridge.go and api.go
• TUI event switch and existing ask-user tests
• server GET/POST /v1/runs/{id}/input in internal/server/http_runs.go
• server event/replay contract where run_id is an envelope field
• non-TUI ask-user flow and headless blocked-event handling

Commands/evidence:

• rg -n "run.waiting_for_user|AskUserQuestion|PendingInput|run_id|RunID" cmd/harnesscli internal/server internal/harness test/e2e
• direct source inspection of decodeSSE, the model waiting handler, and ask-user commands
• disposable localhost SSE bridge/model reproduction above
• open PR review: PR Publish waiting state after pending input exists #1055 changes server-side pending-input publication order but does not touch any cmd/harnesscli/tui file

Current root cause: sseEnvelope decodes only type and payload. decodeSSE returns only payload bytes, while the waiting handler incorrectly expects run_id nested inside those bytes. Existing tests bypass the bridge and construct SSEEventMsg.Raw with a synthetic nested run_id, so they green-light the wrong wire shape.

Adjacent seams ruled out: the event is delivered; GET /input returns the correct pending object; direct POST resumes and persists; the failure occurs before the TUI issues GET.

Blast-radius impact map

• Callers and data flow: server SSE envelope -> runBridge/decodeSSE -> SSEEventMsg -> Model.Update -> fetchAskUserPendingCmd -> overlay -> submitAskUserAnswerCmd -> run.resumed/later events. Fix the shared envelope-to-message boundary once.
• Config/env/defaults: None; no configuration or default changes.
• API/CLI/wire formats/tools: Preserve existing SSE and /input wire formats. Internal TUI message data gains/uses the already-present envelope run ID. No public endpoint/tool change.
• Persistence/schema/cache: None; no schema/migration/cache ownership changes.
• Concurrency/lifecycle/cancellation/retries: Preserve bridge backpressure, terminal handling, bounded reconnects, Last-Event-ID, deadline timers, and cancellation. Replayed waiting events must fetch the pending input exactly as a live event does.
• Security/auth/permissions/privacy: Preserve bearer authentication on initial/reconnected SSE and GET/POST input requests. Do not log prompt/answer contents or credentials.
• TUI/web/macOS/other clients: TUI only. Web visualization, macOS native GUI, ACP, and non-TUI streaming CLI do not consume this TUI decoder/model seam and are out of scope.
• Provider/model/tool catalog: None; AskUserQuestion schema/tool/provider routing is unchanged.
• Deployment/observability/runbooks: client binary-only behavior change; existing visible overlay and errors are the proof signal. No new metric or service rollout.
• Compatibility: additive internal retention of top-level run ID; live and replayed current server envelopes remain compatible. Payload-only synthetic fixtures must be corrected to the real envelope contract, not supported as a second wire shape unless production evidence requires it.
• Existing tests/fixtures: existing ask-user tests manufacture a nested payload run ID and miss the bridge. Bridge resilience/reconnect and unrelated event decoding must remain green.
• Documentation: required long-term intent, plan, impact map, engineering/observational/system log entries, and plan index/active plan updates. No public user docs change because the documented AskUserQuestion behavior is already intended.

Regression test first

Test file/name: add the smallest acceptance regression under cmd/harnesscli/tui, e.g. TestAskUser_WaitingEnvelopeThroughBridge_ShowsSubmitsAndContinues.

Red command:

go test ./cmd/harnesscli/tui -run TestAskUser_WaitingEnvelopeThroughBridge_ShowsSubmitsAndContinues -count=1

Fixture/flow:

1. Local HTTP server emits an actual SSE run.waiting_for_user envelope with top-level run_id and payload call_id only.
2. Drive the real StartSSEBridge output through the model.
3. Require GET /input, visible question/options in View, Enter submission to POST /input, run.resumed, a later assistant delta/terminal event, and visible continuation.
4. Include a replay/reconnect control using event ID/Last-Event-ID or the existing reconnect fixture boundary so top-level run ID survives both initial and resumed decoding.

Expected red failure: no GET /input, no visible overlay, no answer POST, because the bridge drops the top-level run ID.

Why it proves the bug: it crosses the same production SSE decoder/model/HTTP/visible-view boundaries as the failed TUI path rather than injecting a synthetic post-decode payload.

False-positive controls: assert exact run ID and call ID, exact GET/POST paths, submitted answer body, continued assistant output, and no duplicate pending fetch for a single delivered event.

Fix boundaries

In scope:

• retain the top-level SSE envelope run ID once at the bridge/message boundary;
• make the waiting handler use that canonical value;
• correct synthetic tests to the real wire shape;
• add acceptance/replay coverage and required durable docs/logs.

Out of scope:

• server pending-input ordering owned by [Bug]: waiting_for_user becomes visible before pending input exists #1054/PR Publish waiting state after pending input exists #1055;
• native GUI, web, ACP, callbacks, cron execution, approvals, provider/model/tool schema changes;
• epic [Epic]: Reliable scheduled conversation automation across cron, callbacks, harness, and macOS #1000/[Change]: Prove cron and callback conversation automation end to end #1010 status or completion.

Existing abstraction to repair: sseEnvelope -> SSEEventMsg normalization and the existing ask-user state/HTTP commands. No parallel decoder or new run-ID source of truth is needed.

Diagnostic and observability evidence

Before fix: bridge/model reproduction prints raw={"call_id":"call-1"} overlay_active=false; no GET /input occurs. After fix: acceptance test must observe the GET, visible question/options, exact answer POST, run.resumed, and later assistant output. Test fixtures use only synthetic non-sensitive prompts/answers. No new logging of answers.

Verification plan

• Expected red:
  go test ./cmd/harnesscli/tui -run TestAskUser_WaitingEnvelopeThroughBridge_ShowsSubmitsAndContinues -count=1
• Focused green:
  same command, plus existing ask-user and SSE bridge suites.
• Focused race:
  go test -race ./cmd/harnesscli/tui -run 'AskUser|WaitingEnvelope|SSEBridge' -count=1
• Package normal/race:
  go test ./cmd/harnesscli/tui -count=1
go test -race ./cmd/harnesscli/tui -count=1
• Full required gate:
  ./scripts/test-regression.sh
• Real path: exercise a locally running TUI/SSE run or equivalent realistic localhost acceptance that visibly shows the overlay, submits, resumes, and renders the continuation; inspect server logs after smoke.
• Hosted: GitHub PR checks must complete green before review handoff. Do not merge.

Rollout and rollback

Release with the next harnesscli build; no server ordering dependency is required for the decode fix, though #1054 improves the pending-read race independently. Monitor reports of stalled waiting runs and TUI input-fetch errors. Roll back by reverting the isolated client commit if other SSE event handling or reconnect behavior regresses. No data repair or migration is needed; already waiting runs remain answerable through the existing API and become visible after installing the fixed client/reconnecting.

Documentation and handoff

Before implementation: add a linked issue plan, one-page cross-surface impact map, long-term command/user intent/success definition, update active plan and plan index. After the fix: add engineering log symptom/cause/fix/TDD/verification; observational log wire-shape/testing learning; system log canonical SSE envelope-to-TUI flow. Maintain folder indexes. Public docs are unaffected.

Definition of done

• Deterministic production-shape reproduction recorded
• Contract-complete issue and linked plan/impact map exist before implementation
• Acceptance regression fails before the fix for the expected missing-overlay reason
• Top-level run ID is retained once and used by the waiting handler
• Visible overlay fetches/submits and the conversation continues
• Replay/reconnect behavior is covered without duplicate normalization
• Existing AskUser/SSE, package normal/race, and full regression are green
• GitHub hosted checks are green
• Engineering/observational/system logs and indexes are current
• PR uses Closes #N, exact head SHA is reported, and PR is not merged

Process acknowledgement

I will preserve the failing regression test, update the issue if the root cause changes, and record any newly discovered bug as its own issue.

Review expansion: stale pending-input fetch correlation

PR #1061 review comment 3687057509 exposed a second failure at the same TUI wait boundary. A pending GET /input response is asynchronous, but the model currently correlates it only by run ID and accepts it while AskUser state is inactive. If another client answers while the GET is in flight, run.resumed clears the overlay and the late response resurrects the already-answered question. If a later AskUserQuestion supersedes the first within the same run, the older response can overwrite the newer overlay.

Expanded expected behavior and acceptance criteria:

• Each pending-input fetch result and error is correlated with the exact waiting call ID and a model-owned wait generation.
• A result arriving after run.resumed is discarded and cannot reactivate the overlay.
• A result from a superseded call/generation is discarded and cannot overwrite the current question.
• The normal single-waiter path, top-level envelope run-ID handling, answer submission/continuation, and replay/reconnect behavior remain green.
• Deterministic regressions cover resumed-before-GET-completion and a newer same-run call winning over a late older fetch.

Impact-map expansion:

• Concurrency/lifecycle: run.waiting_for_user creates a unique wait generation. run.resumed invalidates that generation. Pending GET success/error messages may mutate AskUser state only when run ID, waiting call ID, and generation all match the active wait.
• Compatibility: no server/API/event wire change. Correlation is internal TUI state. Duplicate/replayed waiting events continue to fetch current pending input, while any older in-flight result becomes stale.
• Rollback: revert the isolated TUI correlation change together with its tests if lifecycle behavior regresses; no migration or data repair is required.

Additional red tests:

go test ./cmd/harnesscli/tui -run 'TestAskUser_(LatePendingAfterResumeIsDiscarded|SupersededPendingFetchIsDiscarded)' -count=1

The existing out-of-scope list remains unchanged. Native GUI, callbacks, cron execution, other TUI event families, and epic #1000/#1010 are not part of this fix.

[dennisonbertram]

Cheap-agent exact-head audit of PR #1061 at 221ab05 found two P2 product defects and two proof gaps before promotion.

Product defects:

• askUser.active is set before pending questions arrive, but the overlay renders nothing and handleAskUserKey consumes all keys. Esc and both Ctrl+C presses are swallowed until the pending-input request returns or reaches its 10-second timeout. Loading must remain visibly actionable and preserve interrupt/cancel behavior.
• a delayed pending-input response can resurrect the overlay after terminal completion or failure because terminal messages neither clear AskUser state nor advance its generation. Terminal completion/failure must invalidate all outstanding pending-input loads.

Required regressions:

• TestAskUser_LoadingStatePreservesInterruptKeys
• TestAskUser_LatePendingAfterTerminalIsDiscarded
• a production-path two-user-turn test covering prompt -> visible question -> keyboard answer -> resumed assistant -> next user turn -> next assistant, with exact transcript cardinality through reconnect/replay.

Existing focused AskUser tests pass and are behavioral, but they stop after the first synthetic continuation and the reconnect test covers only cursor/envelope mechanics. Promotion remains ordered after #1070, #1055, and #1059; after #1059, preserve both assistantTranscriptFinalized and askUserGeneration when resolving model.go.