Publish waiting state after pending input exists

docs/logs/engineering-log.md

@@ -129,6 +129,110 @@
   complete `cmd/harnessd` normal/race suites passed; the repository regression
   gate passed normal, race, and coverage at 85.6% with zero uncovered functions.
 
+## 2026-07-30 — Issue #1054 wait state precedes pending input
+
+- Symptom: Hosted race execution observed `waiting_for_user`, then
+  `PendingInput` returned `no pending input`.
+- Cause: `runner_step_engine.go` publishes status/event before invoking the
+  AskUserQuestion tool; broker registration happens later inside the handler.
+- Impact: Event-driven TUI/macOS clients can render a wait state with no
+  question available to display or submit.
+- Intended fix: Let each broker notify after its pending state is
+  readable/durable, forward that notification through the core tool, and
+  publish the runner wait state from that point.
+- TDD evidence: A gated broker made registration impossible while the tool was
+  entered; pre-fix status was already `waiting_for_user`, proving the gap. The
+  test now keeps status `running` until registration, then requires both
+  readable pending input and `waiting_for_user`.
+- Implementation: `AskUserQuestionRequest.OnPending` is a typed,
+  post-registration notifier. Both built-in brokers start it exactly once with
+  the question's deadline context; the core tool forwards it from context; the
+  runner uses it to publish status and the existing event without polling.
+- Verification: Focused AskUser/wait suites passed 100 normal and 100 race
+  repetitions; complete harness normal/race suites passed; repository normal,
+  race, and coverage gates passed at 85.6% with zero uncovered functions.
+- Review follow-up: Exact-head Codex review correctly noted that both brokers
+  computed `DeadlineAt` before `OnPending` but started their timeout afterward.
+  Regressions that held notification beyond the deadline failed on both
+  backends, then passed after the timer/context moved before notification.
+  These deadline tests passed 10 normal and 10 race repetitions; complete
+  harness normal/race and repository normal/race/coverage gates passed again.
+- Second review follow-up: Starting the clock was insufficient because a
+  notifier that never returned still prevented `Ask` from selecting the
+  expired timer. Strengthened regressions kept both notifiers blocked while
+  requiring `Ask` to return its timeout. Brokers now run notification
+  independently with the same deadline context, while answer/cancel/timeout
+  selection continues immediately; the runner checks that context before
+  status and event publication.
+- Third review follow-up: Letting answer selection race notification created
+  the opposite ordering bug: a quick submission could emit `run.resumed` while
+  waiting-state persistence was still blocked, then cancel the notifier before
+  `run.waiting_for_user`. A deterministic blocking-store regression reproduced
+  the reversed event order. Brokers now wait for notification completion before
+  consuming a buffered answer, while the same deadline remains independently
+  enforceable if notification stalls.
+- Fourth review follow-up: Deadline resolution still exposed two durable-state
+  races. A timely checkpoint answer could be overwritten as expired, and a
+  blocked stale `waiting_for_user` write could land after the terminal failure
+  write. Checkpoint resolution is now serialized and `ExpirePending` never
+  replaces an accepted result; run status persistence uses a monotonic in-memory
+  version and rewrites the latest state after any stale write completes.
+  Deterministic regressions cover both races, followed by the full normal,
+  race, and coverage gate at 85.6% with zero uncovered functions.
+- Fifth review follow-up: The in-memory broker was not symmetric with the
+  checkpoint broker when a timely answer was buffered while notification hit
+  its deadline, and a losing checkpoint resume still returned false success.
+  In-memory submission now publishes the buffered answer before removing the
+  pending entry, deadline cleanup returns any accepted answer, and checkpoint
+  resolution returns exported `ErrAlreadyResolved` when another terminal
+  transition already won. Focused normal/race stress covers both contracts.
+- Sixth review follow-up: The ordinary checkpoint wait deadline still bypassed
+  accepted-answer recovery, and the new sentinel escaped as HTTP 500 through
+  run input, approval/deny, and generic checkpoint resume paths. Both AskUser
+  deadline branches now share one pending-only expiry/recovery function;
+  broker/runner boundaries normalize lost races to their existing no-pending
+  contracts; and generic resume returns stable `409 already_resolved` without
+  changing status, payload, or update time. Deterministic gates cover accepted
+  resume-before-notify, approval/deny expiry races, repeated resume, and both
+  API error shapes without unbounded channel receives.
+- Seventh review follow-up: Accepted-answer recovery at the notifier deadline
+  returned before the blocked pending-state publication completed, so
+  `run.resumed` could still overtake `run.waiting_for_user`. Deterministic
+  regressions now require both brokers to retain the accepted answer without
+  returning it until pending publication finishes. Unresolved timeout paths
+  remain independent, while accepted answers wait on notification completion
+  with the parent context as the cancellation escape hatch.
+- Eighth review follow-up: A broader concurrency review found four remaining
+  ownership gaps. The built-in notifier used a background store context;
+  checkpoint resolution was process-local and service-wide; stale run writes
+  still depended on a fallible corrective retry; and third-party brokers could
+  omit `OnPending`. New deterministic reds pinned each failure. Status writes
+  are now serialized per run and snapshot after a context-aware lock;
+  notification passes its deadline through status and event persistence;
+  checkpoint stores expose atomic pending-only resolution with per-record,
+  context-aware service coordination and cross-service waiter observation; and
+  the runner observes readable broker pending state as a callback fallback.
+  Both callback and fallback paths share exactly-once wait publication.
+  Status mutation, persistence, and its lifecycle event share the per-run lock;
+  terminal state rejects any delayed nonterminal downgrade, so a notifier
+  cannot publish stale waiting state after completion, failure, or cancellation.
+- Ninth review follow-up: Pending publication still used once-on-attempt and
+  the fallback observer cancelled its context as soon as the tool returned.
+  Immediate `UpdateRun` or `AppendEvent` failures could therefore consume the
+  only publication attempt, while a quick accepted answer could cancel an
+  observer already persisting the wait. The callback and observer now share a
+  serialized once-on-success publisher; started observer publication drains to
+  success or the question deadline, and transient failures retry. Strict
+  durable-before-visible event behavior is limited to this waiting lifecycle;
+  ordinary nonterminal events preserve the existing best-effort persistence
+  contract. Failed strict appends roll back the final sequence allocation, so
+  run SSE IDs remain contiguous and `Last-Event-ID` reconnect returns only
+  unseen events. A redaction-policy drop counts as successful suppression and
+  cannot cause retries or block an accepted answer. Cross-Service checkpoint
+  polling is opportunistic after local waiter registration: a transient poll
+  read error is retried instead of unregistering the waiter or masking a later
+  local/remote resolution; the caller context remains the termination bound.
+
 ## 2026-07-30 (Workflow Failure-Event Test Timeout — Issue #1049)
 
 - Symptom: the full race gate reached a stored failed workflow state but timed

docs/logs/long-term-thinking-log.md

@@ -108,6 +108,42 @@
 - Next verification step: Make the direct signal expectation fail first, emit
   it at the provider factory boundary, then run focused and full gates.
 
+## 2026-07-30 (Issue #1054 Waiting/Pending Atomicity)
+
+- Command intent: Fix the exact hosted lifecycle failure blocking the
+  cron/callback repair chain.
+- User intent: Harness and GUI state must agree in real time, not merely pass
+  source-level tests.
+- Success definition:
+  - `PendingInput` succeeds whenever `waiting_for_user` is observable.
+  - Both in-memory and durable checkpoint brokers uphold the invariant.
+  - A resume accepted before a deadline remains accepted even when persistence
+    or pending notification completes after that deadline.
+  - An accepted answer cannot become a resumed run before pending-state
+    publication finishes, including when the notifier deadline wins selection.
+  - Lost resume/approval/deny races return stable no-pending or conflict
+    semantics at the harness and HTTP boundaries, never false success or 500.
+  - Resolution remains single-winner across Service instances sharing a
+    durable store, without serializing unrelated checkpoints or ignoring a
+    waiting caller's context.
+  - Pending publication honors its deadline through persistence, stale run
+    writes cannot overwrite terminal state, and callback-omitting brokers still
+    produce one visible wait/resume lifecycle after exposing readable pending
+    input.
+  - Pending publication is once-on-success across callback and observer:
+    transient status/event failures retry, an observer already publishing is
+    drained, deliberate redaction suppression completes without retry, and
+    failed strict appends cannot create an SSE cursor gap.
+  - Cross-Service waiter polling tolerates transient reads after registration;
+    local notification, a later durable read, or caller cancellation decides
+    the result rather than a single opportunistic poll failure.
+  - Event ordering, cancellation, timeout, and denied-call restoration remain
+    correct and race-clean.
+  - Full repository verification and final native GUI conversation proof pass.
+- Non-goals: Redesigning structured questions or approval behavior.
+- Next verification step: Add a deterministic gated-broker regression, confirm
+  the ordering bug, then introduce post-registration notification.
+
 ## 2026-07-30 (Workflow Failure-Event Test Timeout — Issue #1049)
 
 - Command intent: clear the exact full-gate timeout blocking the verified

docs/plans/2026-07-30-issue-1054-waiting-pending-order-impact-map.md

@@ -0,0 +1,136 @@
+# Cross-Surface Impact Map: Waiting-for-user pending-input ordering
+
+## Task
+
+- Task / issue: #1054
+- Plan link: `2026-07-30-issue-1054-waiting-pending-order-plan.md`
+- Owner: Codex
+- Status: Implemented; promotion pending
+
+## Current Ownership, Callers, and Data Flow
+
+- Entry points: Runner tool-call lifecycle and `AskUserQuestionTool`.
+- Owning packages/types/functions and source of truth:
+  `runner_step_engine.go` owns run status/events; in-memory and checkpoint
+  brokers own pending question state.
+- Callers, consumers, events, and downstream data: HTTP/TUI/macOS clients read
+  run status/events and then call `PendingInput`.
+- Similar abstractions searched: Approval brokers, `PendingInput`,
+  `EventRunWaitingForUser`, AskUserQuestion broker implementations.
+- Search commands/evidence:
+  `rg -n "RunStatusWaitingForUser|PendingInput|AskUserQuestionBroker" internal/harness`.
+- Duplication/ownership conclusion: Registration stays broker-owned; status and
+  event publication stays runner-owned, joined by a typed post-registration
+  callback.
+
+## Config, API, CLI, and Tools
+
+- User-facing config added or changed: None.
+- Defaults / fallbacks: None.
+- Environment variables, config files, or saved settings touched: None.
+- Endpoints, request fields, response fields, or server wiring affected:
+  Existing pending-input endpoint becomes immediately consistent with visible
+  wait state; wire shape unchanged.
+- CLI commands, tools, wire formats, or integrations affected:
+  AskUserQuestion internal request gains an optional readiness callback.
+- Error states / validation changes: Removes transient `ErrNoPendingInput`
+  after visible wait state.
+
+## Persistence and Compatibility
+
+- Schemas, migrations, caches, generated data, or ownership changes: None.
+- Backward/forward compatibility and versioning: Internal Go API addition;
+  existing question/checkpoint records unchanged.
+- Partial rollout and mixed-version behavior: Single binary; no mixed-version
+  protocol.
+
+## Lifecycle, Security, and Reliability
+
+- Concurrency, cancellation, retries, cleanup, and resource ownership:
+  Notification starts exactly once after successful registration in a
+  deadline-bound goroutine. The broker does not consume a buffered answer until
+  notification finishes, which preserves wait-before-resume ordering, while
+  cancellation and unresolved timeout remain independent so a stalled notifier
+  cannot hang `Ask`. If deadline selection recovers an already-accepted answer,
+  it waits for notification completion or parent cancellation before returning
+  so `run.resumed` cannot overtake pending-state publication. The callback must
+  propagate its supplied context through blocking work. Built-in status/event
+  persistence does so; run-status mutation, persistence, and the matching event
+  serialize per run after acquiring that context-aware lock. Terminal state
+  rejects delayed nonterminal downgrades. A runner-side pending observer invokes
+  the same exactly-once notifier when a broker exposes `Pending` but omits the
+  callback.
+- Authentication, authorization, permissions, trust, privacy, and secrets:
+  No new data exposure; callback receives the already-public pending shape.
+- Failure modes, recovery, idempotency, and data repair: Registration failure
+  does not publish wait state; timeout/cancel continue through existing paths.
+  Checkpoint expiry is conditional on unresolved state, and stale run-status
+  writes cannot land after newer state because persistence is serialized per
+  run rather than repaired afterward.
+  Accepted in-memory answers win deadline cleanup, and callers receive an
+  explicit already-resolved error when a checkpoint transition loses a race.
+  AskUser notification and ordinary wait deadlines share the same atomic
+  pending-only expiry and accepted-resume recovery path. Harness input and
+  approval operations translate a lost resolution to their established
+  no-pending result; generic checkpoint HTTP resume exposes the distinct
+  durable conflict as `409 already_resolved` while preserving terminal data.
+  Memory and SQLite checkpoint stores perform atomic pending-only resolution;
+  per-record service locks honor caller cancellation, unrelated records proceed
+  independently, and bounded store observation wakes waiters after a different
+  Service instance resolves the record.
+
+## Product and Integration Surfaces
+
+- Server/runtime: Strengthened lifecycle invariant.
+- TUI/web/macOS/other clients: Wait event/status can be rendered without a
+  retry gap before the question is readable.
+- Provider/model/tool catalog and routing: AskUserQuestion core tool forwards
+  the readiness hook; no catalog changes.
+- External systems and automation: Hosted race suite becomes deterministic.
+- UX states, keyboard/focus/accessibility/motion: Eliminates transient empty
+  input UI; no visual design change.
+
+## Deployment and Operations
+
+- Deployment/migration order and feature flags: None.
+- Logs, metrics, traces, alerts, and support diagnostics: Event payload/order
+  retained.
+- Rollback triggers and recovery steps: Revert if exact-head lifecycle tests or
+  manual question flow regress.
+- Runbooks and operator docs: None.
+
+## Regression Tests
+
+- Characterization and first expected red test: Gated broker proves visible wait
+  state precedes broker registration on current code.
+- New acceptance tests required: Both brokers expose `Pending` inside readiness
+  callback; tool forwards callback; runner invariant.
+- Edge, negative, failure, lifecycle, and security tests: Registration error,
+  timeout, cancellation, denied tool, exactly-once notification, and accepted
+  answer recovery while pending publication is blocked past its deadline;
+  callback deadline propagation; stale-write repair failure; callback omission;
+  per-record cancellation; unrelated resolution; cross-Service single-winner
+  resolution and waiter visibility; once-on-success retry for immediate status
+  and event persistence failures; observer drain after accepted input;
+  redaction-dropped waiting events; ordinary-event best-effort persistence; and
+  contiguous run SSE IDs with duplicate-free `Last-Event-ID` replay after a
+  failed strict append; and waiter survival across a transient cross-Service
+  polling read failure before later resolution.
+- Integration/e2e/real-path proof: Exact-head hosted normal/race checks and
+  final native GUI conversation test.
+- Cross-surface regressions to guard: Existing event order and status
+  restoration suites.
+- Exact targeted and full commands:
+  `go test ./internal/harness ./internal/harness/tools/core -run 'AskUser|WaitForUser' -count=100`;
+  same with `-race`; `./scripts/test-regression.sh`.
+
+## Documentation and Handoff
+
+- Specs/public docs before code: Plan and impact map.
+- Implementation notes/logs/indexes after code: Plan index, active plan,
+  engineering log, long-term-thinking log.
+- Training/onboarding/release notes: None; existing public contract is tightened.
+
+## Warning Check
+
+- Every relevant surface is mapped or explicitly unaffected.