Skip to content

Publish waiting state after pending input exists - #1055

Merged
dennisonbertram merged 8 commits into
mainfrom
codex/issue-1054-waiting-pending-order
Jul 31, 2026
Merged

Publish waiting state after pending input exists#1055
dennisonbertram merged 8 commits into
mainfrom
codex/issue-1054-waiting-pending-order

Conversation

@dennisonbertram

@dennisonbertram dennisonbertram commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Closes #1054

Summary

  • publish waiting_for_user only after AskUserQuestion pending input is readable, with deadline-bound notifier work and accepted-answer ordering
  • add atomic pending-only Memory/SQLite checkpoint resolution, per-record context-aware Service coordination, and cross-Service waiter visibility
  • add a Runner Pending observer fallback for third-party brokers that omit OnPending; callback and observer share serialized once-on-success publication
  • serialize run-status persistence, prevent delayed waiting writes from replacing terminal status, and retry transient waiting status/event persistence failures
  • preserve strict durable-before-visible behavior for the waiting lifecycle and contiguous duplicate-free SSE replay

Exact-head verification

Rebased candidate e6e255d8ba60d1ae3035031f1342ae22b0c5d74d onto main fa2f7c8ed6e576006d14ec6ec989ac2d6f1a5e91.

  • go test ./internal/checkpoints ./internal/harness ./internal/server -count=1: PASS
  • go test -race ./internal/checkpoints ./internal/harness ./internal/server -count=1: PASS
  • foreground non-TTY ./scripts/test-regression.sh: PASS
  • coverage: 85.7% total, zero uncovered functions
  • exact-head frontier review: no findings
  • exact-head cheap-tier rereview: no findings
  • hosted test-fast and test-race: PASS on this exact SHA

The prior hosted results applied to the obsolete pre-rebase head and are not being used as promotion evidence.

Dependency placement

This slice must merge before #1067/#1070 so terminal transition atomicity is composed on top of the accepted waiting-state lifecycle. It also unblocks the #1000 cron/callback promotion chain.

@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact tested head: 56c8b7e. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 56c8b7ee1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/harness/ask_user_broker.go Outdated
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact tested head: 963e0c8. Addressed the P2 deadline finding with red/green coverage. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 963e0c80ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/harness/ask_user_broker.go Outdated
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact tested head: 4cdff23. Addressed the notifier deadline finding with blocked-notifier regressions and a deadline-bound asynchronous notification path. Full normal/race/coverage gate passes. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4cdff23ae3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/harness/runner_step_engine.go Outdated
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact tested head: 5eb6690. Addressed wait-event-before-resume ordering with a deterministic blocked-persistence/quick-answer regression. Full normal/race/coverage gate passes. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5eb6690d6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/harness/checkpoint_brokers.go Outdated
Comment thread internal/harness/runner_step_engine.go Outdated
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact tested head: cf695c8. Addressed both persistence-race findings with deterministic checkpoint and stale-status regressions. Full normal/race/coverage gate passes at 85.6% with zero uncovered functions. @codex review

1 similar comment
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact tested head: cf695c8. Addressed both persistence-race findings with deterministic checkpoint and stale-status regressions. Full normal/race/coverage gate passes at 85.6% with zero uncovered functions. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cf695c8b5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +86 to +90
case <-waitCtx.Done():
b.clearPendingIfMatch(req.RunID, entry)
if err := ctx.Err(); err != nil {
return nil, time.Time{}, err
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Return buffered answers when notification hits deadline

At exact head cf695c8b, if OnPending remains blocked while Submit succeeds before the deadline, Submit deletes the pending entry and buffers the answer, but this deadline branch returns a timeout without checking entry.answerC. The checkpoint broker now recovers an answer that won this race, but the in-memory broker still discards the successfully accepted answer; check the buffered submission before reporting the timeout.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 8e216c4. Both in-memory deadline branches now use bounded cleanup that returns a buffered answer accepted before the deadline; the regression uses bounded gating/receives and passed 30 normal plus 10 race repetitions.

Comment thread internal/checkpoints/service.go Outdated
Comment on lines +186 to +188
if record.Status != StatusPending {
s.resolveMu.Unlock()
return nil

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Report when another checkpoint resolution already won

When an answer, approval, or HTTP resume races deadline expiry, the caller can read a pending checkpoint before ExpirePending wins resolveMu; resolve then reaches this branch and returns nil. Consequently Submit, Approve, or POST /resume acknowledges success even though the checkpoint remains expired and the submitted payload is discarded. Return a distinguishable already-resolved result so callers do not falsely report accepted input.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed end to end in 8e216c4. ErrAlreadyResolved remains distinct in the checkpoint service, maps to existing no-pending semantics for AskUser and approval/deny callers, maps to 409 no_pending_input through /v1/runs/{id}/input, and maps to stable 409 already_resolved for generic checkpoint resume without changing durable status, payload, or updated_at.

@dennisonbertram

dennisonbertram commented Jul 31, 2026

Copy link
Copy Markdown
Owner Author

Exact tested head: 8e216c4. Sixth review repair adds shared checkpoint deadline recovery, stable no-pending/conflict error mapping, immutable repeated-resume behavior, and bounded deterministic race tests. Evidence: focused count=30; focused race count=10; complete affected packages normal/race; unchanged ./scripts/test-regression.sh PASS with coverage 85.6% and zero uncovered functions. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8e216c4b64

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/harness/ask_user_broker.go Outdated
select {
case <-notified:
case <-waitCtx.Done():
return b.finishAskWait(ctx, req, entry)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wait for pending publication before returning recovered answers

When OnPending has entered the runner's setStatus and is stalled in Store.UpdateRun, a submission can be accepted and buffered before the deadline. Fresh evidence at the final head is that this deadline arm now calls finishAskWait, which returns that answer without waiting for notified to close; the step engine can therefore emit run.resumed before the notifier later emits run.waiting_for_user, or the wait event can be dropped if the run becomes terminal first. The checkpoint-backed branch has the same early recovery behavior, so accepted-answer recovery must preserve wait-event publication ordering.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed on exact head 331bc7c44d3d8302ed21f1843a48f79372cdba7c. Accepted-answer recovery now waits for pending publication; notifier persistence uses the supplied deadline context; started observer publication drains; transient status/event failures retry once-on-success. Deterministic ordering, deadline, observer, retry, race, and full regression gates pass.

@dennisonbertram
dennisonbertram force-pushed the codex/issue-1054-waiting-pending-order branch from 8e216c4 to 97665cc Compare July 31, 2026 07:25
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Rebased exact tested head: 97665cc, based on origin/main 8906149. Reviewed checkpoint race and HTTP semantics are preserved. Evidence rerun after rebase: focused normal count=30 PASS; focused race count=10 PASS; complete internal/checkpoints, internal/harness, and internal/server normal/race PASS; unchanged foreground non-TTY ./scripts/test-regression.sh PASS with coverage 85.7% and zero uncovered functions. @codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 97665cc6a0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@dennisonbertram
dennisonbertram force-pushed the codex/issue-1054-waiting-pending-order branch from 97665cc to 331bc7c Compare July 31, 2026 14:49
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact-head verification for 331bc7c44d3d8302ed21f1843a48f79372cdba7c:

  • focused pending publication normal stress (-count=30) PASS
  • focused observer/retry/redaction/SSE race stress (-count=20) PASS
  • checkpoint cross-Service/transient-poll normal -count=100, race -count=50 PASS
  • go test ./internal/checkpoints ./internal/harness ./internal/server PASS
  • go test -race ./internal/checkpoints ./internal/harness ./internal/server PASS
  • unchanged ./scripts/test-regression.sh PASS: normal, race, coverage; coveragegate: PASS (total=85.7%, min=80.0%, zero-functions=0)

The rebased head also includes once-on-success pending publication, observer drain, atomic cross-Service checkpoint CAS, transient poll tolerance, monotonic serialized status persistence, strict waiting-event retry with contiguous SSE cursors, deliberate redaction-drop completion, and preserved ordinary-event best-effort behavior.

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 331bc7c44d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +198 to +201
persisted := j.runner.storeAppendEventContext(ctx, delivery.event, delivery.eventSeq)
if (!persisted && requirePersistence) || ctx.Err() != nil {
j.discardPreparedEvent(delivery)
return false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Do not roll back an event after its append succeeds

When a strict waiting-event append commits just as the notifier deadline expires, persisted is true but the independent ctx.Err() check still calls discardPreparedEvent. The durable store retains that sequence while the in-memory ledger rolls it back; the next event then reuses the same (run_id, seq), causing SQLite and MemoryStore to reject its append and making durable replay diverge from live history.

Useful? React with 👍 / 👎.

Comment on lines +1577 to +1580
if !started {
stopped = true
cancel()
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve failed publication after pending input disappears

When the notifier's first waiting-state persistence attempt fails quickly and a user submits before the polling observer has seen the pending record, submission removes that record and the tool returns successfully. This stop path then cancels the never-started observer, so the transient failure is never retried and run.resumed can be emitted without a preceding run.waiting_for_user; unlike the existing retry test, this scenario does not wait for the observer's second attempt before submitting.

Useful? React with 👍 / 👎.

Comment on lines +151 to +153
if record.Status != StatusPending && !s.resolutionActive(id) {
s.unregister(id, ch)
return waitResultFromRecord(record)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recover approvals that commit before the wait deadline

When an approval's ResolvePending has durably committed before the timeout but its store call has not yet returned, this local-resolution suppression prevents Wait from observing the approved row. checkpointApprovalBroker.Ask therefore reaches its deadline and reports ApprovalTimeoutError, while the subsequent expiration loses to the already-approved checkpoint and the operator's approval call later succeeds. Fresh evidence at this head is that only the ask-user timeout path re-reads the winning resolution; approval waits still unconditionally report timeout.

Useful? React with 👍 / 👎.

@dennisonbertram
dennisonbertram force-pushed the codex/issue-1054-waiting-pending-order branch from 331bc7c to 3a911af Compare July 31, 2026 15:43
@dennisonbertram

Copy link
Copy Markdown
Owner Author

Exact-head review requested for 3a911af on production base d8e7c26. Affected packages pass normal/race; the former hosted failure passes 100x normal and 100x race; full foreground regression passes at 85.6% with zero uncovered functions. @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a911af9ab

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ticker := time.NewTicker(time.Millisecond)
defer ticker.Stop()
for {
if pending, ok := broker.Pending(runID); ok {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make pending-observer reads cancellable

When broker.Pending(runID) blocks—for example, while the checkpoint store is stalled—the observer cannot react to observerCtx cancellation, and stopPendingObserver subsequently waits on done before tool execution can finish. This can delay or indefinitely hang an otherwise answered, timed-out, or cancelled AskUserQuestion run; use a context-aware/bounded pending lookup or avoid synchronously draining an uncancellable read.

Useful? React with 👍 / 👎.

@dennisonbertram
dennisonbertram force-pushed the codex/issue-1054-waiting-pending-order branch from 3a911af to e6e255d Compare July 31, 2026 22:34
@dennisonbertram
dennisonbertram merged commit e6e255d into main Jul 31, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: waiting_for_user becomes visible before pending input exists

1 participant