Skip to content

Cancel timed-out upstream requests before retry - #187

Merged
lynnswap merged 11 commits into
mainfrom
codex/fix-proxy-cancellation-lifecycle
Jul 29, 2026
Merged

Cancel timed-out upstream requests before retry#187
lynnswap merged 11 commits into
mainfrom
codex/fix-proxy-cancellation-lifecycle

Conversation

@lynnswap

@lynnswap lynnswap commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Purpose

Prevent timed-out or abandoned proxy requests from leaving work active on a degraded Xcode bridge while replacement work starts, which can repeatedly create bridge processes and duplicate Xcode agent attachments.

Changes

  • Give RPCHandle one lifecycle state machine for cancellation-handler installation, queue registration, upstream ID assignment, completion, and cancellation.
  • Terminalize cancellation that arrives before handler installation, share one delivery across repeated or reentrant cancellation, and release the reserved request lease when late installation is rejected.
  • Serialize notifications/cancelled after the original request send and preserve exact request identity until cancellation delivery settles.
  • Gate catalog-load admission during backoff, scope timeout reservations to the exact catalog load, reject stale timeout and cancellation callbacks, and advance bounded retry cadence independently of activation identity.
  • Recover rejected cancellation by replacing the degraded channel and preparing a fresh route activation; apply equivalent recovery to ordinary request timeout and disconnect paths.
  • Serialize shutdown admission across initialize, topology, health, and process-control owners so late lifecycle work cannot republish state.
  • Separate process-route health evaluation from effect execution so reconciliation commits under the initialize gate and starts probe I/O only after releasing its non-recursive lock.
  • Make retirement, cancellation-delivery, and health-probe tests await the task group or event that actually owns completion.
  • Add deterministic coverage for cancellation ordering and delivery, pre-install cancellation, stale catalog work, timeout attachment races, retry cadence, rejected recovery, route retirement, health-probe handoff, and shutdown races.

Testing

  • scripts/check.sh
    • 1,022 package tests
    • 24 process runtime tests
    • 13 stdio adapter tests
  • Process routing, recovery, and catalog suites: 124 passed
  • CI-sensitive retirement, cancellation, and health-probe regressions: 20 repeated iterations passed
  • git diff --check
  • Branch-wide codex-review against pinned main: no findings

Screenshots

Not applicable; this changes proxy runtime lifecycle behavior.

Keep process catalog retries on the existing bridge and wait for upstream cancellation delivery before arming the next attempt. This prevents repeated mcpbridge attachment generations from accumulating in Xcode.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 06e1d4cb7c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lynnswap added 2 commits July 28, 2026 18:19
Make RPC cancellation wait for the original upstream send, preserve rejected delivery as a distinct terminal result, and recover the channel instead of reusing it. Gate catalog admission during backoff and advance a route-owned retry ordinal across same-slot retries.
Carry cancellation completion through the empty-catalog path and allow retry scheduling when cancellation is no longer needed. Keep rejected cancellation on the channel-recovery path.

@lynnswap lynnswap left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 97d452180d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lynnswap added 4 commits July 28, 2026 21:09
Keep request and catalog leases owned until send and cancellation outcomes settle. Recover rejected cancellation channels atomically, serialize topology replacement with shutdown, and block replacement traffic until predecessor stop completes.
Move catalog timeout ownership into each CatalogLease load. Preserve sibling loads, reject stale callbacks by reservation generation, and phase-scope initialize timeouts after channel initialization.
Publish prepared request ID mappings atomically with cancellation state so cancellation cleans them without sending an upstream notification before registration. Consume catalog timeout reservations even when callbacks fire before timer attachment, and linearize timeout mutation against shutdown.
Make RPCHandle own handler installation and lifecycle transitions in one state machine so cancellation always reaches a terminal delivery. Reject late installation, release the reserved request lease, and cover pre-install cancellation and reentrant delivery sharing.

@lynnswap lynnswap left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codex review

Route stops are owned by upstreamRetirementTasks, so draining runtimeTasks does not establish completion. Synchronize the three retirement assertions on the upstream stop event they verify.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b8619337a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

lynnswap added 2 commits July 28, 2026 23:43
Keep health probe reservation and process route reconciliation inside the lifecycle gate, then apply control-plane and probe effects only after releasing its non-recursive lock. Cover successful probes and shutdown-winning handoffs deterministically.
Observe the upstream cancellation and drain runtime settlement before asserting the correlated lease was released. The waiter timeout alone does not establish that happens-before edge.

@lynnswap lynnswap left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 2168db9c5f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@lynnswap

Copy link
Copy Markdown
Owner Author

The failing Package Tests (macOS, remaining) job was not caused by the macos-26 runner label. The runner was assigned and the job reached the test suite.

The failure was a race in HTTPConcurrencyTests: the test read an immediate upstream request-history snapshot before the queued request send had completed. Commit 92d2e51 moves that synchronization into EmbeddedControlledUpstreamClient and awaits the observable request-arrival boundary at all equivalent call sites.

Validation:

  • HTTPConcurrencyTests: 12 tests passed
  • failing test: 30 consecutive runs passed
  • scripts/check.sh: 1,022 + 24 + 13 tests passed
  • branch-wide Codex review against main: 0 findings

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 92d2e51f61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@lynnswap
lynnswap merged commit 3c1278a into main Jul 29, 2026
9 checks passed
@lynnswap
lynnswap deleted the codex/fix-proxy-cancellation-lifecycle branch July 29, 2026 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant