Skip to content

fix(agent): wake the shutdown waiter that notify_waiters left behind - #79

Merged
jdfalk merged 1 commit into
mainfrom
fix/shutdown-lost-wakeup
Aug 12, 2026
Merged

fix(agent): wake the shutdown waiter that notify_waiters left behind#79
jdfalk merged 1 commit into
mainfrom
fix/shutdown-lost-wakeup

Conversation

@jdfalk

@jdfalk jdfalk commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

The hang, root-caused

cargo test --workspace hung for 20+ minutes on CI and reproduced locally. It was
diagnosed twice as a stuck runner. It is a real defect, and this fixes it.

A cargo test --workspace from the previous evening was still alive after 26
hours
. Sampling it named the parked frame exactly:

Thread: a_retired_intent_the_server_disowns_is_restored
  connect_client.rs:577   (run.await, after shutdown.stop())
    tokio Runtime::block_on -> park -> Condvar::wait     [0% CPU]

Root cause

Shutdown::stop signals with Notify::notify_waiters, which wakes only the tasks
already parked at that instant and stores no permit for one arriving later.
Both waits — the session loop (client.rs:573) and the reconnect backoff
(client.rs:500) — awaited notified() directly, with no level-triggered read of
the stopped flag that stop had already set.

So when stop landed while the client was anywhere else — mid-recovery,
mid-register, mid-dispatch — the wakeup was lost. The session loop then parked on
a notification already gone while the inbound stream stayed open and silent.
session() never returned, so run() never reached the is_stopped() check that
ends it. Permanent, not slow.

Evidence

build hangs
feat/transport-mode, one test ×260 5
main (pre-merge), same test ×260 0
this fix, whole connect_client binary ×150 0
full workspace suite 526 passed, 0 failed, 33s

The defect is latent in client.rs, which #78 never touched; #78 appears to have
widened the window (the added transport field lengthens register, delaying the
client's arrival at the park). The main-vs-branch split is suggestive, not proof —
separate target dirs mean code layout differs — so treat this as exposed by, not
caused by, #78.

The fix

Shutdown::cancelled registers as a waiter before reading the flag; stop
writes the flag before notifying. Two opposed orderings, so no interleaving can
miss both. Used at both wait sites.

On the test

It asserts under a timeout deliberately. The bug hangs rather than fails, so a
bare assertion would reproduce the disease instead of reporting it — which is
precisely why this cost 26 hours with no failing test to point at.

Verified honestly: cancelled was temporarily sabotaged back to the naive
notified().await and the test went red with Elapsed(()), while the
already-parked-waiter test stayed green. The two cover opposite directions.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U9keBRnGXBP2DiQmrcqm33

`Shutdown::stop` signalled with `Notify::notify_waiters`, which wakes only
the tasks already parked at that instant and stores no permit. Both waits --
the session loop and the reconnect backoff -- awaited `notified()` directly,
with no level-triggered read of the flag `stop` had already set.

When `stop` landed while the client was anywhere else (mid-recovery,
mid-register, mid-dispatch) the signal was lost. The session loop then parked
on a notification already gone, the inbound stream stayed open and silent,
`session` never returned, and `run` never reached its `is_stopped` check.

That is a permanent hang, not a slow one: 26 hours parked on a condvar at
zero CPU on CI, ~1 run in 50 locally. It presented as a hung
`cargo test --workspace` and was twice mistaken for a stuck runner, because
a hang leaves no failing test behind to point at.

`Shutdown::cancelled` registers as a waiter before reading the flag, and
`stop` writes the flag before notifying. Two opposed orderings, so no
interleaving misses both.

The regression test asserts under a timeout on purpose -- the bug hangs
rather than fails, so a bare assertion would reproduce it instead of
reporting it. Verified by sabotaging `cancelled` back to the naive form and
confirming the test goes red.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U9keBRnGXBP2DiQmrcqm33
@jdfalk
jdfalk merged commit c75a72c into main Aug 12, 2026
13 checks passed
@jdfalk
jdfalk deleted the fix/shutdown-lost-wakeup branch August 12, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant