Skip to content

fix(server): stop a timed-out authority call from hanging forever - #413

Open
ryanleecode wants to merge 2 commits into
mainfrom
ryan-405
Open

fix(server): stop a timed-out authority call from hanging forever#413
ryanleecode wants to merge 2 commits into
mainfrom
ryan-405

Conversation

@ryanleecode

Copy link
Copy Markdown

Summary

A get_account could hang forever while the user was logged in. remote_authority_call puts a deadline on every host authority call, but when that deadline fired it cancelled the inner call and then awaited it to completion before returning the timeout error. Cancellation here is cooperative, so an inner future parked on an await that never checks the token — a statement-store connect that never resolves, a subscribe whose ack never arrives — parked the timeout branch with it. The wire caller got no success and no error, just silence: the dispatcher sends a response frame only once the handler resolves.

The three non-success arms now return their error directly and drop the losing future. cancel_with_reason still runs first, so cancel-aware inner futures and any other holder of the shared token keep their cooperative path; the drop is the hard stop behind it. This matches submit_preimage, which already drops its loser on cancel and on timeout.

Closes #405.

Why drop rather than the alternatives

Both alternatives are named in the issue as non-counting outcomes, and both are rejected here:

  • Make the parking awaits cancel-aware and keep the re-await. This treats the awaits that park today. The next await added inside any authority call that does not poll the token brings the hang straight back.
  • Detach onto a spawner with a reaper. This trades the hang for a leak: a never-resolving connect keeps its stack, its connection, and any lock it holds, with nobody waiting on it. remote_authority_call is also a free function with no spawner in scope, so threading one through would touch all 18 call sites to buy that leak.

Two pre-existing assertions were deleted, not adapted

sign_raw_uses_call_context_timeout_for_sso_response_wait and resource_allocation_respects_a_shorter_call_context_timeout each asserted that a timed-out request unsubscribes two statement streams. Both began failing under the bound. A throwaway probe showed why: with the deadline actually enforced, only one subscribe is ever sent, because the call is abandoned before it reaches the second. Those assertions had only ever passed because the re-await let a timed-out call keep working past its deadline — they were asserting the bug's side effect, so they were removed rather than loosened.

Unsubscribe-on-abandonment still has a home in sign_raw_cancellation_unsubscribes_sso_subscriptions, which stages both subscriptions before firing the token.

Known residual — worth reading before approving

Adversarial review found one real P1 that this change does not fix. On the SigningHost, allocate_statement_store_allowance holds registration_lock across an on-chain register_statement_account submission (sso_responder.rs:949-1010), reachable from remote_authority_call at statement_store.rs:344. The deadline now releases that lock while the submission may still be in flight, so a renewal pass can take the lock, scan, see the slot still free, and submit for it — and on a full period each attempt is a revocation.

This is not a reason to hold the PR: before this change that critical section stayed intact only by hanging the caller forever. It needs its own design rather than a patch inside this fix.

docs/residual-review-findings/ryan-405.md records it, two smaller drop-window findings, and the deferrals the plan names — the cross-product account_access_authorization await, and a dispatcher-level response backstop.

Verification

  • Five new unit tests pin the bound with an inner future that returns Pending forever and sets a flag on Drop, driven on a spawned thread. Each asserts four things: the error came back, the shared token recorded its reason, the inner call was dropped before the caller resumed, and wall-clock elapsed stayed under 500 ms. One further test covers the wire path — get_account answers its caller when the statement-store connect never completes.
  • Both gates were revert-checked. Re-adding let _ = call.await; hangs the bounded-return tests and the wire-path test. Deleting cancel_with_reason fails the reason assertion with left: None, right: Some(TimedOut { timeout: 1ms }).
  • The full CI gate set passes locally: workspace build with -D warnings, wasm32-unknown-unknown check, cargo +nightly fmt --check, clippy with -D warnings, and cargo test --workspace --all-features.
  • Flakiness note, measured rather than assumed: truapi-server has pre-existing wall-clock flakiness under back-to-back load, reproduced on the clean tree as well — submit_preimage_recovers_inconsistent_inclusion_via_recheck failed 1 of 4 clean runs with Timeout { phase: Connect }. A replacement test proposed in review was measured to flake 1-in-3 and removed rather than shipped.

…ed call

`remote_authority_call` raced the inner host call against a cooperative
cancel token and a deadline, then ran `let _ = call.await;` on every
non-success arm — re-awaiting the future it had just cancelled. When the
inner future was parked on an await that never observes the token (a
statement-store connect, an unacked subscribe), the timeout branch parked
with it and the call returned nothing: no success, no timeout error. The
dispatcher only sends a response frame once the handler resolves, so an
`account_get` request was left neither answered nor refused.

Drop the `select!` loser instead, matching the precedent in
`bulletin_rpc::submit_preimage`. The cancel signal is still raised first,
so cancel-aware inner futures and other holders of the shared token keep
their cooperative path; the drop is the hard stop behind it.

Restores the host-side `warn!` for an abandoned call, which used to come
from `submit_remote_message`'s unwind and is skipped by the drop.

Two pre-existing assertions that a timed-out SSO request unsubscribes two
statement streams are removed: they only held because the re-await let the
timed-out call keep working past its deadline, subscribing a second time
and then tearing both down. Under the bound only one subscribe is ever
sent. `sign_raw_cancellation_unsubscribes_sso_subscriptions`, which stages
both subscriptions before firing the token, still covers unsubscribe-on-drop.

Tests observed failing against the unfixed code (each on the `wait_until`
hang assertion) before the fix landed, and again on a deliberate revert.

Closes #405
Seeds docs/solutions/ with the transferable rule behind the #405 fix: a
select! that cancels a loser and then awaits it has not bounded anything.
Includes a recurrence scan confirming the fixed site was the only instance
of the shape in rust/**.

Seeds CONCEPTS.md with the account-authority and statement-store
vocabulary the learning depends on.
@ryanleecode
ryanleecode requested a review from a team August 14, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authority-call timeout cannot return while the cancelled call is still awaited

1 participant