fix(server): stop a timed-out authority call from hanging forever - #413
Open
ryanleecode wants to merge 2 commits into
Open
fix(server): stop a timed-out authority call from hanging forever#413ryanleecode wants to merge 2 commits into
ryanleecode wants to merge 2 commits into
Conversation
…ed call `remote_authority_call` raced the inner host call against a cooperative cancel token and a deadline, then ran `let _ = call.await;` on every non-success arm — re-awaiting the future it had just cancelled. When the inner future was parked on an await that never observes the token (a statement-store connect, an unacked subscribe), the timeout branch parked with it and the call returned nothing: no success, no timeout error. The dispatcher only sends a response frame once the handler resolves, so an `account_get` request was left neither answered nor refused. Drop the `select!` loser instead, matching the precedent in `bulletin_rpc::submit_preimage`. The cancel signal is still raised first, so cancel-aware inner futures and other holders of the shared token keep their cooperative path; the drop is the hard stop behind it. Restores the host-side `warn!` for an abandoned call, which used to come from `submit_remote_message`'s unwind and is skipped by the drop. Two pre-existing assertions that a timed-out SSO request unsubscribes two statement streams are removed: they only held because the re-await let the timed-out call keep working past its deadline, subscribing a second time and then tearing both down. Under the bound only one subscribe is ever sent. `sign_raw_cancellation_unsubscribes_sso_subscriptions`, which stages both subscriptions before firing the token, still covers unsubscribe-on-drop. Tests observed failing against the unfixed code (each on the `wait_until` hang assertion) before the fix landed, and again on a deliberate revert. Closes #405
Seeds docs/solutions/ with the transferable rule behind the #405 fix: a select! that cancels a loser and then awaits it has not bounded anything. Includes a recurrence scan confirming the fixed site was the only instance of the shape in rust/**. Seeds CONCEPTS.md with the account-authority and statement-store vocabulary the learning depends on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
get_accountcould hang forever while the user was logged in.remote_authority_callputs a deadline on every host authority call, but when that deadline fired it cancelled the inner call and then awaited it to completion before returning the timeout error. Cancellation here is cooperative, so an inner future parked on an await that never checks the token — a statement-store connect that never resolves, a subscribe whose ack never arrives — parked the timeout branch with it. The wire caller got no success and no error, just silence: the dispatcher sends a response frame only once the handler resolves.The three non-success arms now return their error directly and drop the losing future.
cancel_with_reasonstill runs first, so cancel-aware inner futures and any other holder of the shared token keep their cooperative path; the drop is the hard stop behind it. This matchessubmit_preimage, which already drops its loser on cancel and on timeout.Closes #405.
Why drop rather than the alternatives
Both alternatives are named in the issue as non-counting outcomes, and both are rejected here:
remote_authority_callis also a free function with no spawner in scope, so threading one through would touch all 18 call sites to buy that leak.Two pre-existing assertions were deleted, not adapted
sign_raw_uses_call_context_timeout_for_sso_response_waitandresource_allocation_respects_a_shorter_call_context_timeouteach asserted that a timed-out request unsubscribes two statement streams. Both began failing under the bound. A throwaway probe showed why: with the deadline actually enforced, only one subscribe is ever sent, because the call is abandoned before it reaches the second. Those assertions had only ever passed because the re-await let a timed-out call keep working past its deadline — they were asserting the bug's side effect, so they were removed rather than loosened.Unsubscribe-on-abandonment still has a home in
sign_raw_cancellation_unsubscribes_sso_subscriptions, which stages both subscriptions before firing the token.Known residual — worth reading before approving
Adversarial review found one real P1 that this change does not fix. On the
SigningHost,allocate_statement_store_allowanceholdsregistration_lockacross an on-chainregister_statement_accountsubmission (sso_responder.rs:949-1010), reachable fromremote_authority_callatstatement_store.rs:344. The deadline now releases that lock while the submission may still be in flight, so a renewal pass can take the lock, scan, see the slot still free, and submit for it — and on a full period each attempt is a revocation.This is not a reason to hold the PR: before this change that critical section stayed intact only by hanging the caller forever. It needs its own design rather than a patch inside this fix.
docs/residual-review-findings/ryan-405.mdrecords it, two smaller drop-window findings, and the deferrals the plan names — the cross-productaccount_access_authorizationawait, and a dispatcher-level response backstop.Verification
Pendingforever and sets a flag onDrop, driven on a spawned thread. Each asserts four things: the error came back, the shared token recorded its reason, the inner call was dropped before the caller resumed, and wall-clock elapsed stayed under 500 ms. One further test covers the wire path —get_accountanswers its caller when the statement-store connect never completes.let _ = call.await;hangs the bounded-return tests and the wire-path test. Deletingcancel_with_reasonfails the reason assertion withleft: None, right: Some(TimedOut { timeout: 1ms }).-D warnings,wasm32-unknown-unknowncheck,cargo +nightly fmt --check, clippy with-D warnings, andcargo test --workspace --all-features.truapi-serverhas pre-existing wall-clock flakiness under back-to-back load, reproduced on the clean tree as well —submit_preimage_recovers_inconsistent_inclusion_via_recheckfailed 1 of 4 clean runs withTimeout { phase: Connect }. A replacement test proposed in review was measured to flake 1-in-3 and removed rather than shipped.