Skip to content

Authority-call timeout cannot return while the cancelled call is still awaited #405

Description

@ryanleecode

Problem Statement

remote_authority_call puts a deadline on a host call. When that deadline fires it cancels the call and then awaits the cancelled future to completion before returning the timeout error. Cancellation here is cooperative, so if the inner future is parked on an await that does not observe the cancel signal, the timeout branch parks with it and the outer call never returns anything — no success, no timeout error. The caller is left with a request that was neither answered nor refused, which is how a get_account can go permanently unanswered while the user is logged in.

Goal

When an authority call's deadline elapses, the timeout error is returned to its caller within a bounded time regardless of whether the cancelled inner future ever completes.

Evidence: the timeout branch awaits the call it just cancelled

rust/crates/truapi-server/src/runtime.rs:274-290

futures::select! {
    result = call => result,
    reason = cancelled => { /* same shape: builds the error, then `let _ = call.await;` */ },
    () = timeout => {
        let reason = CancellationReason::TimedOut { timeout: timeout_duration };
        cx.cancel().cancel_with_reason(reason.clone());
        let error = authority_cancellation_error(cx, reason);
        let _ = call.await;
        Err(error.into())
    }
}

The error is constructed, then let _ = call.await; blocks on the cancelled future before it is returned. The same pattern appears three times: :278, :287, and :296 (the no-deadline arm).

Evidence: three awaits run before cancellation can reach anything

rust/crates/truapi-server/src/runtime/pairing_host/sso_channel.rs:236-254, inside submit_remote_message:

let rpc_client = self.statement_store.client("SSO statement-store").await
    .map_err(|err| SsoRemoteResponseError::Failure(err.to_string()))?;
let own_subscription = subscribe_statement_topic(&rpc_client, sso.session_id_own).await
    .map_err(|err| { /* ... */ })?;
let peer_subscription = subscribe_statement_topic(&rpc_client, sso.session_id_peer).await
    .map_err(|err| { /* ... */ })?;

cx.cancel() is handed only to wait_for_sso_remote_response(..) at :270-280, which runs after all three. Those three bottom out in runtime/statement_store_rpc.rs:130-139 (self.platform.connect(..).await) and :143-155 (rpc_client.subscribe(..).await) — neither carries a deadline at any layer in this crate.

Orientation

  • rust/crates/truapi-server/src/runtime.rsremote_authority_call, the deadline from cx.timeout() at :271, and the three let _ = call.await; sites.
  • rust/crates/truapi-server/src/runtime/pairing_host/sso_channel.rssubmit_remote_message; note the submit future at :257-267 is raced under cancellation, so it is not the problem.
  • rust/crates/truapi-server/src/runtime/statement_store_rpc.rs:130-155 — where the unbounded awaits terminate.
  • Reachability from the wire: runtime.rs:963-966 (product_account_public_key) is one caller on the account_get path.

Non-Counting Outcomes

  • Bounding only the TimedOut arm at :287 and leaving :278 and :296 able to park forever on an explicit cancellation or a caller-side disconnect.
  • Making these three specific awaits cancel-aware while keeping let _ = call.await; — that fixes today's stall and leaves the structure, so the next non-cancel-aware await reintroduces the same bug.
  • Adding a deadline to the statement-store client acquisition only, when both subscribe_statement_topic calls are equally unbounded.
  • Detaching the inner future so the caller returns while the future keeps a subscription, connection, or lock alive indefinitely — a resource leak substituted for a bound. If the future is dropped or detached, say explicitly what happens to what it holds.
  • A test whose "stalled" future yields at a cancellation point: it passes by construction and never exercises the case that hangs.

Acceptance Criteria

  • (gatekeeper) A test builds an inner call parked on an await that ignores the cancel signal and asserts remote_authority_call still returns its timeout error within a bounded time; it fails when the change is reverted. Runnable via cargo test -p truapi-server.
  • cargo test --workspace exits 0.
  • All three let _ = call.await; sites (:278, :287, :296) either carry a bound or the pattern is removed.
  • The fate of a still-running inner future after the caller returns is stated and tested: dropped, detached with a reaper, or bounded — not unspecified.
  • For every case where an authority call on the account_get path exceeds its deadline, a response is delivered to the wire caller rather than nothing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions