The RPC Tests workflow fails on a large majority of pushes to main: 23 of the last 30 push-triggered runs failed. The failures don't point to one regression — a different integration test fails each time, and the failing job varies too (seen on both ubuntu-latest-8-cores and macos-15-large). For example, integration::tx::clawback::clawback has failed on an immediate post-clawback balance assertion on macOS, while integration::auto_build::deploy_workspace_with_package_auto_builds has failed on Ubuntu in a different run. This pattern looks like timing/eventual-consistency races and possibly resource contention in the live-network integration suite rather than a single code defect.
A few directions that might be worth exploring: replacing immediate post-action assertions (like the clawback balance check) with a poll-with-timeout, adding retries for the integration test suite (e.g. via cargo-nextest --retries), and possibly separating out the flakiest tests into a non-blocking job so they stop gating main while they're investigated.
Below are 15 recent runs on main where this workflow failed, most recent first, as evidence of the pattern:
The RPC Tests workflow fails on a large majority of pushes to
main: 23 of the last 30 push-triggered runs failed. The failures don't point to one regression — a different integration test fails each time, and the failing job varies too (seen on bothubuntu-latest-8-coresandmacos-15-large). For example,integration::tx::clawback::clawbackhas failed on an immediate post-clawback balance assertion on macOS, whileintegration::auto_build::deploy_workspace_with_package_auto_buildshas failed on Ubuntu in a different run. This pattern looks like timing/eventual-consistency races and possibly resource contention in the live-network integration suite rather than a single code defect.A few directions that might be worth exploring: replacing immediate post-action assertions (like the clawback balance check) with a poll-with-timeout, adding retries for the integration test suite (e.g. via
cargo-nextest --retries), and possibly separating out the flakiest tests into a non-blocking job so they stop gatingmainwhile they're investigated.Below are 15 recent runs on
mainwhere this workflow failed, most recent first, as evidence of the pattern: