tests: kill the process tree on integ command timeout and widen the callback wait - #9203
Open
roger-zhangg wants to merge 3 commits into
Open
tests: kill the process tree on integ command timeout and widen the callback wait#9203roger-zhangg wants to merge 3 commits into
roger-zhangg wants to merge 3 commits into
Conversation
…allback wait A timed-out command killed only the direct child, so cargo/zig grandchildren survived holding the cargo package-cache lock, making every rerun time out too. Also raise the callback-ID wait, which is an upper bound the happy path never pays.
![aws-sam-tooling-bot[bot]](https://avatars.githubusercontent.com/b/1290?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) does this change fix?
N/A — nightly integration test failures.
Why is this change necessary?
Two Windows jobs failed in the 2026-08-26 nightly (run 32926245225). Both are test-harness flakes, not product bugs — the same commit passed in the runs on either side of it.
tier1-windows-build-2: all 4 x86_64 Rust builds timed out at 600s, on all 4 attempts each (91 min job).run_commandkills only the direct child on timeout, socargo/ziggrandchildren survive holding the$CARGO_HOME/.package-cachelock, which cargo waits on indefinitely. One slow build therefore poisons every later build in the job.tier1-windows-other:test_tier1_callbackgot 30s for the callback ID to appear. The invoke was healthy and mid-startup (poll()=None, last lineFound credentials in environment variables.at ~19s) — the budget is just too tight for container startup on Windows. Same failure on 2026-08-20.How does it address the issue?
kill_processhelper (kills the tree, already used by the long-running-process tests) in the timeout path ofrun_commandandrun_command_with_input, wrapped so it can never mask the originalTimeoutExpired.wait_for_callback_iddefault from 30s to 120s.What side effects does this change have?
The callback wait is polled and returns as soon as the line lands, so the happy path is unchanged; only a genuinely failing wait takes longer before asserting. Timeouts now also reap grandchildren, which is what the non-timeout paths already do.
Verified locally that a timed-out command's orphaned grandchild survives on
developand is killed with this change, withTimeoutExpiredstill raised in both cases.make black-checkandmake lintpass.Mandatory Checklist
PRs will only be reviewed after checklist is complete
make prpassesmake update-reproducible-reqsif dependencies were changedBy submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.