tests: fix command-queue race in mock-ssh-server exec handling - #72
Merged
Conversation
Handler.run() creates the per-channel command queue with a check-then-assign while paramiko's transport thread creates it with setdefault() and puts the command into it from check_channel_exec_request(). When the exec request lands inside the window, run() replaces the queue that already holds the command, handle_client() blocks on Queue.get() forever and the client never receives an exit status. This is the intermittent CI hang: every _execute()-based operation (cp_file, checksum, the move fallback) rolls these dice, and test_concurrency_for_raw_commands rolls them 16 at a time. Confirmed by the faulthandler dump from the first run with #71 merged (handle_client threads parked on Queue.get with the client waiting), and reproduced deterministically by widening the window with a 5ms sleep: upstream logic deadlocks on the first round, the same logic with an atomic setdefault() survives, with or without the delay. mock-ssh-server is unmaintained, so the method is patched in conftest instead of upstream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the actual fix for the intermittent CI hang that #71 instrumented. The first
mainrun with #71 merged hung again (ubuntu 3.11) and the new faulthandler dump pointed straight at it: main thread insidetest_concurrency_for_raw_commands, mocksshhandle_clientthreads parked forever onQueue.get(), asyncssh loop idle waiting for an exit status that never comes.The race (in mock-ssh-server, unmaintained since 2019)
Handler.run()— accept-loop thread:check_channel_exec_request()— paramiko transport thread:When the exec request lands between the check and the assignment,
run()replaces the queue that already holds the command;handle_client()then blocks on the empty replacement forever and the client never gets an exit status. Every_execute()-based op (cp_file,checksum, themovefallback) rolls these dice —test_concurrency_for_raw_commandsrolls them 16 at a time. Hangs, not failures — which is why--rerunscouldn't save the run and why this reddened nightlies intermittently for months (one random ubuntu job at a time; macOS scheduling apparently never opened the window).Proof
Widening the window with a 5ms sleep between check and assign makes it deterministic: upstream logic deadlocks on the first round of 24 concurrent execs; the same logic with an atomic
setdefault()survives — with or without the delay, and under the stock timing via this conftest patch.Fix
tests/conftest.pyreplacesHandler.runwith an identical copy whose queue creation is a single atomicsetdefault(). Patched locally since upstream is unmaintained.🤖 Generated with Claude Code