Skip to content

tests: fix command-queue race in mock-ssh-server exec handling - #72

Merged
shcheklein merged 1 commit into
mainfrom
fix-mockssh-exec-race
Aug 5, 2026
Merged

tests: fix command-queue race in mock-ssh-server exec handling#72
shcheklein merged 1 commit into
mainfrom
fix-mockssh-exec-race

Conversation

@shcheklein

Copy link
Copy Markdown
Collaborator

This is the actual fix for the intermittent CI hang that #71 instrumented. The first main run with #71 merged hung again (ubuntu 3.11) and the new faulthandler dump pointed straight at it: main thread inside test_concurrency_for_raw_commands, mockssh handle_client threads parked forever on Queue.get(), asyncssh loop idle waiting for an exit status that never comes.

The race (in mock-ssh-server, unmaintained since 2019)

Handler.run() — accept-loop thread:

if channel.chanid not in self.command_queues:      # check
    self.command_queues[channel.chanid] = Queue()  # assign — not atomic with the check

check_channel_exec_request() — paramiko transport thread:

self.command_queues.setdefault(channel.get_id(), Queue()).put(command)

When the exec request lands between the check and the assignment, run() replaces the queue that already holds the command; handle_client() then blocks on the empty replacement forever and the client never gets an exit status. Every _execute()-based op (cp_file, checksum, the move fallback) rolls these dice — test_concurrency_for_raw_commands rolls them 16 at a time. Hangs, not failures — which is why --reruns couldn't save the run and why this reddened nightlies intermittently for months (one random ubuntu job at a time; macOS scheduling apparently never opened the window).

Proof

Widening the window with a 5ms sleep between check and assign makes it deterministic: upstream logic deadlocks on the first round of 24 concurrent execs; the same logic with an atomic setdefault() survives — with or without the delay, and under the stock timing via this conftest patch.

Fix

tests/conftest.py replaces Handler.run with an identical copy whose queue creation is a single atomic setdefault(). Patched locally since upstream is unmaintained.

🤖 Generated with Claude Code

Handler.run() creates the per-channel command queue with a
check-then-assign while paramiko's transport thread creates it with
setdefault() and puts the command into it from
check_channel_exec_request(). When the exec request lands inside the
window, run() replaces the queue that already holds the command,
handle_client() blocks on Queue.get() forever and the client never
receives an exit status.

This is the intermittent CI hang: every _execute()-based operation
(cp_file, checksum, the move fallback) rolls these dice, and
test_concurrency_for_raw_commands rolls them 16 at a time. Confirmed
by the faulthandler dump from the first run with #71 merged
(handle_client threads parked on Queue.get with the client waiting),
and reproduced deterministically by widening the window with a 5ms
sleep: upstream logic deadlocks on the first round, the same logic
with an atomic setdefault() survives, with or without the delay.
mock-ssh-server is unmaintained, so the method is patched in conftest
instead of upstream.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@shcheklein
shcheklein merged commit 1822765 into main Aug 5, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant