ci: make the intermittent connect hang diagnosable and self-healing - #71
Merged
Conversation
The suite intermittently freezes at its first SSH connection against mock-ssh-server (always a single ubuntu job, random Python version; observed across months of scheduled runs, e.g. Apr 26, Jul 25-28 and Aug 1-4 which were four red nightlies in a row). The 1-minute step timeout killed the job before anything could report, so every failure was an undiagnosable freeze. - pytest faulthandler_timeout=60 dumps all thread stacks when a test stalls, so the next hang shows where it is stuck - step timeout of 3 minutes leaves room for the dump and for asyncssh's login_timeout (120s) to raise a real error - pytest-rerunfailures retries the failed test once so a transient hang no longer reddens the whole run, while the dump stays in the log Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shcheklein
added a commit
that referenced
this pull request
Aug 5, 2026
Handler.run() creates the per-channel command queue with a check-then-assign while paramiko's transport thread creates it with setdefault() and puts the command into it from check_channel_exec_request(). When the exec request lands inside the window, run() replaces the queue that already holds the command, handle_client() blocks on Queue.get() forever and the client never receives an exit status. This is the intermittent CI hang: every _execute()-based operation (cp_file, checksum, the move fallback) rolls these dice, and test_concurrency_for_raw_commands rolls them 16 at a time. Confirmed by the faulthandler dump from the first run with #71 merged (handle_client threads parked on Queue.get with the client waiting), and reproduced deterministically by widening the window with a 5ms sleep: upstream logic deadlocks on the first round, the same logic with an atomic setdefault() survives, with or without the delay. mock-ssh-server is unmaintained, so the method is patched in conftest instead of upstream. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The test suite intermittently freezes at its first SSH connection against mock-ssh-server — always exactly one ubuntu job per run, random Python version, never macOS. Scheduled runs show it across months (Apr 26, Jul 25, Jul 27–28, and Aug 1–4 were four red nightlies in a row). Because the
Teststep hastimeout-minutes: 1, the job is killed before anything can report: every failure is a mute freeze right aftertest_sftp_pools.py, with no traceback.The suite opens essentially one real SSH connection (fsspec's instance cache reuses the filesystem across tests), so the hang always lands on the session's first
test_sshfs.pytest. asyncssh'slogin_timeout(120s) would raise a diagnosable error on a stalled handshake, but the 60s step axe always wins.Change
faulthandler_timeout = 60in pytest config: a test stalling for a minute dumps all thread stacks to the log, so the next occurrence tells us exactly where it is stuck (asyncssh handshake vs mockssh/paramiko accept thread).timeout-minutes: 1 → 3on the Test step: leaves room for the dump and forlogin_timeoutto fire.--reruns 1 --reruns-delay 2(pytest-rerunfailures): a transient hang no longer reddens the whole run — the stack dump stays in the log while the retried test goes green.This intentionally does not claim to fix the underlying race (mock-ssh-server is unmaintained); it converts an opaque recurring red into either a self-healed run with evidence in the log, or a real traceback.
🤖 Generated with Claude Code