[FEAT]: Durable xdist shard transport primitives (no wiring)#113
Open
nina-msft wants to merge 1 commit into
Open
[FEAT]: Durable xdist shard transport primitives (no wiring)#113nina-msft wants to merge 1 commit into
nina-msft wants to merge 1 commit into
Conversation
Introduces the building blocks for durable per-worker result transport under pytest-xdist, addressing the durability (#3) and granular size-cap (microsoft#4) gaps in the workeroutput transport. These are pure additions with no call sites -- existing behavior is unchanged; the wiring lands in a follow-up PR. - ShardWriter: append-only, flush-per-result JSONL writer so results a worker has already produced survive an abrupt worker death. Oversized records become a marker line naming the dropped nodeid/index rather than discarding the whole shard. - read_worker_shard: line-by-line recovery tolerating a truncated final line; corrupt/oversized records are surfaced as DroppedRecord entries so the caller can mark the run incomplete. nodeid/index are taken authoritatively from the record envelope, mirroring deserialize_worker_data's trust posture. - shard_eligible: gates shard transport to runs where every --tx gateway is a local popen worker (shared filesystem); remote/proxied topologies fall back to workeroutput. - SHARD_DIR_KEY stash constant. Adds 25 unit tests covering eligibility, write/flush/oversize, recovery, corruption tolerance, authoritative ordering metadata, and full-fidelity round-trip. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
First PR in a stack of two that adds durable result transport for pytest-xdist, closing two gaps in the current in-memory
workeroutputtransport:pytest_sessionfinish).This PR lands only the primitives for the fix. Every symbol is a pure addition to
rampart/pytest_plugin/_xdist.pywith no call sites — existing behavior is completely unchanged. It establishes the vocabulary the wiring PR consumes, so it can be reviewed small and green on its own.What's added
ShardWriternodeid/indexinstead of discarding the whole shard.read_worker_shardDroppedRecordentries so the caller can mark the run incomplete.nodeid/indexare taken authoritatively from the record envelope, mirroringdeserialize_worker_data's existing trust posture (never trusting values embedded in the serialized result body).shard_eligible--txgateway is a localpopenworker (shared filesystem). Remote (ssh=/socket=) or proxied (via=) topologies are ineligible and will fall back toworkeroutputin PR B.SHARD_DIR_KEYThe full-fidelity
Resultround-trip reuses the existing_serialize_result/_deserialize_resultpair, so shards carry the same fidelity as the current transport.🔜 Coming in PR B — wire durable transport (the behavior change)
See draft here: nina-msft#3
PR A is inert on its own. The follow-up (
dev/nina-msft/xdist-durable-transport, stacked on this branch) turns the primitives on:pytest_configure_nodechannel (SHARD_DIR_KEY)."transport": "shard".workeroutputis slimmed to a completeness sentinel (result bodies removed) andhandle_testnodedownswitches to reading results from shards — these two changes must land together, which is exactly why they're held out of this PR.workeroutput(so single-process and remote runs are unaffected).pytestertests proving the wiring: worker-kill recovery, oversized-record isolation, crash-item-absent, and clean-run parity; plus thedocs/usage/xdist.mdupdate.Splitting here keeps the "flip the source of truth" as one reviewable semantic change and keeps this PR a zero-risk, additive foundation.
Breaking changes
None. No existing code path calls any of the new symbols; behavior is identical to
main.Checklist
pre-commit run --all-filespasses