Skip to content

[FEAT]: Durable xdist shard transport primitives (no wiring)#113

Open
nina-msft wants to merge 1 commit into
microsoft:mainfrom
nina-msft:dev/nina-msft/xdist-shard-primitives
Open

[FEAT]: Durable xdist shard transport primitives (no wiring)#113
nina-msft wants to merge 1 commit into
microsoft:mainfrom
nina-msft:dev/nina-msft/xdist-shard-primitives

Conversation

@nina-msft

@nina-msft nina-msft commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Description

First PR in a stack of two that adds durable result transport for pytest-xdist, closing two gaps in the current in-memory workeroutput transport:

  • Durability — a worker killed mid-run currently loses all of its finished results (nothing is written until a clean pytest_sessionfinish).
  • Granular size cap — a single oversized transcript currently replaces a whole worker's payload with a truncation marker, discarding every other result from that worker.

This PR lands only the primitives for the fix. Every symbol is a pure addition to rampart/pytest_plugin/_xdist.py with no call sites — existing behavior is completely unchanged. It establishes the vocabulary the wiring PR consumes, so it can be reviewed small and green on its own.

What's added

Symbol Purpose
ShardWriter Append-only, flush-per-result JSONL writer. A worker's finished results hit disk immediately, so they survive an abrupt worker death (e.g. SIGKILL). An oversized record is written as a marker line naming its nodeid/index instead of discarding the whole shard.
read_worker_shard Controller-side recovery. Parses a worker shard line by line, tolerates a truncated/partial final line, and surfaces corrupt/oversized records as DroppedRecord entries so the caller can mark the run incomplete. nodeid/index are taken authoritatively from the record envelope, mirroring deserialize_worker_data's existing trust posture (never trusting values embedded in the serialized result body).
shard_eligible Gates shard transport to runs where every --tx gateway is a local popen worker (shared filesystem). Remote (ssh=/socket=) or proxied (via=) topologies are ineligible and will fall back to workeroutput in PR B.
SHARD_DIR_KEY Stash key constant for the controller→worker shard-directory channel used in PR B.

The full-fidelity Result round-trip reuses the existing _serialize_result/_deserialize_result pair, so shards carry the same fidelity as the current transport.

🔜 Coming in PR B — wire durable transport (the behavior change)

See draft here: nina-msft#3

PR A is inert on its own. The follow-up (dev/nina-msft/xdist-durable-transport, stacked on this branch) turns the primitives on:

  • Controller creates the shard directory and hands it to workers via a pytest_configure_node channel (SHARD_DIR_KEY).
  • Workers append each result to their shard as it completes and stamp "transport": "shard".
  • The atomic flip: workeroutput is slimmed to a completeness sentinel (result bodies removed) and handle_testnodedown switches to reading results from shards — these two changes must land together, which is exactly why they're held out of this PR.
  • Mandatory remote/non-eligible fallback to workeroutput (so single-process and remote runs are unaffected).
  • Slow pytester tests proving the wiring: worker-kill recovery, oversized-record isolation, crash-item-absent, and clean-run parity; plus the docs/usage/xdist.md update.

Splitting here keeps the "flip the source of truth" as one reviewable semantic change and keeps this PR a zero-risk, additive foundation.

Breaking changes

None. No existing code path calls any of the new symbols; behavior is identical to main.

Checklist

  • pre-commit run --all-files passes
  • Tests added or updated for changes
  • Documentation updated

Introduces the building blocks for durable per-worker result transport
under pytest-xdist, addressing the durability (#3) and granular
size-cap (microsoft#4) gaps in the workeroutput transport. These are pure
additions with no call sites -- existing behavior is unchanged; the
wiring lands in a follow-up PR.

- ShardWriter: append-only, flush-per-result JSONL writer so results a
  worker has already produced survive an abrupt worker death. Oversized
  records become a marker line naming the dropped nodeid/index rather
  than discarding the whole shard.
- read_worker_shard: line-by-line recovery tolerating a truncated final
  line; corrupt/oversized records are surfaced as DroppedRecord entries
  so the caller can mark the run incomplete. nodeid/index are taken
  authoritatively from the record envelope, mirroring
  deserialize_worker_data's trust posture.
- shard_eligible: gates shard transport to runs where every --tx gateway
  is a local popen worker (shared filesystem); remote/proxied topologies
  fall back to workeroutput.
- SHARD_DIR_KEY stash constant.

Adds 25 unit tests covering eligibility, write/flush/oversize, recovery,
corruption tolerance, authoritative ordering metadata, and full-fidelity
round-trip.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant