feat(server): byte plumbing for the streaming transport, and the spec that lost it - #80
Merged
Conversation
The byte plumbing `TransportMode::Stream` needs, as a standalone unit: read a source into an ordered chunk stream, and receive chunks into a file that the server can vouch for. Nothing is wired to the RPCs yet -- those still refuse explicitly -- so this adds no behaviour an agent can reach. Two properties are the whole point, and both have tests that fail without them: `offset` is checked, not trusted. A receiver handed a chunk claiming offset 0 after writing 900 MB knows the stream restarted, rather than appending and producing a corrupt file of an entirely plausible length. The last chunk carries a blake3 of the whole file and a mismatch refuses the transfer. A truncated transfer is *smaller*, and size is never an accept criterion in this project -- without the hash, a half-received encode installs cleanly over a good original. Reads run on the blocking pool: these are multi-gigabyte files and doing that on a runtime worker would stall every other agent's stream. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9keBRnGXBP2DiQmrcqm33
`distributed-architecture.md` documented only direct mounted access. It contained no mention of upload, download, byte ranges, `TransportMode`, `FetchSource` or `PushOutput`, and stated `MountCovers` as an unconditional requirement -- which made the second transport impossible to express. A node with no usable share was undispatchable no matter how good its encoder was. That was a defect in this document, not a decision. The code faithfully implemented a spec that had dropped the requirement, which is why fixing the code alone would have left the next reader with the same wrong picture. The new section sets out both modes side by side, why the server performs the commit ritual under streaming (the agent has no path to the destination and could not perform it if it wanted to), why `TM_MOUNT` is the proto3 zero value, and why `MountCovers` applies only to mount-mode nodes -- a dispatch-matching change rather than merely a file-copying feature. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9keBRnGXBP2DiQmrcqm33
`Cargo.lock` was left out of the transfer commit -- the `blake3` dependency it adds belongs in the same change, or a fresh clone resolves a different tree than the one the tests ran against. `NEXT-SESSION.md` becomes tracked. It has been untracked by convention, but it is now the only record of how the 26-hour hang was diagnosed, the cross-compile recipe for the Windows agent, and the journal trap waiting in step 3. A working-tree-only file is one `git clean` from gone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9keBRnGXBP2DiQmrcqm33
Tracking NEXT-SESSION.md brought it under Lint Documentation, which requires a language on every fenced block (MD040). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U9keBRnGXBP2DiQmrcqm33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for
TransportMode::Stream. Nothing here is reachable by an agentyet —
FetchSourceandPushOutputstill refuse explicitly. This adds thetransfer primitives they will use, plus the specification section that was
missing.
transcodarr-server::transfersource_streamreads a file into an ordered chunk stream;Sinkreceiveschunks into a file the server can vouch for. Reads run on the blocking pool —
these are multi-gigabyte files, and doing that on a runtime worker would stall
every other agent's stream.
Two properties carry the correctness, and each has a test that fails without it:
offsetis checked, not trusted. A receiver handed a chunk claimingoffset 0 after writing 900 MB knows the stream restarted, rather than
appending and producing a corrupt file of an entirely plausible length.
bytes are used. A truncated transfer is smaller, and size is never an accept
criterion in this project — without the hash, a half-received encode installs
cleanly over a good original.
The round-trip test uses a body of
CHUNK_BYTES * 2 + 12345deliberately: not amultiple of the chunk size, so an off-by-one in the final partial read turns it
red.
The document
distributed-architecture.mdhad zero mentions of upload, download, byteranges,
TransportMode,FetchSourceorPushOutput, and statedMountCoversas unconditional — which made the second transport impossible to express. A node
with no usable share was undispatchable no matter how good its encoder was.
That was a defect in the document, not a decision: the code faithfully
implemented a spec that had dropped the requirement. Fixing only the code would
have left the next reader with the same wrong picture, which is why
todo.d/2026-08-10-streaming-transport-mode.mdsays to fix the document first.Still to come
The RPC wiring (
FetchSource/PushOutputonAgentSession), the server-sideinstall reusing
runner.rs's ritual, and the agent-side stream path inworker.rs. Called out so this is not mistaken for a working transport.531 tests pass, 36s.
🤖 Generated with Claude Code
https://claude.ai/code/session_01U9keBRnGXBP2DiQmrcqm33