Erase internal height-indexing and T/B-genericity from synchronization protocol - #37
Merged
Conversation
One runtime representation per backend (Backend::Erased) with paired erase/assume conversions, plus the ErasedNode trait carrying the height-independent observations (span, hash, len). Local's conversions are the phantom wrap/unwrap the typed node already is; the Failing, Charged, and Materializing test backends pass erasure through their wrappers (Charged settles and reopens its ledger entry, so the census peak is untouched). Pure addition: no caller changes, no behavior change. The seam exists for the erased session plumbing (design/height-erasure.md step 1); channels and workers move onto it in subsequent commits. Baseline measured at the parent commit for attribution: cargo llvm-lines --test pairwise (debug, default features) = 2,884,887 lines / 109,613 copies.
Every bounded channel in the materialized walk now carries the height-erased twin of its payload (streaming/erased.rs), behind typed facades minted per edge: both halves of an edge are created at one height parameter, so the walk's stage code keeps exactly the typed surface it had, and the tokio mpsc machinery instantiates once per backend instead of once per height. ErasedPrefix (the prefix bytes without the height tag, length re-checked on every re-tag in debug builds) is the runtime witness traveling with every payload. The one unerased edge is leaf_requests: its item is already the single-height Prefix<Z>. Measured on --test pairwise (debug, default features), parent 2,884,887 lines / 109,613 copies -> 2,609,789 / 94,312 (-9.5%); rows naming tokio's mpsc fall 632,145 -> 234,535 lines. The remaining per-height machinery (the async_stream walk generators at ~400k and the proxy's channels) is the target of the next steps (design/height-erasure.md steps 3-4).
The walk now runs on the erased vocabulary end to end, one instantiation per backend: Query/Resolution/Resolve are generic over the erased node representation (their prefix an ErasedPrefix whose byte length is the height witness), and the level loops, answerer, resolver, assembler, and deletion-honoring filter are shared bodies behind thin typed shells. Each shell erases its stage's request stream on the way in; Work::respond's reply-channel exit re-tags responses at the stage's height on the way out — the walk's only remaining typed boundary, plus the two fixed-height root re-tags at the finish futures. Erased code reaches the height-typed Backend surface through a 33-arm runtime-to-type dispatch (erased::ops), keyed on the prefix length so the coordinate and the witness cannot drift apart; the numbered height aliases it indexes live in typed::height with a pinned-endpoint tripwire. The Unknown trait tower dissolves into one prefix-guided recursion (depth bounded by the 32-byte path, boxed per step exactly as the typed tower was). Step 2's per-payload facades dissolve with it: the channels carry the erased vocabulary bare, and their QueueRole height labels stay runtime data for the instrumented diagnostics and capacity suites. Measured on --test pairwise (debug, default features): 2,609,789 lines / 94,312 copies -> 1,717,836 / 67,158 (parent 2,884,887 / 109,613; -40.5% cumulative). The materialized-labeled rows fall 662k -> 274k and async_stream 400k -> 229k; the remaining per-height machinery is the proxy's (remote-labeled rows: 837k), design/height-erasure.md step 4.
The proxy joins the walk on the erased vocabulary. Scope collapses to
one type whose parent prefix is the height witness; the adapter's
encode/decode workers, the reply pumps, and the encoders are shared
bodies behind thin typed phase methods, with the decoded-reply exit
re-tagging at each stage's height exactly like the walk's respond. Two
new dispatches (erased::ops::{leaves, assemble}) carry the stream-shaped
backend operations, and the proxy's progress trace takes its heights as
the runtime data they already were.
Measured on --test pairwise (debug, default features):
1,717,836 lines / 67,158 copies -> 1,039,534 / 41,172. The
remote-labeled rows fall 837k -> 162k, tokio-mpsc-naming rows to 29k.
Cumulative from the parent: 2,884,887 / 109,613 -> 1,039,534 / 41,172
(-64% lines, -62% copies), completing design/height-erasure.md
steps 1-4. The wire snapshots are byte-identical throughout.
The design doc graduates to .agent-notes as the decision and measurement record: per-step IR readings (2,884,887 -> 1,039,534 lines, -64%, on --test pairwise), the runtime pin (gossip_fixed_bidir_insertions/V2/5000 on ox-east-1 reserved cores: 54.51 ms -> 54.10 ms, no movement), the compile-cost A/B (non-incremental fleet rebuild: 7,819 -> 3,093 CPU-seconds, -60%), the declined step-5 Tagged collapse with its rationale, and the resolutions of the sketch's open questions. The streaming module's layer map gains the erased seam. design/item-erasure.md is the phase-2 sketch (item-type erasure at the leaf boundary), status sketch, awaiting review: the measured residue shows the tree layers now dominate what every downstream binary re-buys, so the sketch weighs erasing the session boundary alone against erasing the tree's stored payload too.
Message already pairs its Arc'd value with the canonical serialized bytes, and the tree and session consume only the bytes outside the typed reads — so erasing the value is an unsizing coercion of the existing Arc, read back by checked downcast, with a per-T deserialize witness at wire ingress as the only other per-T residue. This replaces the decode-on-read variant, which charged every read a CBOR decode to save a fat pointer.
The public observers speak (Version, Arc<T>) and nothing re-exports Message (verified against the public rustdoc surface), so the erased payload reshapes only crate internals; the observers' retained most-recent-leaf slot becomes the per-yield downcast point. The open questions shrink to witness minting, facade sealing, and whether the tree erases in the same stroke.
borrow_next and the lending TryNext existed to avoid cloning a Version whose clone once materialized ITC structure; with Version CoW over shared bytes, a clone is a refcount bump and the lending forms' whole value collapsed to two atomic increments per message. The Stream face becomes the observers' one engine: try_next keeps its non-blocking Message/Quiet/Ended trichotomy but yields the owned (Version, Arc<T>) pair (TryNext loses its lifetime parameter), and the retained most-recent-leaf lending slots dissolve with the borrowing entry points. This also removes the one observer seam the item-erasure sketch had to thread a downcast through (design/item-erasure.md).
Witness at peer construction; non-generic core sealing with the public API unchanged (fall back to erasing generic shells if it turns hairy; legibility outranks purity); tree and session in one stroke; landing on the height-erasure branch.
Message becomes non-generic: { message: Arc<dyn Any + Send + Sync>,
serialized: Bytes } — the caller's own Arc<T> allocation, unsized in
place. Typed constructors sit at the insert boundary; the read boundary
downcasts (message::<T>/arc::<T>), and a payload-type mismatch panics as
a crate-bug tripwire (every message reachable from a typed facade was
constructed with that facade's type).
Owner-ruled public API movement (rulings 5 and 6 in
design/item-erasure.md): Snapshot::get returns Option<Arc<T>> — the
echoed version was always the queried one, since a leaf's path derives
from its version — and iter/range/IntoIterator yield (&Version, Arc<T>)
owned, because a coerced fat pointer has no Arc<T> object to lend.
T: 'static joins the insert-path bounds (safe erasure is TypeId-based,
and Any requires it; gossip already demanded it).
Message equality and hashing now compare the cached serialization; the
Ord and serde-Deserialize forms dissolved (no consumers outside the
type's own tests), and the V1 wire decode routes through the typed
Message::from_reader. The tree stores erased Messages with T phantom on
NodeInner until stage 2 dissolves it; Action de-genericized; the Leaf
seam and the streaming codec's leaf records speak erased Message.
Measured (cargo llvm-lines --test pairwise, debug, default features):
1,039,827 lines / 41,186 copies vs 1,039,534 / 41,172 at the parent —
flat, as this stage predicts: the payload type left storage, but every
instantiation still exists while tree and session stay generic over the
phantom T. The dedup is stage 2's and 3's to collect.
… II stage 2)
The tree's layers drop their payload parameter: the untyped storage
node, the radix fan, the borrowing and owned walks, the typed height
veneer, the traversal trio, and the V1 zipper are all payload-free —
the stored Message was the parameter's only remaining occurrence.
Tree<T> stays as the phantom-typed facade whose faces downcast (stage
1's boundary), and tree::Root goes bare. The V1 protocol messages keep
T as decode context only (a phantom field); their node decode reworks
from a wire::Decode impl tower into the DecodeNode height trait
(read_node::<T>), the typed analog of the streaming codec's
parse_record::<T>.
Sealing, and the measured argument for it: dropping T alone moved
little. cargo llvm-lines counts the named target's crate, so a
binary's total IS its marginal cost, and it fell only 1,039,827 →
971,148 until the batch-apply entry went monomorphic (a Vec of
actions in, a &mut dyn FnMut observer): the generic entry had been
re-instantiating the entire per-height apply tower — radix grouping
included, ~155k lines — in every consumer crate. With act sealed the
way join's shape already was (both towers verify as present in the
lib's codegen and absent from the binary's):
pairwise: 1,039,827 lines / 41,186 copies -> 719,899 / 30,285
(-30.8% per consumer binary, from stage 1)
lib: 32,262 / 1,691 -> ~102k / ~4.5k (paid once, in the rlib)
The residue is the streaming session, still generic over (B, T):
stage 3's scope, and where the erasure's cost/benefit verdict lands.
…er (part II stage 3) The streaming session drops its payload parameter everywhere: Backend, Node, Leaf, and Measure lose T, and with them the message vocabulary, the typestates, the walk and proxy workers, the codec's Frame/LeafRun, the stream binders, and the drivers. The payload type's one residue in a running session is a PayloadDeserializer — a plain fn pointer minted by Message::deserializer::<T>() at Peer construction (seed and bootstrap now carry T: DeserializeOwned; the gossip entry points carry no serde bounds at all) and threaded to each protocol's handshake entry, where wire ingress builds every supplied payload through it (Message::from_wire). A mispaired deserializer fails at ingress — the tripwire caught exactly that during the sweep, when a blanket unit-payload deserializer in the proxy tests met u64 leaves. The V1 oracle joins the same regime rather than staying typed: DecodeNode::read_node and the payload-bearing messages' DecodeWith take the deserializer, recv_msg_with carries it through the framing, and the remote Exchange stores it — which dissolves the phantom decode-context fields stage 2 had introduced and leaves the V1 protocol traits non-generic. One deliberate error-taxonomy change, named per the snapshot policy: the error atlas is re-accepted because record-level trailing payload bytes now classify as a malformed payload (DecodeLeafError::Message(InvalidData)) — the payload runs to the record's end, so the deserializer owns the exactly-one-value check — and the unreachable TrailingBytes variant is gone. The wire snapshots are untouched: this is diagnostics classification, not wire format. Measured (cargo llvm-lines, debug, default features): the lib grows ~102k -> 354,431 lines / 12,716 copies — the session now compiles once, into the rlib — while pairwise sits flat at 719,377 / 30,252, because Peer::<T>'s session-driving methods are still generic funnels: gossip_inner::<u64> monomorphizes in the consumer crate and drags the whole (now T-free, still B-generic) session tower with it. Sealing those entry points non-generic is the next commit, and the number that decides the erasure's cost/benefit.
…ns (part II stage 4)
The reconcile arms of gossip_inner and bootstrap_erased move into
non-generic bodies -- Reconciliation::{v2,v1} and bootstrap_{v2,v1} --
that return their futures boxed. Extraction alone moves nothing: an
async fn body is a closure item codegen'd into whichever crate polls
the future, so a bare non-generic async fn hands the protocol state
machine right back to the consumer crate. The dyn coercion inside this
crate is the seal -- vtable, poll, and everything the body awaits
codegen here -- and #[inline(never)] on the small shells keeps
optimized builds' automatic cross-crate MIR inlining from moving the
coercion back out.
Measured (cargo llvm-lines, debug): pairwise 719,377 -> 207,278 lines
(-71% this stage; -80% across part II); the lib 354,431 -> 867,372,
the towers' once-paid residence. A minimal one-gossip-call downstream
binary: 3,401,418 lines against pre-erasure rumors, 1,095,377 against
the height-erased tree, 34,699 fully erased. Runtime pin:
gossip_fixed_bidir_insertions/V2/5000 at 15.67ms -> 15.62ms,
overlapping confidence intervals (same-machine back-to-back A/B).
… family Message::message (the &T borrow) had no production caller -- every typed read goes through arc -- so the unexercised downcast face dissolves; its panic contract and the provenance argument move onto arc, and the tests exercising it re-target arc. The complement of the payload-evolution suite lands beside it: payload bytes that do not decode as the receiving type fail the session as a clean decode error at the receiver's own ingress -- never a panic, moving nothing into its set. Ingress decodes through the receiving peer's own deserializer, so no wire input can reach the typed-read downcast with a foreign payload; the new tests are the committed demonstration of that claim (an undecodable bootstrap, and an established cross-typed pair failing at the first out-of-range value).
plaidfinch
marked this pull request as ready for review
August 20, 2026 06:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously: downstream consumers recompiled basically all of rumors, because it all got deferred to be codegenned in the consuming binaries, thanks to generics. This makes LLVM sad: we emitted about 3M lines of IR for the biggest test binary. This also makes our test suite sad, and makes it annoying to recompile things that depend on Rumors when they change in development.
This PR erases all the things that cause generic instantiation of large towers of types in the consuming crate, and forces them to be instead processed once, when the library is compiled. It preserves the type-safe tower of heights, etc. in as many places as possible, essentially consolidating all the erased bits into small portions of "TCB". No unsafe is used to do this, and if the TCB has an error, it should panic loudly in development (it does not).
This results in a literal 100x decrease in the LLVM IR generated for a minimal consuming binary of Rumors, and a 12x decrease in wall-clock time related to Rumors' compilation during its own test suite execution against an otherwise warm build.
In the course of this change, this also gets rid of the now-vestigial borrowing API for Rumors event streams, since now both
VersionandArc<T>are cheaply cloned (this was required to internally erase theArcwithout double-boxing).