confirm: loadq=8 + IBD RAM meters (one RecentCreates Arc) - #134
Merged
Conversation
residual≈1.7GiB on the host was a meter hole. Count PublishedIds layers, hash→height, fence runs, and the duplicate RecentCreates maps separately so accounted includes them. No behavior change.
Inner.live was a full clone of the 1.6M-key map beside ArcSwap (~150–200MiB extra, plus a 680ms write memcpy). Overlay/tombstones already hold unflushed notes. publish_if_dirty now builds one Arc from published ⊕ overlay ⊖ dead. Steady-state recent RAM is one map + fifo. leftover get/stamp contract unchanged.
rearden-grok
Bot
force-pushed
the
confirm/ibd-ram
branch
from
August 18, 2026 18:38
666e00a to
375319c
Compare
unresolved_heights skipped missing path_lo and resolved later heights. Lookup is in-order again: walk from path_lo and break on the first height not on the queue. Complete / skip are not a gap.
Lookup will fill eight load-sized batches (same pack_stop_after as 8000 inputs / 144 blocks). Claim-ahead includes the new cap. The channel is not wired yet.
take_raw removes the row and returns the payload. lookup_taken_hi is the contiguous consume water-mark (MAX = none). Densify and offer skip H <= taken_hi so in-flight loadq bodies are not re-getdata'd. Lookup still promotes until loadq is wired.
confirm_bq_resolve_wave takes raw rows on emit (hold leaves them) and returns decoded items. Lookup slices those into load-sized batches and blocking-sends onto loadq. Load recvs; it no longer packs or re-decodes from BQ. Densify already skips taken_hi.
Drop 2×soft_win (floor 256). Horizon is clamp(ewma+ewma/4, 32, 32*144) so the ring matches the in-order pipeline, not the 1-min BQ window. LiveUnion keep uses the same lead span.
progress/perf/sizes print the real lookup→load channel. accounted includes loadq wire. #134 union=/h2h=/fence=/recent= tokens stay.
Owner docs match the shipped pipeline: lookup dequeues into loadq, densify uses lookup_taken_hi, RecentCreates is no longer 2×soft_win.
rearden-grok
Bot
changed the title
CI -Dwarnings failed: take_raw used a single-arm match, and four pack/claim helpers were test-only after loadq replaced BQ pack.
The DEBUG perf line uses format_conf_q. After loadq replaced the fake ready= queue token the pin still looked for ready=0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Lookup is fat again (decode + precompute + TipOnly). Load was gated on BQ
ready=/ promote-in-place. Process RSS meters were also a hole (residualand a duplicate RecentCreates map).
What
RAM (#134 original):
ibd: sizesnamesunion=/h2h=/fence=/recent= live=/pub=/ov= fifo=(inaccounted).Lookup→load queue:
3. Lookup walks BQ in order from
path_lo(stops at the first hole).4. On emit, lookup dequeues raw and sends load-sized batches on
loadq=8.5. Load recvs; it does not pack or re-decode from BQ.
6. Densify / offer skip
H <= lookup_taken_hi.7. RecentCreates horizon is EWMA(
lookup_taken_hi − tip)+25% (floor 32, cap 32×144).8. Logs:
loadq=n/8;ready=/bq_dec=are not queue tokens. LiveUnion stays.Test
cargo test -p rbitcoin-store --lib unresolved_heightscargo test -p rbitcoin-query --lib lookup_take recent_createscargo test -p rbitcoin-consensus --lib bq_resolvecargo test -p rbitcoin-net --lib split_wave load_recv lookup_blocks format_sizes confirm_engineDo not merge unless asked.