Skip to content

fix(ws): skip tier:fallback escape on TipHW tip re-fetch - #13

Draft
shpookas wants to merge 7 commits into
feat/websocket-supportfrom
fix/skip-fallback-escape-on-tip-refetch
Draft

fix(ws): skip tier:fallback escape on TipHW tip re-fetch#13
shpookas wants to merge 7 commits into
feat/websocket-supportfrom
fix/skip-fallback-escape-on-tip-refetch

Conversation

@shpookas

@shpookas shpookas commented Jul 30, 2026

Copy link
Copy Markdown

Summary

TipHW tip re-fetch (forwardGetBlockByNumber on latest only) sets SkipFallbackEscape, so empty tip races on healthy primaries refuse-stale instead of escalating to tier:fallback (Infura / Chainstack / dRPC).

Tip routing stays on existing EvmLeaderUpstream pin (PR11) — no HTTP-twin id rewriting.

Why

After ws-tip-http-floor-testtest8, TipHW re-fetch escapes into paid fallbacks and burns Infura quota. Leader pin is the routing fix already on this branch; this PR is the cost stopgap when that miss still happens.

Test plan

  • TestHttpServer_GetBlockByNumberLatest_TipRefetchSkipsFallbackEscape
  • TestHttpServer_GetBlockByNumberLatest_RefetchPinsEvmLeaderUpstream
  • go test ./erpc/ -run 'TestHttpServer_GetBlockByNumberLatest_'
  • Canary roll

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown
File Lines Key changes Risk
🔵 eth_getBlockByNumber.go +93/-18
🔵 networks.go +28/-2
2 test files +395

xray — see through AI slop with deterministic architecture PR diff reviews

@shpookas
shpookas force-pushed the fix/skip-fallback-escape-on-tip-refetch branch from 0dc4636 to abe25ab Compare July 30, 2026 12:39
Empty tip races on healthy primaries were fanning out to pay-per-call
fallbacks (Infura etc.) via the emptyish escape hatch. Latest TipHW tip
re-fetch now sets SkipFallbackEscape so TipHW still refuses stale latest
without burning 3P quota; finalized re-fetch keeps escape for HA.
Direct client concrete-block escapes are unchanged.
@shpookas
shpookas force-pushed the fix/skip-fallback-escape-on-tip-refetch branch from abe25ab to a785c15 Compare July 30, 2026 12:39
Cross-node tip race: TipHW from reth-0 WS then getBlock on lagging reth-1
returns null. Record tip-source on SuggestLatestBlock, bump HTTP twin
poller, and pin EnforceHighestBlock re-fetch to that source (fallback
EvmLeaderUpstream). Keep SkipFallbackEscape so misses refuse-stale
instead of burning Infura.

Co-authored-by: Cursor <cursoragent@cursor.com>
@shpookas shpookas changed the title fix(ws): skip tier:fallback escape on TipHW tip re-fetch fix(ws): pin tip re-fetch to WS tip-source (PR11-shaped) Jul 30, 2026
Drop tip-source registry. Bumping the *-ws-* HTTP twin on SuggestLatestBlock
is enough for existing EvmLeaderUpstream / partition routing. Keep
SkipFallbackEscape for Infura.

Co-authored-by: Cursor <cursoragent@cursor.com>
@shpookas shpookas changed the title fix(ws): pin tip re-fetch to WS tip-source (PR11-shaped) fix(ws): bump HTTP twin on WS tip + skip fallback escape Jul 30, 2026
Tip routing stays on EvmLeaderUpstream (PR11). This PR only stops tip
re-fetch from escaping to tier:fallback when internals miss.

Co-authored-by: Cursor <cursoragent@cursor.com>
@shpookas shpookas changed the title fix(ws): bump HTTP twin on WS tip + skip fallback escape fix(ws): skip tier:fallback escape on TipHW tip re-fetch Jul 30, 2026
The first tip re-fetch pins to EvmLeaderUpstream only when the leader
poller already owns TipHW at resolve time. When TipHW was adopted from
Redis before the local WS delivery, that resolve fails, the first
re-fetch goes out with only a stale-responder exclusion, and the second
re-fetch dropped the pin entirely — sweeping siblings that are provably
one block behind (cross-node live test: 12/12 null at t=0, catch-up in
115-193ms; the announcing node serves its own head in ~0ms).

Resolve the leader again before the second re-fetch when the first was
not leader-pinned: the leader has had the first Forward's retry budget
plus a forced poll (PollLatestBlockNumberNow bypasses debounce) to
catch up, so the pin lands on the node that has the block. If the
first re-fetch WAS leader-pinned and still missed, the leader cannot
serve — keep the unpinned primary sweep as before. Fallback escape
stays suppressed on both paths.

Adds a discriminating test: the lagging stale responder must receive
zero concrete-tip fetches once the caught-up leader is pinned on the
second re-fetch (without the re-resolve, the unpinned sweep hits it
first). Gock filter counters are gated on r.URL.Host because filters
also run while matching requests bound for other hosts.
@shpookas

Copy link
Copy Markdown
Author

Added a5ebf70: the tip re-fetch now uses the leader logic on both attempts.

  • First re-fetch already pinned to EvmLeaderUpstream when its poller owns TipHW (unchanged).
  • When that resolve fails (TipHW adopted from Redis before the local WS delivery), the second re-fetch now re-resolves the leader instead of sweeping unpinned. By then the leader has had the first Forward's retry budget plus a forced poll (PollLatestBlockNumberNow, no debounce) to catch up, so the pin lands on the node that delivered the head. Cross-node live test on eth-mainnet reths: sibling is null 12/12 at announce time and catches up in 115-193ms, while the announcer serves its own head immediately - so pinning to the deliverer is always correct and sweeping siblings never is.
  • If the first re-fetch WAS leader-pinned and still missed, the leader genuinely cannot serve; the unpinned primary sweep is kept for that case. SkipFallbackEscape stays on both paths.

New test TestHttpServer_GetBlockByNumberLatest_SecondRefetchResolvesLeaderPin: leader poller lags TipHW at first resolve, catches up after its first concrete-tip miss; asserts the stale responder receives zero concrete-tip fetches and the tip is served via the pinned second re-fetch. Note for reviewers: gock Filter counters must be gated on r.URL.Host - filters also execute while gock matches requests bound for other hosts, which silently inflates hit counters.

shpookas and others added 2 commits July 31, 2026 10:57
On first tip re-fetch miss, log tipHW, stale upstream, leader id/latest,
whether pass-1 was leader-pinned, pin directives, and recovery/refuse
outcome so canary can prove whether empties are pinned-leader misses
or pin-empty Redis races.

Co-authored-by: Cursor <cursoragent@cursor.com>
Direct client requests for a concrete tip block are the majority of the
fallback-escape volume (evm:1, 6h: 66k getBlockByNumber escapes vs at
most 31k attributable to enforcement re-fetches) and were untouched by
the SkipFallbackEscape directive, which only covers the TipHW re-fetch.

Generalize the guard in the escape hatch: when the requested block is at
or one ahead of the primary leader's poller, the miss is the sibling
import race (measured 100-200ms) — primaries serve it within a block
time and pay-per-call fallbacks share the same race, so escaping buys
nothing. The failsafe retry (emptyResultDelay / blockUnavailableDelay)
re-visits primaries instead.

Scope guards: blocks further ahead of the leader (primaries genuinely
stuck) still escape — that is the HA case, covered by the existing
gate-skip subtests; older-block data gaps still escape; block-less
methods (eth_getTransactionReceipt) still escape.

Repurposes EscapesOnEmptyishGetBlockByNumber into
NearTipEmptyishDoesNotEscape: same fixture (primaries at tip returning
null for it) now asserts the escape counter does not move and the
response is not served by a fallback.

Note: TestFailover_GateSkipsAccumulateErrorRate is flaky on the base
branch (fails 2/3 runs, errorRate lands exactly on the 0.7 threshold
depending on poller timing) — unrelated to this change.
@shpookas

Copy link
Copy Markdown
Author

Added 93f5b00: generalizes the guard to direct client requests - the majority of the escape volume (evm:1 over 6h: 66k getBlockByNumber escapes, at most 31k attributable to enforcement re-fetches; the rest are direct concrete-tip requests the directive-based guard never saw).

18 lines in the escape hatch: when the requested block is at or one ahead of the primary leader's poller, the miss is the sibling import race (measured 100-200ms on our reths) - primaries serve it within a block time and paid fallbacks share the same race, so escaping buys nothing. Failsafe retry re-visits primaries instead.

Deliberately NOT suppressed (HA preserved):

  • blocks further ahead of the leader (primaries genuinely stuck) - existing gate-skip subtests cover this
  • older-block data gaps
  • block-less methods (receipts) - those still escape; separate policy question

With this, expected coverage goes from ~36% of escapes (re-fetch only) to ~77% (re-fetch + direct getBlockByNumber/eth_call at tip). Receipts (~17%) remain.

Heads up: TestFailover_GateSkipsAccumulateErrorRate is flaky on the base branch (2/3 failures, errorRate lands exactly on the 0.7 threshold depending on poller timing) - pre-existing, not from these commits.

@shpookas

Copy link
Copy Markdown
Author

Superseded by #14 — minimal pin of near-tip eth_getBlockByNumber to EvmLeaderUpstream on first forward (PR11-shaped). Escape-hatch / SkipFallbackEscape changes dropped per review.

@shpookas
shpookas marked this pull request as draft July 31, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant