fix(ws): skip tier:fallback escape on TipHW tip re-fetch - #13
Conversation
xray — see through AI slop with deterministic architecture PR diff reviews |
0dc4636 to
abe25ab
Compare
Empty tip races on healthy primaries were fanning out to pay-per-call fallbacks (Infura etc.) via the emptyish escape hatch. Latest TipHW tip re-fetch now sets SkipFallbackEscape so TipHW still refuses stale latest without burning 3P quota; finalized re-fetch keeps escape for HA. Direct client concrete-block escapes are unchanged.
abe25ab to
a785c15
Compare
Cross-node tip race: TipHW from reth-0 WS then getBlock on lagging reth-1 returns null. Record tip-source on SuggestLatestBlock, bump HTTP twin poller, and pin EnforceHighestBlock re-fetch to that source (fallback EvmLeaderUpstream). Keep SkipFallbackEscape so misses refuse-stale instead of burning Infura. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop tip-source registry. Bumping the *-ws-* HTTP twin on SuggestLatestBlock is enough for existing EvmLeaderUpstream / partition routing. Keep SkipFallbackEscape for Infura. Co-authored-by: Cursor <cursoragent@cursor.com>
Tip routing stays on EvmLeaderUpstream (PR11). This PR only stops tip re-fetch from escaping to tier:fallback when internals miss. Co-authored-by: Cursor <cursoragent@cursor.com>
The first tip re-fetch pins to EvmLeaderUpstream only when the leader poller already owns TipHW at resolve time. When TipHW was adopted from Redis before the local WS delivery, that resolve fails, the first re-fetch goes out with only a stale-responder exclusion, and the second re-fetch dropped the pin entirely — sweeping siblings that are provably one block behind (cross-node live test: 12/12 null at t=0, catch-up in 115-193ms; the announcing node serves its own head in ~0ms). Resolve the leader again before the second re-fetch when the first was not leader-pinned: the leader has had the first Forward's retry budget plus a forced poll (PollLatestBlockNumberNow bypasses debounce) to catch up, so the pin lands on the node that has the block. If the first re-fetch WAS leader-pinned and still missed, the leader cannot serve — keep the unpinned primary sweep as before. Fallback escape stays suppressed on both paths. Adds a discriminating test: the lagging stale responder must receive zero concrete-tip fetches once the caught-up leader is pinned on the second re-fetch (without the re-resolve, the unpinned sweep hits it first). Gock filter counters are gated on r.URL.Host because filters also run while matching requests bound for other hosts.
|
Added a5ebf70: the tip re-fetch now uses the leader logic on both attempts.
New test TestHttpServer_GetBlockByNumberLatest_SecondRefetchResolvesLeaderPin: leader poller lags TipHW at first resolve, catches up after its first concrete-tip miss; asserts the stale responder receives zero concrete-tip fetches and the tip is served via the pinned second re-fetch. Note for reviewers: gock Filter counters must be gated on r.URL.Host - filters also execute while gock matches requests bound for other hosts, which silently inflates hit counters. |
On first tip re-fetch miss, log tipHW, stale upstream, leader id/latest, whether pass-1 was leader-pinned, pin directives, and recovery/refuse outcome so canary can prove whether empties are pinned-leader misses or pin-empty Redis races. Co-authored-by: Cursor <cursoragent@cursor.com>
Direct client requests for a concrete tip block are the majority of the fallback-escape volume (evm:1, 6h: 66k getBlockByNumber escapes vs at most 31k attributable to enforcement re-fetches) and were untouched by the SkipFallbackEscape directive, which only covers the TipHW re-fetch. Generalize the guard in the escape hatch: when the requested block is at or one ahead of the primary leader's poller, the miss is the sibling import race (measured 100-200ms) — primaries serve it within a block time and pay-per-call fallbacks share the same race, so escaping buys nothing. The failsafe retry (emptyResultDelay / blockUnavailableDelay) re-visits primaries instead. Scope guards: blocks further ahead of the leader (primaries genuinely stuck) still escape — that is the HA case, covered by the existing gate-skip subtests; older-block data gaps still escape; block-less methods (eth_getTransactionReceipt) still escape. Repurposes EscapesOnEmptyishGetBlockByNumber into NearTipEmptyishDoesNotEscape: same fixture (primaries at tip returning null for it) now asserts the escape counter does not move and the response is not served by a fallback. Note: TestFailover_GateSkipsAccumulateErrorRate is flaky on the base branch (fails 2/3 runs, errorRate lands exactly on the 0.7 threshold depending on poller timing) — unrelated to this change.
|
Added 93f5b00: generalizes the guard to direct client requests - the majority of the escape volume (evm:1 over 6h: 66k getBlockByNumber escapes, at most 31k attributable to enforcement re-fetches; the rest are direct concrete-tip requests the directive-based guard never saw). 18 lines in the escape hatch: when the requested block is at or one ahead of the primary leader's poller, the miss is the sibling import race (measured 100-200ms on our reths) - primaries serve it within a block time and paid fallbacks share the same race, so escaping buys nothing. Failsafe retry re-visits primaries instead. Deliberately NOT suppressed (HA preserved):
With this, expected coverage goes from ~36% of escapes (re-fetch only) to ~77% (re-fetch + direct getBlockByNumber/eth_call at tip). Receipts (~17%) remain. Heads up: TestFailover_GateSkipsAccumulateErrorRate is flaky on the base branch (2/3 failures, errorRate lands exactly on the 0.7 threshold depending on poller timing) - pre-existing, not from these commits. |
|
Superseded by #14 — minimal pin of near-tip |
Summary
TipHW tip re-fetch (
forwardGetBlockByNumberon latest only) setsSkipFallbackEscape, so empty tip races on healthy primaries refuse-stale instead of escalating totier:fallback(Infura / Chainstack / dRPC).Tip routing stays on existing EvmLeaderUpstream pin (PR11) — no HTTP-twin id rewriting.
Why
After
ws-tip-http-floor-test→test8, TipHW re-fetch escapes into paid fallbacks and burns Infura quota. Leader pin is the routing fix already on this branch; this PR is the cost stopgap when that miss still happens.Test plan
TestHttpServer_GetBlockByNumberLatest_TipRefetchSkipsFallbackEscapeTestHttpServer_GetBlockByNumberLatest_RefetchPinsEvmLeaderUpstreamgo test ./erpc/ -run 'TestHttpServer_GetBlockByNumberLatest_'