feat(replica): portable heartbeat-token fence with snapshot-local reader routing - #3268
Open
tlongwell-block wants to merge 11 commits into
Open
feat(replica): portable heartbeat-token fence with snapshot-local reader routing#3268tlongwell-block wants to merge 11 commits into
tlongwell-block wants to merge 11 commits into
Conversation
Single CHECK'd-row table whose monotonic token, committed after the fence's ordered writer scan, gives readers a freshness observation that Aurora reader endpoints can actually serve (unlike pg_last_wal_replay_lsn, which they hide). The epoch column detects token resets. Registered as operator-global: it describes replication topology, not tenant data. Numbered 0026: version 25 was burned by 0025_users_agent_owner_lookup (#2615, reverted by #3168). DBs migrated during that window have version 25 recorded with those bytes; a different 0025 would fail their startup with VersionMismatch. 25 stays vacant as a tombstone. Migration shape assertions extended; the operator-global allowlist in the tenant-scoping lint gains the new table. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Pull in ~40 commits landed since work began (per Tyler, merge not rebase), including the #2615 revert that vacated migration version 25. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz> * origin/main: (35 commits) fix(desktop): keep project Inbox previews compact (#3193) Inbox refactor (#2045) Fix composer selection formatting and drop overlay (#3172) Refine pending message status (#3153) feat(admin): show reported message content in report detail (#3149) fix(desktop): recover full local storage on startup (#3182) Replace mobile reconnect banners with skeleton shimmer (#3143) fix(desktop): keep collapsed table separators out of spoilers (#3169) chore(deps): update plugin org.jetbrains.kotlin.android to v2.2.21 (#3058) resolve findings (#3150) Revert "fix(cli,relay): resolve agents by verified owner" (#3168) feat(desktop): redesign agent runtime settings (#3093) fix(mobile): match markContextRead signature in activity test fake (#3158) fix(desktop): use forward slashes for git credential.helper on Windows (#3023) fix(mobile): tapping threaded message in Inbox navigates to top level of channel (#2103) fix(mobile): retry channel-sections startup sync when relay rate-limits cold start (#3004) chore(desktop): add AgentCreationPreview file-size override to unblock main CI (#3154) fix(cli,relay): resolve agents by verified owner (#2615) fix(desktop): make the test loader work on Windows (#2758) fix(desktop): make lint and unit-test gates work on Windows (#2943) ...
The 0025 slot was briefly occupied by 0025_users_agent_owner_lookup (#2615), but it lived on main for only ~1 hour before the revert (#3168) and never reached any deployed image (prod rolled dd222a5 -> sha-137185e, both post-revert or pre-add). No production database has version 25 recorded, so the number is free; a dev DB migrated inside that window is already broken on current main (VersionMissing) and is repaired by deleting its stale version-25 row. Per Tyler in thread 39d1e174. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
0025 collision resolved: main landed 0025_relay_invites (#3141) while this branch held 0025_replica_heartbeat. Heartbeat migration renumbered to 0026 (published migration bytes are immutable; two same-version migrations merge cleanly in git but kill the sqlx migrator on a fresh DB). Also adds the replica_heartbeat table to schema/schema.sql (desired-state), which the original commit missed. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
…ting
Replace the WAL-LSN read-side observation (pg_last_wal_replay_lsn, which
Aurora reader endpoints hide) with a portable heartbeat token, and make
the freshness proof connection-local per Rev 2.
Probe (replica_fence.rs): the ordered writer-side scan is unchanged and
load-bearing (S = clock_timestamp -> pg_stat_activity oldest-xact scan,
masked-visibility fail-closed) but now ends by committing a heartbeat
token LAST on the same pinned connection (single-row UPDATE ... RETURNING
token, epoch on replica_heartbeat, migration 0026). The single-row update
serializes all pods' probes, so tokens are globally commit-ordered and a
reader session observing token >= M has replayed everything the
three-bucket argument covers for M. Each probe retains
(token, committed_at, fence_wall) in a bounded ring; epoch changes reset
the ring, and a same-epoch token regression (restore adversary) clears it
and rotates the epoch on the writer so pre-rewind readers fail the epoch
check instead of proving stale coverage.
Routing (lib.rs): eligibility is decided per request ON the serving
connection. A routed read acquires a reader session, observes
token/epoch there (observe_heartbeat, with backend identity for live
evidence), resolves the strongest retained entry <= the observation, and
serves the page on that same session. Cursor pages keep today's
completeness math (Predicate B: cursor_ts <= proved fence_wall; thread
pages keep the full-page + tail-below-wall writer re-verification).
Head fetches are Predicate A (bounded staleness): routed only when
BUZZ_REPLICA_HEAD_MAX_AGE_SECS is set (> 0; default off, clamped to the
fence staleness gate) and the proved entry is within budget. Everything
fails closed to the writer: acquire error, missing heartbeat row, epoch
mismatch, token below the ring, over-budget entry.
Aux closure (bridge.rs): get_channel_window_with_session returns the
serving session as a ReadSession, and the window aux closure runs its
hops on that exact connection - hopping pooled reader sessions would
discard the proof.
Observability: buzz_db_route_decision{path, decision, reason} counters
(channel_head/channel_cursor/thread_head/thread_cursor/thread_eof x
replica/writer x fresh/covered/stale/reader_token_behind/
reader_validation_error/uninitialized/disabled), a
buzz_db_replica_heartbeat_age_seconds gauge, and debug logs with the
proved token and backend identity.
Probe cadence moves 5s -> 1s (head budget needs cadence well under B;
cost is one single-row UPDATE tuple of WAL per beat per pod).
Tests: resolve/record/ring/rotation unit coverage; probe + rotation
against private scratch databases; head-gate routing (off by default,
replica within budget, writer when over budget) and the existing
divergent-fixture routing suite updated to the connection-local flow;
config parse coverage for BUZZ_REPLICA_HEAD_MAX_AGE_SECS.
Co-authored-by: Tyler Longwell <tlongwell@block.xyz>
Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
* origin/main: fix(buzz-acp): accept id-keyed config options when resolving model switch (#2795) fix(desktop): probe legacy Goose install dir on Windows (#3248) refactor(desktop): extract install command execution into install_exec (#3251) Polish composer activity layout and transitions (#3151) Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
…ransaction Wren's review hardening on 17ea2ff: the routed request previously ran proof + page + participants + aux as sequential autocommit statements on one held reader connection (the Rev 2 plan shape, sound via standby snapshot monotonicity). This strengthens the binding from monotone to exact and makes it robust against any future statement-level multiplexing between relay and reader: - proved_reader now opens the request with BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY (the strongest isolation a hot standby supports) and observes the heartbeat as the transaction's FIRST statement, so the snapshot the proof was taken against is exactly the snapshot every follow-up statement reads from. Begin failure fails closed to the writer like every other proof failure. - RouteDecision/ReadSession carry the open transaction; the page, participants batch, and the bridge aux closure all execute inside it. Dropping the ReadSession rolls the read-only transaction back and returns the connection to the pool. - New Postgres-gated regression test routed_request_holds_one_snapshot_across_page_and_aux distinguishes the transaction contract from mere connection reuse: a mid-request commit on the replica is visible to a control autocommit session but must NOT be visible to the held request session. Mutation-verified: swapping the BEGIN to a plain (READ COMMITTED) transaction makes the test fail on exactly the leak assertion. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
Max's E2E finding #2: inet_server_addr() alone is weaker than the plan's production identity tuple and cannot distinguish readers behind one address (or record which backend process served the proof). Blocked the Aurora canary on this fast-follow. - observe_heartbeat now reports `addr:port pid=N` (`local pid=N` on unix sockets), and prefixes the Aurora instance id (`aurora_server_id() @ ...`) when the endpoint supports it. - aurora_server_id() cannot be referenced parse-safely on plain Postgres, so support is probed once per process by reader_supports_aurora_identity on a plain autocommit checkout — never inside the request transaction, where undefined_function would abort it. SQLSTATE 42883 caches a definitive false; transient errors degrade to the plain tuple for that request and retry later. Identity is evidence, never a routing gate. - New Postgres-gated test: the capability probe answers false (not an error) on plain Postgres and leaves the session usable; the existing probe test now pins the addr:port/pid shape. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
* origin/main: Publish symbol-bearing debug relay images (#3250) feat(tracing): add datastore tracing plumbing (#2760) Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
aurora_server_id is the MySQL-family spelling; Aurora PostgreSQL exposes aurora_db_instance_identifier() (AWS Aurora PostgreSQL user guide; awslabs/pg-collector). As written, the capability probe would have hit 42883 on real Aurora, cached a permanent false, and silently stripped the instance id from canary evidence — the exact gap the fast-follow exists to close (Wren, delta review of a472327). The function name now lives in one const (AURORA_IDENTITY_FN) shared by the probe and the observation query, pinned by a unit test so the near-miss cannot recur. The real Aurora canary proves the positive branch. Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz>
* origin/main: feat(relay): make Postgres pool size configurable, default 50 (#3191) Co-authored-by: Tyler Longwell <tlongwell@block.xyz> Signed-off-by: Tyler Longwell <tlongwell@block.xyz> # Conflicts: # crates/buzz-relay/src/main.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expands read-replica usage on the relay per the Rev 2 design (thread 39d1e174 in #buzz-read-only-replica-usage): replaces the Aurora-incompatible WAL-LSN read-side fence observation with a portable heartbeat token, makes the freshness proof snapshot-local to the serving reader session, and adds a default-off bounded-staleness gate for head fetches.
Probe (writer side, unchanged ordering)
The ordered writer scan is kept verbatim —
S = clock_timestamp()→ masked-visibilitypg_stat_activityoldest-xact scan — and now ends by committing a heartbeat token last on the same pinned connection (single-rowUPDATE replica_heartbeat ... RETURNING token, epoch, migration0026). The single-row update serializes all pods' probes, so tokens are globally commit-ordered; the three-bucket completeness argument carries over unchanged. Ring retains(token, committed_at, fence_wall); epoch change resets the ring; a same-epoch token regression (restore adversary) clears the ring and rotates the epoch on the writer so pre-rewind readers fail the epoch check. Cadence 1s.Routing (snapshot-local proof)
Every routed read opens
BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLYon a reader session and observes the heartbeat as the transaction's first statement — the proof's snapshot is exactly the snapshot the page, participants batch, and bridge aux closure read from (ReadSessioncarries the open transaction; drop = rollback). Fail-closed everywhere: begin failure, missing heartbeat row, epoch mismatch, token below the ring, over-budget entry → writer.BUZZ_REPLICA_HEAD_MAX_AGE_SECS(0 = off, clamped to fence staleness). Bounded-stale head semantics are an explicit product decision — do not enable anywhere without Tyler's backdated-event-semantics acceptance.Observability
buzz_db_route_decision{path, decision, reason}across all five paths,buzz_db_replica_heartbeat_age_secondsgauge, and per-decision debug logs carrying the proved token plus backend identity:addr:port pid=N, prefixed withaurora_db_instance_identifier()when the endpoint supports it (probed once per process on an autocommit checkout; SQLSTATE 42883 caches a definitive false; identity is evidence, never a routing gate).Review & verification
5f81b10e5; identity delta re-review approved at exact headfedb46368(90/90 unit, clippy-D warnings, fmt independently reproduced).5f81b10e5— isolated PG17 writer + two streaming standbys behind HAProxy; paused-reader legs split exactly as designed (6 replica/fresh + 6 writer/stale), recovery clean, RR snapshot hardening observed in runtime logs. Evidence:WORK_LOGS/2026-07-28_REPLICA_HEARTBEAT_REPLACEMENT_SHA_E2E.md.mesh_demoflake, red on base; clippy-D warnings+ fmt clean.routed_request_holds_one_snapshot_across_page_and_aux(mutation-verified both directions), same-epoch rotation, capability-probe negative, head-gate truth table, divergent-fixture routing suite.Post-merge plan
Merge publishes the immutable
sha-*main image → bb-public PR pins it → ArgoCD sync → Max runs the production cursor-routing canary on Aurora (positive identity branch proven live). Head gate stays off. Clientsince-widening ships separately.