Skip to content

Preserve storage offsets when replaying nested subquery moves#4699

Open
joshdchang wants to merge 6 commits into
electric-sql:rob/restore-subqueries-bug-5from
ai-sapien:josh/nested-subquery-replay-offsets
Open

Preserve storage offsets when replaying nested subquery moves#4699
joshdchang wants to merge 6 commits into
electric-sql:rob/restore-subqueries-bug-5from
ai-sapien:josh/nested-subquery-replay-offsets

Conversation

@joshdchang

Copy link
Copy Markdown

Summary

This is a focused follow-up stacked on #4681. It preserves authoritative shape-log offsets while replaying missed materializer moves, fixing nested subquery dependencies whose generated control messages and spliced move-in rows do not contain lsn / op_position JSON headers.

Problem

#4681 correctly persists each outer consumer's per-dependency move position and asks the dependency materializer to replay the missing range after restart. The initial decoder reconstructs each replay offset from JSON headers, however:

  • generated move-in / move-out controls have no source LSN headers;
  • spliced move-in snapshot rows have no source LSN headers;
  • snapshot-end controls may occur between visible replay records;
  • shape-log offsets are sparse, so incrementing the previous visible offset is not authoritative.

Nested materializers can therefore emit a replayed move with the wrong position, advance the outer cursor incorrectly, and miss a later move after another restart.

Changes

  • Add an optional get_log_stream_with_offsets/3 storage capability.
  • Preserve exact offsets through in-memory storage and PureFileStorage's ETS, disk, hibernated, and mixed disk/ETS paths.
  • Give the new reader strict lower-exclusive / upper-inclusive bounds while retaining legacy get_log_stream/3 chunk behavior.
  • Reconstruct the materializer seed exactly at the persisted dependency cursor and lazily replay only the missing tail.
  • Stop bounded disk reads immediately at the inclusive max instead of scanning the remainder of the file.
  • Fail with a clear storage capability error for custom adapters that do not implement offset-preserving replay.

The storage format and HTTP Shape API are unchanged.

Tests

  • storage conformance for snapshots, sparse ordinary records, generated controls, spliced move-in rows, live ETS, disk after hibernation, and mixed disk/ETS bounds;
  • read-count regression proving a bounded disk stream stops after its max;
  • materializer seed/replay tests for offset-less move-out controls and move-in rows;
  • repeated three-level nested subquery move-out/move-in oracle across three restarts;
  • existing 200-transaction restart catch-up oracle.

Locally passed:

  • mix format --check-formatted
  • mix compile --force --warnings-as-errors
  • 206 focused storage/materializer/consumer tests on the Sapien integration branch
  • full sync-service suite
  • restore oracle suite
  • 100-shape / 500-mutation property oracle

This PR is intentionally based on rob/restore-subqueries-bug-5, so its diff contains only the nested-offset follow-up and can be folded into #4681 independently of Sapien's fork.

robacourt and others added 5 commits July 14, 2026 09:23
…l timeout

The per-dependency moves-position is now staged when the move pipeline drains
and only committed+persisted once the writer confirms the flush has passed the
move's splice (and, at terminate, after the writer has been flushed). This
keeps the persisted position from running ahead of durable storage across a
restart, which could otherwise leave a subquery shape permanently missing rows.

Also raise the oracle-restore test's long_poll_timeout: a very short one trips
a separate post-restart long-poll readiness race (bug 3), unrelated to the
subquery-restore behaviour under test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It is called from terminate/2 after terminate_writer/1 has popped :writer off
the state, so the value is no longer typed as %State{} and the %State{} clauses
were flagged as never matching. Match a bare map and read the fields via
Map.get instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@robacourt
robacourt force-pushed the rob/restore-subqueries-bug-5 branch from fdf08f4 to aafa5c1 Compare July 14, 2026 08:30
@joshdchang
joshdchang force-pushed the josh/nested-subquery-replay-offsets branch from fcbdac6 to a5981ea Compare July 14, 2026 12:57
@joshdchang
joshdchang marked this pull request as ready for review July 15, 2026 02:53
@robacourt
robacourt force-pushed the rob/restore-subqueries-bug-5 branch 4 times, most recently from 2262fc6 to 2a1a1d1 Compare July 16, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants