Skip to content

fix(sync-service): cold nested subquery shape times out subscribing to a dependency materializer (#4715)#4729

Open
robacourt wants to merge 3 commits into
mainfrom
rob/fix-issue-4715
Open

fix(sync-service): cold nested subquery shape times out subscribing to a dependency materializer (#4715)#4729
robacourt wants to merge 3 commits into
mainfrom
rob/fix-issue-4715

Conversation

@robacourt

Copy link
Copy Markdown
Contributor

Summary

A cold nested subquery shape could return an initial HTTP 500 while its dependency snapshots were still in progress. A retry succeeded once the dependency snapshots finished.

Fixes #4715

Problem

When an outer shape's consumer initializes, Consumer.all_materializers_alive?/1 subscribes to each dependency materializer via Materializer.subscribe/1. That was the only materializer call using the default 5s GenServer.call timeout — wait_until_ready/1 and new_changes/3 both use :infinity.

A materializer can't answer any call until handle_continue(:start_materializer) returns, and that blocks on Consumer.await_snapshot_start(..., :infinity) until its own snapshot starts. For a cold nested topology, the intermediate materializer itself waits on a deeper dependency, so its snapshot can start later than 5s after the outer consumer subscribes. The subscribe then times out, Electric removes the outer shape, and the client's first request 500s:

Removing shape "..." due to abnormal shutdown: {:timeout, {GenServer, :call, [#PID, :subscribe, 5000]}}
  ... consumer.ex: Electric.Shapes.Consumer.all_materializers_alive?/1
  ... consumer.ex: Electric.Shapes.Consumer.finish_initialization/3

This matches the logs reported in #4715.

Solution

Materializer.subscribe/* now waits with :infinity, consistent with the other materializer calls. This matches the intended semantics from the issue — "a single cold nested shape waits for dependency materializers and completes without an externally visible 500."

Why this is safe:

  • Liveness is already handled by the caller: all_materializers_alive?/1 does Process.monitor(pid, ...) before subscribing, so a dead materializer surfaces as a call exit rather than being masked by a short timeout.
  • No deadlock: the shape-dependency graph is a DAG (outer → inner), so no dependency can transitively wait on the outer consumer.
  • Replication is not stalled by the longer block: the outer shape is registered with the ShapeLogCollector's EventRouter (via SetupEffectsShapeLogCollector.add_shape) only after all_materializers_alive?/1 returns, inside initialize_event_handler. While the consumer is blocked in the subscribe it is not yet a routing target, so the collector never broadcasts a transaction to it. The overall wait is bounded by the request-level await_snapshot_start(outer, 45000) budget.

Reproduction / Test Plan

Added a :slow router test (test/electric/plug/router_test.exs) that deterministically reproduces the bug by stalling only the dependency shape's snapshot past the 5s subscribe window (via an injected :create_snapshot_fn), then asserting the cold nested shape still serves 200.

  • Test fails on main with the {:timeout, {GenServer, :call, [_, :subscribe, 5000]}} signature
  • Test passes with the fix
  • materializer_test.exs, consumer_test.exs, and the /v1/shapes - subqueries router tests pass

Run the reproduction:

mix test test/electric/plug/router_test.exs --include slow --only line:3505

Generated with Claude Code

robacourt and others added 3 commits July 15, 2026 12:43
A cold nested subquery shape can return an initial HTTP 500. The outer
consumer subscribes to each dependency materializer with a hardcoded 5s
`GenServer.call` timeout (`Consumer.all_materializers_alive?/1` ->
`Materializer.subscribe/1`). The dependency materializer stays blocked in
`handle_continue(:start_materializer)` on
`Consumer.await_snapshot_start(..., :infinity)` until its own snapshot
starts, so if that snapshot takes longer than 5s the subscribe times out,
Electric removes the outer shape, and the client's first request fails.

This adds a failing (`:slow`) router test that stalls only the dependency
shape's snapshot past the 5s window and asserts the cold nested shape still
serves 200.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…aterializers (#4715)

`Materializer.subscribe/1` was the only materializer call using the default
5s `GenServer.call` timeout; `wait_until_ready/1` and `new_changes/3` both
use `:infinity`. A materializer cannot answer any call until
`handle_continue(:start_materializer)` returns, and that blocks on
`await_snapshot_start(:infinity)` until its own snapshot starts. When a cold
dependency snapshot takes longer than 5s, the outer consumer's
`all_materializers_alive?/1` subscribe timed out, the outer shape was removed,
and the client's first request returned 500.

Subscribe now waits with `:infinity`. Liveness is already covered by the
caller monitoring the materializer before subscribing, so a dead materializer
surfaces as a call exit rather than being masked by a short timeout. The
dependency graph is a DAG, so there is no deadlock risk.

Flips the `#4715` reproduction test to green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.01%. Comparing base (4051e0f) to head (b2a0d7a).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4729      +/-   ##
==========================================
- Coverage   60.04%   60.01%   -0.03%     
==========================================
  Files         397      397              
  Lines       43766    43766              
  Branches    12586    12588       +2     
==========================================
- Hits        26281    26268      -13     
- Misses      17403    17416      +13     
  Partials       82       82              
Flag Coverage Δ
packages/agents 72.64% <ø> (ø)
packages/agents-mcp 77.70% <ø> (ø)
packages/agents-mobile 80.67% <ø> (ø)
packages/agents-runtime 83.73% <ø> (+0.01%) ⬆️
packages/agents-server 75.47% <ø> (-0.21%) ⬇️
packages/agents-server-ui 8.32% <ø> (ø)
packages/electric-ax 51.06% <ø> (ø)
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/start 82.83% <ø> (ø)
packages/typescript-client 91.78% <ø> (ø)
packages/y-electric 56.05% <ø> (ø)
typescript 60.01% <ø> (-0.03%) ⬇️
unit-tests 60.01% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cold nested subquery shape can time out subscribing to an intermediate dependency materializer

1 participant