Skip to content

feat: forward remote dynamic filter updates to coordinator - #635

Open
jayshrivastava wants to merge 2 commits into
js/2-forward-dynamic-filter-updates-to-coordinatorfrom
js/3-plan-dynamic-filters
Open

feat: forward remote dynamic filter updates to coordinator#635
jayshrivastava wants to merge 2 commits into
js/2-forward-dynamic-filter-updates-to-coordinatorfrom
js/3-plan-dynamic-filters

Conversation

@jayshrivastava

@jayshrivastava jayshrivastava commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Stack

This stack of PRs implements distributed dynamic filtering #528

  1. coordinator: display consumer dynamic filters after execution #623
  2. feat: plan distributed dynamic filters #634
  3. feat: forward remote dynamic filter updates to coordinator #635 <- you are here
  4. coordinator: merge partial dynamic filters  #636
  5. coordinator: forward merged dynamic filters to consumers #637
  6. [do not review] worker: apply merged dynamic filters during execution #639

Problem

The QueryCoordinator needs to receive partial dynamic filter updates from workers.

Solution

We introduce a new WorkerToCoordinatorMsg which

message ProducedDynamicFilter {
  uint64 expression_id = 1;
  // Serialized datafusion.proto.PhysicalExprNode.
  bytes expression_proto = 2;
}

In this PR makes each worker unconditionally send updates (via wait_update() and wait_complete()) to the coordinator for any dynamic_filter_remote_producer_ids in the SetPlanRequest. The purpose of dynamic_filter_remote_producer_ids is to exclude any dynamic filters who only have local consumers - these don't need to be forwarded to the coordinator.

@jayshrivastava jayshrivastava changed the title plan distributed hash join filters coordinator: plan distributed dynamic filters Aug 13, 2026
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch from 60dc39a to 7bee97e Compare August 13, 2026 19:27
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch from 7bee97e to 53be7d0 Compare August 17, 2026 18:54
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch 2 times, most recently from 4fd1433 to c29bcf1 Compare August 17, 2026 19:38
@jayshrivastava jayshrivastava changed the title coordinator: plan distributed dynamic filters feat: forward remote dynamic filter updates to coordinator Aug 18, 2026
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch from c29bcf1 to a7bb152 Compare August 18, 2026 18:15
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch from a7bb152 to 8bc2dc9 Compare August 18, 2026 18:52
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch 2 times, most recently from e1a6b47 to 77c9528 Compare August 21, 2026 16:48
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch from 77c9528 to dd39b56 Compare August 21, 2026 20:58
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch from dd39b56 to fceebde Compare August 22, 2026 15:00
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch 2 times, most recently from 5912188 to a60f964 Compare August 23, 2026 15:27
Task-local consumer

producer F ────────────────> consumer F
          shared expression; no RPC

Cross-task consumer

producer F ── boundary anchor F ──> SetPlanRequest.report_ids=[F]
    │
    ├── wait_update() -----+
    ├── wait_complete() ---+--> latest observed F --RPC--> coordinator
    └── query cancellation +--> stop

Derive the report allowlist from producer IDs intersected with network-boundary anchor IDs. Workers observe only allowlisted producers, while DataFusion continues updating task-local consumers directly in memory. Watch-based updates may naturally coalesce, and completion remains observed separately because it does not advance the generation.
@jayshrivastava
jayshrivastava force-pushed the js/3-plan-dynamic-filters branch from a60f964 to 8d666d6 Compare August 23, 2026 15:35
@jayshrivastava
jayshrivastava marked this pull request as ready for review August 23, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant