Skip to content

docs: filter pushdown is connected, not future work #9637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
myrrc merged 1 commit into from
Aug 26, 2026
+4 −4
Merged

docs: filter pushdown is connected, not future work #9637

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/developer-guide/integrations/spark.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@ Projection pushdown is supported through Spark's `SupportsPushDownRequiredColumn
The scan builder prunes the column list to only those referenced by the query, and the pruned
column set is passed to the native scan via `ScanOptions`.

Filter pushdown infrastructure exists in the `ScanOptions` type but is not yet connected to
Spark's `SupportsPushDownFilters` interface. This is planned future work.
Filter pushdown is supported through `SupportsPushDownV2Filters`. The scan builder converts each
predicate it recognizes into a Vortex expression and keeps the rest for Spark to evaluate after the
scan; the converted expression reaches native code as the `ScanOptions` filter.

## Data Export

Expand All @@ -53,5 +54,4 @@ through direct byte buffers.

The current integration builds directly on the native file and scan APIs via JNI. Future work
will migrate it to use the [Scan API](/concepts/scanning) `Source` trait, which will provide a
standard interface for file discovery, partitioning, and pushdown. This will also enable
connecting Spark's filter pushdown to Vortex's expression-based filtering.
standard interface for file discovery, partitioning, and pushdown.
Loading