From 04fd8f5a65fc018c3dbeacab9c39805d59be6e21 Mon Sep 17 00:00:00 2001 From: jackylee Date: Wed, 26 Aug 2026 08:40:19 +0800 Subject: [PATCH] docs: filter pushdown is connected, not planned future work VortexScanBuilder has implemented SupportsPushDownV2Filters since #7785, and the converted predicate reaches native code as the ScanOptions filter. The guide still described it as infrastructure that exists but is not wired up, so a developer reading it plans around a limitation that is gone. The Future Work section keeps the Scan API Source trait migration, which is still outstanding. Signed-off-by: jackylee --- docs/developer-guide/integrations/spark.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/developer-guide/integrations/spark.md b/docs/developer-guide/integrations/spark.md index 2342f95063f..6758645c0dc 100644 --- a/docs/developer-guide/integrations/spark.md +++ b/docs/developer-guide/integrations/spark.md @@ -39,8 +39,9 @@ Projection pushdown is supported through Spark's `SupportsPushDownRequiredColumn The scan builder prunes the column list to only those referenced by the query, and the pruned column set is passed to the native scan via `ScanOptions`. -Filter pushdown infrastructure exists in the `ScanOptions` type but is not yet connected to -Spark's `SupportsPushDownFilters` interface. This is planned future work. +Filter pushdown is supported through `SupportsPushDownV2Filters`. The scan builder converts each +predicate it recognizes into a Vortex expression and keeps the rest for Spark to evaluate after the +scan; the converted expression reaches native code as the `ScanOptions` filter. ## Data Export @@ -53,5 +54,4 @@ through direct byte buffers. The current integration builds directly on the native file and scan APIs via JNI. Future work will migrate it to use the [Scan API](/concepts/scanning) `Source` trait, which will provide a -standard interface for file discovery, partitioning, and pushdown. This will also enable -connecting Spark's filter pushdown to Vortex's expression-based filtering. +standard interface for file discovery, partitioning, and pushdown.