perf(parquet): skip RowFilter on statically fully-matched row groups#23696
Draft
zhuqi-lucas wants to merge 1 commit into
Draft
perf(parquet): skip RowFilter on statically fully-matched row groups#23696zhuqi-lucas wants to merge 1 commit into
zhuqi-lucas wants to merge 1 commit into
Conversation
Builds on the `peek_next_row_group` API landed in apache/arrow-rs#10158. When Parquet stats prove that every row of a row group already satisfies the pushdown predicate (`fully_matched`), running the per-row `RowFilter` inside that row group is pure overhead — every row passes anyway. This change installs an *empty* `RowFilter` on fully-matched runs and only pays the row-level machinery on RGs that still need filtering. ## What changes * `access_plan.rs` — `ParquetAccessPlan` now tracks per-RG `fully_matched` state, produced during static pruning. * `metrics.rs` — new `row_filter_skipped_fully_matched` counter to observe how often the toggle fires. * `push_decoder.rs` — `RgPlanEntry` carries the per-RG toggle; `RowFilterContext` swaps between the real filter (from the `prebuild_row_filter_candidates` cache) and an empty filter as the decoder crosses row-group boundaries. * `row_filter.rs` — split `build_row_filter` into `prebuild_*` and `row_filter_from_prebuilt`: the expensive tree walk + candidate construction runs once per file, and the cheap per-RG bind runs at each toggle. Preserves the existing public `build_row_filter` API for non-toggling callers. * `opener/mod.rs` — wires the prebuilt cache into the decoder builder; skips page-index loading for fully-matched RGs (their per-row filter is a no-op, so page pruning saves nothing). * `sort.rs` — sort-order-aware RG reorder preserves the per-RG toggle through reordering. ## Tests * `fully_matched_rgs_skip_row_filter` (`dynamic_row_group_pruning.rs`) — 4 RGs of 3 rows each, predicate `v >= 3` makes RGs 1..=3 fully matched. Asserts: (a) results are correct, (b) `row_filter_skipped_fully_matched` fires at least once at the RG 0 → RG 1 boundary. * All 6 pre-existing `dynamic_row_group_pruning` tests still pass. * Full `datasource-parquet` lib suite (158 tests) passes.
github-actions
Bot
added
core
zhuqi-lucas
marked this pull request as draft
July 19, 2026 03:28
Contributor
Author
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Parquet scan performance by avoiding per-row RowFilter evaluation for row groups proven (via statistics/static pruning) to be fully matched, and by enabling cheap per-row-group rebuilds of the row filter using prebuilt candidates. This builds on arrow-rs’s peek_next_row_group() to keep DataFusion’s row-group plan aligned with the decoder’s actual frontier.
Changes:
- Track per-row-group
fully_matchedstate in the Parquet access plan and preserve it through reorder/reverse operations. - Toggle the decoder’s
RowFilterat row-group boundaries (empty filter for fully-matched RGs; real filter otherwise) and add a new metric for observability. - Refactor row-filter construction to prebuild filter candidates once per file and rebuild cheaply per row group; update integration + SQL logic tests accordingly.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| datafusion/datasource-parquet/src/access_plan.rs | Carries per-RG fully_matched flags into the prepared plan and strips empty-selection RGs to keep plan/decoder alignment. |
| datafusion/datasource-parquet/src/metrics.rs | Adds row_filter_skipped_fully_matched metric to Parquet file metrics. |
| datafusion/datasource-parquet/src/push_decoder.rs | Adds per-RG plan entries and implements boundary-time RowFilter toggling using peek_next_row_group(). |
| datafusion/datasource-parquet/src/row_filter.rs | Splits row-filter building into “prebuild once per file” + “instantiate per RG” and expands nested/struct field planning logic. |
| datafusion/datasource-parquet/src/opener/mod.rs | Wires the prebuilt row-filter context into decoder construction and propagates new metric/state into the stream. |
| datafusion/datasource-parquet/src/sort.rs | Updates prepared-plan reverse test to reflect stripping of all-empty selections. |
| datafusion/core/tests/parquet/dynamic_row_group_pruning.rs | Adds an end-to-end integration test asserting fully-matched RGs trigger row-filter skipping behavior. |
| datafusion/sqllogictest/test_files/push_down_filter_parquet.slt | Updates EXPLAIN ANALYZE expectations to include the new metric. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+64
to
+68
| /// Number of row groups for which the per-row | ||
| /// [`RowFilter`](parquet::arrow::arrow_reader::RowFilter) was skipped | ||
| /// because the static stats proved every row of the RG satisfies the | ||
| /// predicate. The decoder is rebuilt at the boundary with an empty | ||
| /// row filter so the upcoming RG decodes without per-row evaluation. |
| && (!DataType::is_nested(return_type) | ||
| || self.is_nested_type_supported(return_type)) | ||
| { | ||
| // try to resolve all field name arguments to strinrg literals |
Comment on lines
+192
to
+196
| /// The result of resolving which Parquet leaf columns and Arrow schema fields | ||
| /// are needed to evaluate an expression against a Parquet file | ||
| /// | ||
| /// This is the shared output of the column resolution pipeline used by both | ||
| /// the row filter to build `ArrowPredicate`s and the opener to build `ProjectionMask`s |
| @@ -797,9 +795,6 @@ impl ParquetMorselizer { | |||
| .transpose()?; | |||
| } | |||
|
|
|||
Contributor
Author
|
run benchmarks env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Contributor
Author
|
run benchmark sort_tpch env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: sort_tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagesort_tpch — base (merge-base)
sort_tpch — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
fully_matchedRowFilter skip · needs arrow-rs#10158 (peek_next_row_group) #23067.Rationale for this change
Builds on the
peek_next_row_groupAPI landed in apache/arrow-rs#10158 (in releasedarrow-rs 59.1.0). When Parquet stats prove that every row of a row group already satisfies the pushdown predicate (fully_matched), running the per-rowRowFilterinside that row group is pure overhead — every row passes anyway. This change installs an emptyRowFilteron fully-matched runs and only pays the row-level machinery on RGs that still need filtering.Two-phase reads for those RGs also skip fetching the filter columns when they aren't in the projection — this pattern is common on time-partitioned tables (
WHERE ts >= X AND ts < Ywhere the RG's min/max is contained in[X, Y)) and on the middle stretch of aTopKscan where the dynamic threshold is still loose.What changes are included in this PR?
access_plan.rs—ParquetAccessPlannow tracks per-RGfully_matchedstate, produced during static pruning.metrics.rs— newrow_filter_skipped_fully_matchedcounter to observe how often the toggle fires.push_decoder.rs—RgPlanEntrycarries the per-RG toggle;RowFilterContextswaps between the real filter (from theprebuild_row_filter_candidatescache) and an empty filter as the decoder crosses row-group boundaries viapeek_next_row_group().row_filter.rs— splitbuild_row_filterintoprebuild_row_filter_candidates(once per file: tree walk + candidate construction) androw_filter_from_prebuilt(per RG boundary: cheap bind against the fresh array readers). The existing publicbuild_row_filterAPI is preserved for callers that don't need per-RG toggling.opener/mod.rs— wires the prebuilt cache into the decoder builder; skips page-index loading for fully-matched RGs (their per-row filter is a no-op, so page pruning saves nothing).sort.rs— sort-order-aware RG reorder preserves the per-RG toggle through reordering.Are these changes tested?
Yes.
fully_matched_rgs_skip_row_filter(datafusion/core/tests/parquet/dynamic_row_group_pruning.rs) — 4 RGs of 3 rows each, predicatev >= 3makes RG 0 a straddler and RGs 1..=3 fully matched. Asserts (a) results are correct and (b)row_filter_skipped_fully_matchedfires at least once at the RG 0 → RG 1 boundary.dynamic_row_group_pruningintegration tests still pass.datafusion-datasource-parquetlib suite (158 tests) passes.Expected benchmark impact (I'll follow up with numbers):
sort_tpch(target workload): the tightening dynamic threshold produces fully-matched runs on the way up, then fully-pruned RGs after the threshold snaps — expect additional 10–30% on the queries already accelerated by feat(parquet): intra-file early stopping via statistics + dynamic filters #22450.URL LIKE …/ equality filters used there rarely fully cover an RG.Are there any user-facing changes?
row_filter_skipped_fully_matchedmetric appears inEXPLAIN ANALYZEon parquet scans (0 when the toggle didn't fire).build_row_filterstill exists and behaves identically for non-toggling callers.