feat(parquet): add per-column row selection policy - #10710
Draft
hhhizzz wants to merge 11 commits into
Draft
Conversation
Mask fragments were trimmed only at the front, so a chunk that ends in a long skip run decoded every trailing row just to filter it away. This is reachable at row-group boundaries, where the window compiler extends a chunk to the boundary regardless of how far the last selected row is, and it made per-column Mask execution decode more than the global Mask path, which bounds its chunks by the rows still needed. Fragments now stop at their last selected row, using set_slices() so both ends are found with word-sized scans instead of per-bit probes. Trailing rows fall through to the skip that already closes each chunk. Lower chunks into an explicit ColumnInstruction stream while here. Both strategies now flow through one loop in read_column_batch, and the trailing skip that keeps a Mask lane aligned with its peers is emitted by the lowering itself rather than left to the caller to remember. The per-chunk row count is checked against the chunk width, which catches a lane that drifts without changing any column's length. Also mark RowSelectionPolicy #[non_exhaustive]; this release already breaks exhaustive matches by adding AutoPerColumn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…Column AutoPerColumn rewrote every predicate result into a bitmap-backed RowSelection, whether or not per-column planning went on to engage. When planning declined - which it does whenever the projection is uniform, and so for any projection whose columns are all outside the cost model - the scan still carried the bitmap into the global path. Resolving to Selectors there converts the whole mask back into selectors, an O(total_rows) pass the default policy never runs, and any intervening selection algebra runs on bitmaps rather than run lists. The comment justifying this claimed selectors were materialized only when a column picked that strategy, but compiling the execution windows consumes selectors unconditionally, so bitmap backing was paying for a conversion in every case and saving one only for columns that chose Mask. Selector backing is the better source for both paths: it is what window compilation already needs, and SelectionRepresentations builds the mask lazily and shares it across the columns that want it. Predicate output now keeps whatever backing the default policy would produce, so declining to engage costs nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The cost model covers exactly the four families the sampler can build, so every other column type falls back to the global threshold. Add the three families that dominate the TPC-DS and ClickBench projections and that each exercise a distinct decode path: Int64 and Float64 for the 8-byte PrimitiveArrayReader, and plain Utf8 for the byte-array reader that Utf8View fixtures do not cover. No planner thresholds change here. These fixtures make the families sampleable; the values still have to come from a refinement run, which --kind can now target one family per workspace. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The refinement sampler picks 9 for both sampled fixed-binary widths on an AMD EPYC host across three seeds. At 5, the heterogeneous boundary_50_run8 policy-validation case split its projection into 16 Mask and 16 Selectors columns and lost ~17% against the global Auto policy; at 9 it decides uniformly and reaches a practical tie. Retarget the mix-columns reader test at an average run of 12, which still separates the Int32/narrow Utf8View thresholds from the wide byte thresholds, and make the per-row-group statistics test iterate the model constants instead of hardcoded literals.
The cost model reached only `Int32`, `Dictionary(Int32, Utf8)`, `FixedSizeBinary` and non-dictionary `Utf8View`. Measured against TPC-DS SF10 through DataFusion, that set covers 2 of the 183 columns in the eight largest tables, so `AutoPerColumn` fell back to the legacy global threshold on every scan and the planner never ran. ClickBench was the same story for its 28 string columns. Model `Int64`, `Decimal128` in the INT32-backed precision range, `Date32`, and dictionary-encoded `Utf8View`. The last one previously returned `None` on purpose; it is the largest single slice of real string columns (31 TPC-DS columns, 28 in ClickBench), and it samples well clear of the plain `Utf8View` threshold, so it gets its own value instead of being excluded. Coverage goes from 2 to 183 of those columns. Thresholds come from a fresh refinement pass on aarch64: five seeds, unit resolution over runs 1..=16, 24 minimum pairs, control drift -1.35%. The analysis is restricted to the sampler's `balanced-grid` family, the only one whose `select_run` equals the average run length that the thresholds are compared against; pooling it with `dense-grid` and `sparse-grid` makes unrelated shapes look like anomalies on a single curve. Every family crosses between 4 and 16, so `refinement_shapes` now samples that window at unit resolution instead of stepping 12 -> 16 -> 20. Thresholds are hardware-specific: the same types sampled 21/16/13 on x86_64 EPYC and 14/15/15 here, so they cannot be carried between machines. `FixedSizeBinary` no longer shares a constant with wide `Utf8View` because the two now sample three points apart. Add `Decimal128`, `Date32` and `DictStringView` fixtures to the sampler. The last writes `Utf8View` with dictionary encoding enabled so the encoding the planner keys on is actually produced (verified: `RLE_DICTIONARY`). TPC-DS SF10 through DataFusion, nine interleaved `Auto` / `AutoPerColumn` pairs at five iterations each: paired deltas -0.15% +1.14% -0.08% +0.22% +0.36% -0.33% +0.88% -0.16% +0.27%, mean +0.24%, 95% CI [-0.07%, +0.55%]. The interval spans zero and four of nine deltas are negative, so there is no aggregate regression to report, but the effect is also not a win. Per query, six regress reproducibly (q9 +5.4%, q37 and q89 +3.0%, q91 +2.9%, q47 +2.1%, q38 +2.0%) against one improvement (q75 -2.6%); the other 92 sit inside the control spread. Row counts match `Auto` on all 99 queries in all nine runs.
`AutoPerColumn` built a split execution plan whenever any two decisions differed, including when every column inside a row group agreed and only the row groups differed from each other. The split executor cannot exploit that case — it gives each column its own reader and replays chunks through all of them — so the plan paid for the split machinery and returned the same rows a single global choice would have. Worse, discovering it required the per-row-group pass over the selection. On TPC-DS SF10 q9 that pass alone cost 5.2% (796ms -> 838ms), and it cost it unconditionally: forcing every threshold to 32, so that no column could possibly disagree, reproduced the full slowdown. No threshold could have fixed that, because the cost was paid before any threshold was consulted. Decide it from column metadata instead. Columns that resolve to the same threshold can never disagree, and thresholds come from metadata, so the check runs without touching the selection. When they do share one, defer to the global policy exactly as before. Keep the existing planning path for genuinely mixed projections, and require real column-level disagreement before building a split plan. TPC-DS SF10, seven interleaved `Auto` / `AutoPerColumn` pairs at five iterations: paired deltas -0.20% +0.77% +0.99% +0.04% -0.22% +0.20% +0.42%, mean +0.29%, 95% CI [-0.03%, +0.61%]. q9 no longer separates from the control spread, and two queries now improve reproducibly (q36 -4.3%, q75 -2.8%) where before there was one. Five still regress (q83 +4.8%, q19 +4.3%, q47 +2.6%, q63 +2.3%, q70 +2.1%); only q83 and q47 reproduce across measurement rounds, the rest move between rounds. Row counts match `Auto` on all 99 queries in all seven runs. `test_auto_per_column_cross_row_group_metadata_planning` asserted the row-group-only split that this removes. It is renamed and now pins the new contract: two `Int32` columns share a threshold, so the policy defers and records four identical decisions. Its correctness assertion is unchanged.
The five seeds put `Int64`'s crossover at 12, 14 and 15, so any of them is a
defensible reading of the sampled spread. 14 and 15 are indistinguishable as
decisions, but they are not indistinguishable structurally: at 14 an `Int64`
column and a `Decimal128` column resolve to different thresholds, which
defeats the shared-threshold check on every scan that projects both. TPC-DS
fact tables are exactly that shape, so those scans fell back to the
per-row-group pass over the selection for a distinction that never changed
an outcome. On q47 moving `Int64` to 15 is worth 2.3 points (+2.6% -> +0.3%)
for that reason alone.
TPC-DS SF10, five interleaved pairs at five iterations: paired deltas -0.18%
+0.25% +0.32% +0.13% +0.36%, mean +0.18%, 95% CI [+0.00%, +0.35%].
Two queries improve reproducibly: q36 -3.6% and q75 -3.1%, the same two in
every measurement round. Nothing regresses reproducibly. Each round flags
about five queries as slower, which is what a two-sigma threshold over 99
queries produces by construction, and the sets do not overlap between rounds
({19, 47, 63, 70, 83} against {8, 33, 38, 40, 71}, no common member) while
the two improvements repeat exactly. Row counts match `Auto` on all 99
queries in all five runs.
The previous values came from an aarch64 refinement pass. Rerunning the same five-seed, unit-resolution manifest on x86_64 (AMD EPYC) moves every crossover well up: `Int64` 15 -> 28, `Int32` 15 -> 24, `Date32` 16 -> 28, `Decimal128` 15 -> 18, dictionary `Utf8View` 11 -> 16, `FixedSizeBinary` 5 -> 9. Only plain wide `Utf8View` lands on the same value. Mask holds its edge over Selectors across a much wider run-length range on EPYC, and the model has to follow the hardware it runs on. The difference decides whether the policy is worth having. On TPC-DS SF10, five interleaved pairs at five iterations on that machine: paired deltas -1.93% -2.29% +0.24% -0.58% -1.01%, mean -1.12%, 95% CI [-1.92%, -0.31%]. The interval lies entirely below zero, so this is a win rather than a wash. Eight queries improve reproducibly — q36 -14.5%, q75 -13.0%, q43 -9.2%, q76 -8.9%, q69 -8.4%, q65 -7.1%, q70 -6.1%, q51 -4.6% — and nothing regresses at all. For comparison, the same machine with the pre-coverage model traded 20 improvements against 22 regressions for an aggregate inside the control drift. The aarch64 values are not wrong, they are calibrated for different hardware; on that machine they are worth -3.6% on q36 where these would leave most of it unclaimed. The doc comment now says so, and points at the sampler stage to rerun for any other target. `MIN_RUN_THRESHOLD` follows `WIDE_BYTE_RUN_THRESHOLD` rather than `FIXED_BINARY_RUN_THRESHOLD`, which is no longer the smallest arm.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft PR, WIP
Which issue does this PR close?
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?