fix: preserve range partitions when scaling file scans - #648
Conversation
It is not valid to claim a listing scan is hash-partitioned. The join tests now declare Range output partitioning on the hive-layout testdata instead. file_scan_config_scale_up_leaf_node no longer flattens those file groups. FileScanConfig::repartitioned already refuses to rearrange them, because moving a file to another slot breaks the range (or hash) claim and can produce wrong partitioned joins and SinglePartitioned aggregates. Each group stays in its original slot and is assigned to one task. scale_partitioning keeps Range only when the partition count is unchanged. Concatenating several range-partitioned tasks, as NetworkCoalesceExec does, is not a valid Range of the new size, so it degrades to UnknownPartitioning. Stage display treats Range like Hash: every task shares the same partition indices. Also escape Windows paths in snapshot filters so cwd is not parsed as a regex. Fixes datafusion-contrib#628
Clippy -D warnings failed on ptr_arg: writing &PathBuf instead of &Path.
|
Clippy failed on Pushed 8c6d2e3: |
|
@shinzoxD I see you opened several PRs while there were already some in-flight. These PRs look AI generated without supervision, as the approach to most of them shows several questionable traits of one-off behaviors, unnecessary special cases and unnecessarily verbose testing approach. In order to allow other people to contribute, I'm going to close without a review some of your PRs, leaving just the first 3 you opened. Please, upon investing time in this project, prefer quality over quantity, and prefer investing time in getting just a small number of PRs up to a high standard, rather than a big quantity of purely AI generated PRs. As always, thanks for your contributions, and see you in the other PRs! |
Summary
Fixes #628. Depends on #540 (this branch is based on
branch-55becausePartitioning::Rangedoes not exist on DataFusion 54).A listing scan cannot honestly claim
Hash(expr). The join suite now registers the hive-layout testdata with declaredRangeoutput partitioning, so the optimizer can skip shuffles for a real distribution instead of an invalid hash claim.The tests also answer the questions on that issue:
FileScanConfig::repartitionedalready returnsNonewhenoutput_partitioningis set.file_scan_config_scale_up_leaf_nodewas flattening files and round-robin assigning them, which moved e.g. partition 2 into slot 1. It now keeps groupias partitioniand gives each group to a single task.target_partitionsdoes not collapse range partitions. With 8 range partitions andtarget_partitions=2, DataFusion still builds 8 file groups. Scaling to 2 leaf tasks keeps those 8 slots; even partitions go to task 0 and odd ones to task 1. The plan staysSinglePartitionedon the range key (no extra shuffle).scale_partitioningcannot change a Range's partition count. The count is the split-point width. Identity scaling keepsRange;NetworkCoalesceExecmultiplying by input tasks degrades toUnknownPartitioning. Stage display treats Range like Hash (shared partition indices across tasks).Validation