Skip to content

fix: preserve range partitions when scaling file scans - #648

Closed
shinzoxD wants to merge 2 commits into
datafusion-contrib:branch-55from
shinzoxD:tests-range-partitioned-data
Closed

fix: preserve range partitions when scaling file scans#648
shinzoxD wants to merge 2 commits into
datafusion-contrib:branch-55from
shinzoxD:tests-range-partitioned-data

Conversation

@shinzoxD

Copy link
Copy Markdown
Contributor

Summary

Fixes #628. Depends on #540 (this branch is based on branch-55 because Partitioning::Range does not exist on DataFusion 54).

A listing scan cannot honestly claim Hash(expr). The join suite now registers the hive-layout testdata with declared Range output partitioning, so the optimizer can skip shuffles for a real distribution instead of an invalid hash claim.

The tests also answer the questions on that issue:

  1. Rearranging range file groups is incorrect. FileScanConfig::repartitioned already returns None when output_partitioning is set. file_scan_config_scale_up_leaf_node was flattening files and round-robin assigning them, which moved e.g. partition 2 into slot 1. It now keeps group i as partition i and gives each group to a single task.
  2. target_partitions does not collapse range partitions. With 8 range partitions and target_partitions=2, DataFusion still builds 8 file groups. Scaling to 2 leaf tasks keeps those 8 slots; even partitions go to task 0 and odd ones to task 1. The plan stays SinglePartitioned on the range key (no extra shuffle).
  3. scale_partitioning cannot change a Range's partition count. The count is the split-point width. Identity scaling keeps Range; NetworkCoalesceExec multiplying by input tasks degrades to UnknownPartitioning. Stage display treats Range like Hash (shared partition indices across tasks).

Validation

cargo test -p datafusion-distributed --lib scale_
cargo test -p datafusion-distributed --lib events::defaults::file_scan_config
cargo test -p datafusion-distributed --features integration --test join --test range_partitioned

It is not valid to claim a listing scan is hash-partitioned. The join
tests now declare Range output partitioning on the hive-layout testdata
instead.

file_scan_config_scale_up_leaf_node no longer flattens those file
groups. FileScanConfig::repartitioned already refuses to rearrange them,
because moving a file to another slot breaks the range (or hash) claim
and can produce wrong partitioned joins and SinglePartitioned
aggregates. Each group stays in its original slot and is assigned to one
task.

scale_partitioning keeps Range only when the partition count is
unchanged. Concatenating several range-partitioned tasks, as
NetworkCoalesceExec does, is not a valid Range of the new size, so it
degrades to UnknownPartitioning. Stage display treats Range like Hash:
every task shares the same partition indices.

Also escape Windows paths in snapshot filters so cwd is not parsed as a
regex.

Fixes datafusion-contrib#628
Copilot AI lite review requested due to automatic review settings August 15, 2026 20:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Clippy -D warnings failed on ptr_arg: writing &PathBuf instead of &Path.
@shinzoxD

Copy link
Copy Markdown
Contributor Author

Clippy failed on tests/range_partitioned.rs (clippy::ptr_arg: &PathBuf instead of &Path).

Pushed 8c6d2e3: register_range_table now takes &Path. Format-check and tpch-plans-test were already green on the previous SHA; waiting on the new CI run for clippy/unit-test and the remaining integration jobs.

@gabotechs

Copy link
Copy Markdown
Collaborator

@shinzoxD I see you opened several PRs while there were already some in-flight. These PRs look AI generated without supervision, as the approach to most of them shows several questionable traits of one-off behaviors, unnecessary special cases and unnecessarily verbose testing approach.

In order to allow other people to contribute, I'm going to close without a review some of your PRs, leaving just the first 3 you opened. Please, upon investing time in this project, prefer quality over quantity, and prefer investing time in getting just a small number of PRs up to a high standard, rather than a big quantity of purely AI generated PRs.

As always, thanks for your contributions, and see you in the other PRs!

@gabotechs gabotechs closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants