feat: add destination-aware native shuffle execution (3/n) - #5481
Merged
sunchao merged 1 commit intoAug 27, 2026
Merged
Conversation
sunchao
approved these changes
Aug 26, 2026
sunchao
left a comment
Member
There was a problem hiding this comment.
Reviewed commit 8281c3490c9cc007e846d02f3bc6ef500bf7ba16 with five independent reviewers covering execution lifecycle, partitioning/spills, callbacks/framing, local compatibility, and edge cases. No outstanding verified P1/P2 findings.
Local validation passed: 60 shuffle unit tests (including all 10 added tests), 24 local/RSS output comparisons, 3 partial callback failure checks, and formatting/diff checks. Actual JVM/RSS integration was not exercised; callback binding is deferred.
Member
|
Merged, thanks @pingzh !!! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #5352. This is the third foundational PR and does not close the issue.
Previous PRs:
Rationale for this change
Previous PRs introduced the RSS partition writer and destination-aware shuffle plans, but native shuffle execution still supports only local files. This PR enables the existing shuffle execution path to write to either local files or a remote shuffle service.
What changes are included in this PR?
ShuffleWriterDestinationwith local and RSS destinations.ShuffleWriterExecconstructor while preserving the existing local constructor.Binding RSS destinations to task-owned JVM callbacks will be addressed in a subsequent PR.
How are these changes tested?
CometNativeShuffleSuite.