[AURON #2470] Support MapType payloads in native round-robin shuffle - #2471
[AURON #2470] Support MapType payloads in native round-robin shuffle#2471weimingdiit wants to merge 1 commit into
Conversation
…uffle Signed-off-by: weimingdiit <weimingdiit@gmail.com>
slfan1989
left a comment
There was a problem hiding this comment.
Using the 32-bit Murmur3 hash as the only sort key does not provide a deterministic total ordering.
Distinct rows can have the same hash, and Auron uses an unstable sort for this short key, so the relative order of colliding rows can still depend on the upstream input order.
On task retry, those rows may therefore be assigned to different round-robin partitions, which is exactly what the pre-sort is intended to prevent. Spark uses the row hash only as a prefix and resolves collisions with a full binary-row comparison.
Could we add a deterministic Map-compatible tie-breaker, or retain fallback until one is available?
A regression test with deliberately colliding rows and different input orders would also help.
Which issue does this PR close?
Closes #2470
Rationale for this change
Auron currently falls back from native round-robin shuffle when its payload contains
MapType.Although native shuffle serialization supports Map values, the internal pre-sort used for stable round-robin repartitioning relies on Arrow row encoding, which does not support Map.
What changes are included in this PR?
Allows
MapTypepayloads in native round-robin shuffle.Uses Auron's Spark-compatible Murmur3 hash over all payload columns as the internal sort key when the schema contains a Map.
Keeps the existing all-column sort path unchanged for schemas without MapType.
Shares the Map-compatible pre-sort behavior between local and RSS shuffle writers.
Updates the existing Map repartition test to assert that
NativeShuffleExchangeExecis used.Are there any user-facing changes?
Queries using round-robin repartitioning can remain native when their payload contains MapType.
There are no user-facing API or configuration changes.
How was this patch tested?
UT.
Was this patch authored or co-authored using generative AI tooling?