Skip to content

fix: normalize scalar float sort and window rank keys #5469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sunchao wants to merge 2 commits into
base: main
Choose a base branch
from
+530 −41
Open

fix: normalize scalar float sort and window rank keys #5469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4f43d76
fix: normalize scalar float sort and window rank keys
sunchao Aug 26, 2026
96eafdf
fix: preserve floating window partition ordering
sunchao Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 13 additions & 10 deletions docs/source/user-guide/latest/compatibility/floating-point.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,18 @@ to Spark in some cases, especially when the data contains both positive and nega
case that is not of concern for many users. If it is a concern, setting `spark.comet.exec.strictFloatingPoint=true`
will make relevant operations fall back to Spark.

## Ordering: signed zero (`-0.0` vs `+0.0`)
## Ordering: NaN and signed zero (`-0.0` vs `+0.0`)

Spark's `ORDER BY`, `RANK`, `DENSE_RANK`, and window frame comparisons route through
`SQLOrderingUtil.compareDoubles` / `compareFloats`, which explicitly define `-0.0 == 0.0`. Comet's
native sort and `WindowGroupLimitExec` use the `arrow-row` row-format encoder for `ORDER BY` keys,
which applies Rust's total-ordering transform to the raw IEEE-754 bits. Under that encoding `-0.0`
sorts strictly less than `+0.0`, so a partition that mixes the two zeros can produce a rank
distribution that differs from Spark. For example, `RANK() OVER (ORDER BY v ASC)` over
`[-0.0, 0.0, 1.0]` filtered to `rk <= 1` returns two rows in Spark (both zeros tied at rank 1) but
one row in Comet (`-0.0` at rank 1, `+0.0` at rank 2). If your workload materially mixes `-0.0`
and `+0.0` in a ranked column, prefer Spark for that stage or normalize the column to `+0.0`
upstream.
`SQLOrderingUtil.compareDoubles` / `compareFloats`, which equate all NaN representations and
define `-0.0 == 0.0`. NaN sorts above every non-NaN value.

For scalar `FLOAT` and `DOUBLE` keys, Comet normalizes NaNs and signed zeros before native
sorting, window peer comparisons, and `WindowGroupLimitExec` rank comparisons. Native range
partitioning normalizes its keys and sampled boundaries in the same way. Only comparison keys
are normalized; returned values retain their original NaN representations and zero signs.

Floating-point values nested in arrays or structs still use Arrow's raw total ordering and can
produce different ordering or rank results from Spark. The existing
`spark.comet.exec.strictFloatingPoint=true` fallback policy is unchanged, including its
conservative fallback for scalar floating-point sort keys.
Loading
Loading