fix: price delta's bases and tail padding in its compression estimate - #9602
Conversation
Signed-off-by: rapour <reza.apour@gmail.com>
Merging this PR will regress 2 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | cold_misaligned[(16, 64)] |
344.6 µs | 390.2 µs | -11.67% |
| ❌ | Simulation | compress_fsst[(500, 64, 8)] |
480.4 µs | 538 µs | -10.72% |
| ⚡ | WallTime | arrow_checked_add_u32_neon[16384] |
20.4 µs | 13.5 µs | +50.77% |
| ⚡ | WallTime | arrow_checked_add_u32_avx512[16384] |
21.4 µs | 17.6 µs | +21.29% |
| ⚡ | WallTime | arrow_checked_add_u32_avx2[16384] |
21.3 µs | 17.7 µs | +20.38% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rapour:fix/delta-estimate (31a5290) with develop (17bd9e2)
Footnotes
-
106 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Regenerated the golden snapshots against the merged code: develop's delta estimate fix (#9602) and BlockedFoR both target the same shapes, so the unstable snapshots now match develop and only the default-feature ones change. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Two defects, one root cause: BlockedFoRScheme was registered in place of FoRScheme and declared both encodings as its output. An encoding must belong to an enabled edition before the writer will serialize it, and fastlanes.blockedfor belonged to none. Where the file writer configures an encoding policy, retain_allowed_encodings saw a scheme producing a forbidden encoding and dropped it -- taking global FoR with it, because no other scheme produced it. Integer columns that would have used a frame of reference fell back to plain bit-packing, so ClickBench files grew 0.6% overall (152 of 200 larger). Where no policy was configured, nothing filtered the scheme and the write failed outright with "Array encoding fastlanes.blockedfor not permitted by ctx", which is what the SQL logic tests hit. BlockedFoRScheme now produces only BlockedFoR and skips wherever a single global reference is the better shape, so FoRScheme rejoins ALL_SCHEMES and competes for those arrays on its own estimate. An encoding policy that forbids blockedfor now drops just the blocked scheme and keeps frame of reference, pinned by retain_allowed_encodings_keeps_global_for_without_blocked. fastlanes.blockedfor joins the new PREVIEW_2026_08_0 edition, following fastlanes.delta. Preview is only enabled with unstable_encodings, so the scheme is gated behind that feature -- selecting an encoding the writer cannot serialize fails the write. Under that feature Delta already wins these shapes after #9602, so every golden snapshot now matches develop apart from one compact list case that shifts delta -> pco and shrinks. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Pin fastlanes to =0.6.1: the 0.6.2 release changed the bit-transpose API, and the lock-free wasm-test build resolves the latest compatible release, so the WASM smoke test stopped compiling. Regenerate the compact golden snapshot for #9602's delta compression estimate fix, which makes the compact selector encode list offsets with pco instead of fastlanes delta (4434 B vs 4748 B). The snapshot was not updated when that change landed, leaving the golden-corpus check red on develop. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Fixes: #9601
Changes
DeltaSchemeestimatedfull_width / delta_bits * DELTA_PENALTY, the cost of theresidual payload alone. Delta also stores a bases child and pads its deltas child up
to whole 1,024-value chunks, and neither term was in the estimate, so the scheme can
win a column and then lose it on real bytes.
Both are cheap to price exactly:
FastLanes keeps one base per lane per chunk, and a chunk of
W-bit values has1024/Wlanes, so each chunk's bases occupy 1024 bits whateverWis: one bit perstored element.
The deltas child is
len.next_multiple_of(1024)long, so a column that overshoots achunk boundary pays for residuals it stores no values for.
The same helper now supplies the
best_so_farearly-exit bound since Delta always pays for the bases.API Changes
N/A