feat(btrblocks): let FSSTScheme compress binary arrays behind a sampled gate - #9580
feat(btrblocks): let FSSTScheme compress binary arrays behind a sampled gate#9580joseph-isaacs wants to merge 1 commit into
Conversation
…ed gate FSSTScheme gated on is_utf8(), but its compress path never validates UTF-8 -- it trains and compresses over the raw varbinview bytes, so the dtype gate was excluding binary columns from a scheme that already works on them. Widen matches() to accept binary. Because arbitrary binary payloads (hashes, ciphertexts, compressed blobs) often have no intra-value structure, binary columns do not go straight to sampling. FSST first trial-compresses a tiny strided sample of up to 64 values and enters scheme selection only when its code bytes beat the plain VarBin value bytes (views excluded) by more than 15%. Once past the gate, the ranked ratio comes from the compressor's standard sampling estimator via the new CascadingCompressor::estimate_by_sampling, so FSST competes on the same measurement basis as every other scheme. Measured at 100k rows (tests/varbin_scheme.rs), compressed nbytes: shared prefix 1,600,000 -> 734,685 nulls every 7th 1,647,348 -> 690,230 random 16B (hash) 1,600,000 -> 1,600,000 (gate skips FSST) random 256B 25,600,000 -> 25,600,000 (gate skips FSST) Binary lands byte-identical to the same content stored as Utf8, which confirms the dtype gate was not protecting the compress path. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Claude <noreply@anthropic.com>
Merging this PR will improve performance by 12.5%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | WallTime | words_gather_dispatch_avx512[1024] |
9 ns | 8 ns | +12.5% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/fsst-binary-scheme (81d6bb8) with develop (e4b3421)2
Footnotes
-
54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
claude/varbin-scheme(d598e09) during the generation of this report, sodevelop(e4b3421) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Summary
Stacked on #9579.
FSSTSchemegated onis_utf8(), but its compress path never validates UTF-8 — it trains and compresses over the rawvarbinviewbytes, so the dtype gate was excluding binary columns from a scheme that already works on them. This PR widensmatches()to accept binary, but because arbitrary binary payloads (hashes, ciphertexts, compressed blobs) often have no intra-value structure, binary columns must first pass a cheap trial: FSST compresses a tiny strided sample of up to 64 values and enters scheme selection only when its code bytes beat the plainVarBinvalue bytes (views excluded) by more than 15%.Changes
FSSTScheme::matchesaccepts binary; for binary dtypesexpected_compression_ratioreturns aDeferredEstimate::Callbackimplementing the gate (vortex-btrblocks/src/schemes/string/fsst.rs).CascadingCompressor::estimate_by_samplingwrapper (vortex-compressor/src/compressor/sample.rs), so FSST competes on the same measurement basis as every other scheme — a gate-local ratio turned out to be incomparable with sampled scores and mis-ranked FSST against dict/varbin.vortex-btrblocks/tests/varbin_scheme.rs:fsst_binary_gateasserts FSST appears in the compression tree for structured payloads and never for random 16B/256B payloads;fsst_versus_varbin_on_identical_bytesasserts the same bytes compress byte-identically as Binary and Utf8. Measured at 100k rows: shared prefix 1,600,000 → 734,685; nulls every 7th 1,647,348 → 690,230; random payloads are untouched by FSST (gate skips).API Changes
Adds public
CascadingCompressor::estimate_by_samplingfor schemes whose deferred-callback estimates want to fall back to the standard sampled score.Generated by Claude Code