Skip to content

feat(btrblocks): let FSSTScheme compress binary arrays behind a sampled gate - #9580

Open
joseph-isaacs wants to merge 1 commit into
claude/varbin-schemefrom
claude/fsst-binary-scheme
Open

feat(btrblocks): let FSSTScheme compress binary arrays behind a sampled gate#9580
joseph-isaacs wants to merge 1 commit into
claude/varbin-schemefrom
claude/fsst-binary-scheme

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

Stacked on #9579.

FSSTScheme gated on is_utf8(), but its compress path never validates UTF-8 — it trains and compresses over the raw varbinview bytes, so the dtype gate was excluding binary columns from a scheme that already works on them. This PR widens matches() to accept binary, but because arbitrary binary payloads (hashes, ciphertexts, compressed blobs) often have no intra-value structure, binary columns must first pass a cheap trial: FSST compresses a tiny strided sample of up to 64 values and enters scheme selection only when its code bytes beat the plain VarBin value bytes (views excluded) by more than 15%.

Changes

  • FSSTScheme::matches accepts binary; for binary dtypes expected_compression_ratio returns a DeferredEstimate::Callback implementing the gate (vortex-btrblocks/src/schemes/string/fsst.rs).
  • Once past the gate, the ranked ratio comes from the compressor's standard sampling estimator via a new CascadingCompressor::estimate_by_sampling wrapper (vortex-compressor/src/compressor/sample.rs), so FSST competes on the same measurement basis as every other scheme — a gate-local ratio turned out to be incomparable with sampled scores and mis-ranked FSST against dict/varbin.
  • Tests in vortex-btrblocks/tests/varbin_scheme.rs: fsst_binary_gate asserts FSST appears in the compression tree for structured payloads and never for random 16B/256B payloads; fsst_versus_varbin_on_identical_bytes asserts the same bytes compress byte-identically as Binary and Utf8. Measured at 100k rows: shared prefix 1,600,000 → 734,685; nulls every 7th 1,647,348 → 690,230; random payloads are untouched by FSST (gate skips).

API Changes

Adds public CascadingCompressor::estimate_by_sampling for schemes whose deferred-callback estimates want to fall back to the standard sampled score.


Generated by Claude Code

…ed gate

FSSTScheme gated on is_utf8(), but its compress path never validates
UTF-8 -- it trains and compresses over the raw varbinview bytes, so the
dtype gate was excluding binary columns from a scheme that already works
on them. Widen matches() to accept binary.

Because arbitrary binary payloads (hashes, ciphertexts, compressed blobs)
often have no intra-value structure, binary columns do not go straight to
sampling. FSST first trial-compresses a tiny strided sample of up to 64
values and enters scheme selection only when its code bytes beat the
plain VarBin value bytes (views excluded) by more than 15%. Once past the
gate, the ranked ratio comes from the compressor's standard sampling
estimator via the new CascadingCompressor::estimate_by_sampling, so FSST
competes on the same measurement basis as every other scheme.

Measured at 100k rows (tests/varbin_scheme.rs), compressed nbytes:

  shared prefix     1,600,000 ->   734,685
  nulls every 7th   1,647,348 ->   690,230
  random 16B (hash) 1,600,000 -> 1,600,000  (gate skips FSST)
  random 256B      25,600,000 -> 25,600,000 (gate skips FSST)

Binary lands byte-identical to the same content stored as Utf8, which
confirms the dtype gate was not protecting the compress path.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Claude <noreply@anthropic.com>
@codspeed-hq

codspeed-hq Bot commented Aug 24, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 12.5%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
✅ 1980 untouched benchmarks
⏩ 54 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime words_gather_dispatch_avx512[1024] 9 ns 8 ns +12.5%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/fsst-binary-scheme (81d6bb8) with develop (e4b3421)2

Open in CodSpeed

Footnotes

  1. 54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on claude/varbin-scheme (d598e09) during the generation of this report, so develop (e4b3421) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants