feat(btrblocks): compress binary arrays with VarBin offsets and FSST

[joseph-isaacs]

Summary

Binary columns had exactly one scheme in ALL_SCHEMES: BinaryDictScheme. Once the dictionary declined a column, nothing else was eligible, so the array fell through to a canonical VarBinViewArray written as payload plus a fixed 16 bytes per element of opaque views buffer — regardless of content.

Two consequences, both measurable:

• Size became insensitive to the data. A binary column that zlib -9 shrinks 6.8x was written at exactly the same size as an incompressible one, because no scheme ran on either.
• The per-element overhead was a flat 16.0 B/value independent of width, so a 16-byte column doubled (2.00x payload) and a 4-byte column quintupled (5.01x).

For comparison, the same values typed as Utf8 compressed to 713,324 bytes where Binary produced 3,204,100 — a 4.5x gap decided purely by dtype.

This PR gives binary two schemes that already existed in spirit elsewhere in the compressor, and adds the tests to pin their behavior.

Changes

VarBinScheme (new, schemes/binary/varbin.rs) re-encodes the canonical VarBinViewArray as a VarBinArray, replacing the opaque views buffer with an offsets child array. That child is then compressed by the ordinary integer schemes, so a fixed-width column's constant-stride offsets collapse to a sequence and the per-element overhead disappears. This mirrors what FSSTScheme already does for strings, which builds its codes as a VarBinArray with compressed offsets.

FSSTScheme now matches binary as well as utf8. Its matches gated on is_utf8(), but the compress path never validates UTF-8 — it trains and compresses over the raw array_as_varbinview() bytes. The gate excluded binary from a scheme that already worked on it, leaving any binary payload with intra-value structure (shared prefixes, zero padding, common framing) uncompressed once the dictionary declined it.

The two are complementary rather than competing, and selection picks correctly between them: FSST wins where the payload has structure, VarBinScheme wins on incompressible data where FSST's symbol table is pure overhead.

Measured at 100k rows, compressed nbytes against the same compressor with these schemes excluded:

case	before	after	selected
shared prefix	3,200,000	734,685	FSST
nulls every 7th	2,966,827	690,230	FSST
random 16B	3,200,000	1,600,000	VarBin
random 256B	27,200,000	25,600,000	VarBin

The random 16B case is now at its payload floor (1,600,000 bytes of incompressible data) with no metadata overhead.

Tests (tests/varbin_scheme.rs) cover both schemes with roundtrip assertions through assert_arrays_eq!, including a nullable case, plus two invariants: enabling VarBinScheme must never grow the output, and identical values must compress identically whether typed binary or utf8 (FSST compresses bytes, not codepoints).

One golden snapshot moves, and it improves: binary_low_cardinality dictionary values go from 96 to 52 bytes as VarBinScheme cascades into the dictionary's values child.

Review notes

• VarBinScheme is registered in ALL_SCHEMES, so it affects every binary column on a default write. Worth confirming that blast radius is intended rather than gating it behind with_compact.
• On the FSST gate: compress demonstrably does not validate UTF-8, but I have not read through fsst_train_compressor/fsst_compress. The byte-identical binary/utf8 assertion is evidence the widening is safe, not proof. If the original restriction was deliberate for a reason deeper in the FSST implementation, that is the part to push back on.

Checks

• cargo test -p vortex-btrblocks — all pass
• cargo test -p vortex-file — all pass
• cargo clippy -p vortex-btrblocks --all-targets --all-features — clean
• cargo +nightly fmt --all

Not measured: decode throughput for either scheme, and the file-level (rather than in-memory nbytes) size effect. Both are worth a follow-up before leaning on these numbers for anything latency-sensitive.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GJdgPga43u5rXYwv6R5b19

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will degrade performance by 10.53%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

❌ 1 regressed benchmark
✅ 1980 untouched benchmarks
⏩ 54 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	WallTime	words_gather_dispatch_avx2[1024]	17 ns	19 ns	-10.53%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/binary-compression-schemes (54fb31b) with develop (e4b3421)}

Footnotes

1. 54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[lwwmanning]

surely this isn't the most efficient way to do this?