Add T64 and GORILLA packing transform codecs - #19306
Open
xiangfu0 wants to merge 1 commit into
Open
Conversation
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #19305 (
xiangfu0/codex/codec-stack/03-delta-codecs). Part of the split of #18229.Summary
Adds the two packing transform codecs to the package-private codec runtime in
pinot-segment-localorg.apache.pinot.segment.local.io.codec:T64CodecDefinition— frame-of-reference (FOR) bit-packing over fixed 64-value blocks forSV INT/LONG. Each block stores
[baseline][bitWidth][bit-packed deltas]inside a[flag][count]frame. Partial last blocks still emit full 64-slot packed payloads (zero-filledtail) so payload size scales with block count.
GorillaCodecDefinition— Gorilla-style XOR compression (Facebook VLDB 2015) for SVINT/LONG: verbatim first value, then per-value
0bit for repeats or1 + window metadata + meaningful bits, with window reuse between consecutive XOR deltas.Both are
CodecKind.TRANSFORMstages withpreservesTypedValueLayout() == false: they consume atyped value array and emit a headered byte frame, so the existing
CodecPipelineValidatorstructural rule automatically enforces that they must be the last transform (only compression
stages may follow). No validator changes were needed — the rule generalizes from the synthetic
PACKINGhandler introduced in PR 02.Both are registered in
CodecRegistry.DEFAULT(namesT64/GORILLAare frozen on-diskcontract strings). Both codecs are corrupt-segment defensive: header flag/count validation runs
before any count-based short-circuit, bit widths and window
(leading, width)pairs arerange-checked, and
decodeIntoenforces dst-capacity bounds.Tests
T64CodecDefinitionTest— round-trips for empty/single/all-equal/partial-block/max-rangeinputs; a parameterized sweep over every bitWidth 1..32 (INT) and 1..64 (LONG); an on-disk
byte-layout pin test; encoded-size-within-
maxEncodedSizebound checks; corrupt-segmentdefenses (bad flag, bad bitWidth, negative count, frame/column type mismatch).
GorillaCodecDefinitionTest— round-trips for repeat/window-reuse/explicit-window transitions,boundary values, timestamp/counter-like sequences; corrupt-segment defenses including
window-reuse-before-explicit.
CodecPipelineValidatorTest— real-codec packing-rule scenarios throughCodecRegistry.DEFAULT:DELTA,T64,LZ4/DELTADELTA,GORILLA,ZSTD(3)/DELTA,T64accepted;compression-after-packing (
T64,ZSTD(3),GORILLA,SNAPPY,T64,LZ4,GZIP) accepted;T64,DELTAandT64,GORILLArejected; type restrictions and no-arg enforcement;preservesTypedValueLayout()contract pinned tofalsefor both.CodecRegistryTest—T64/GORILLAnow resolve from the default registry (case-insensitive).TableConfigUtilsTest— a mis-orderedT64,DELTA,LZ4codecSpec is still gate-rejected atconfig time with
codecSpec is not supported yet for column: intCol.Deliberately excluded (carried by later PRs)
segment chunks) — next PR in the stack (05, from local
codec-split/07-v7-format).reload change-detection via canonical specs) — final PR (from local
codec-split/08-reload-enable).io.codec; the onlycross-package entry point remains
CodecPipelineExecutor.Why master stays safe with only this merged
The codecSpec feature gate is untouched and still closed: every config/creator/reader/reload
surface continues to throw
codecSpec is not supported yet for column: %s, so no productionpath can reach these codecs. The new classes are package-private additions to
io.codecthat areonly reachable through
CodecPipelineExecutor, which nothing in production instantiates yet.No on-disk format, wire protocol, or public API changes.
Verification
Stack (split of #18229)
Each PR is based on its predecessor's branch; GitHub retargets the next PR as each merges.