Skip to content

Add T64 and GORILLA packing transform codecs - #19306

Open
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/03-delta-codecsfrom
xiangfu0/codex/codec-stack/04-t64-gorilla
Open

Add T64 and GORILLA packing transform codecs#19306
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/03-delta-codecsfrom
xiangfu0/codex/codec-stack/04-t64-gorilla

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Stacked on #19305 (xiangfu0/codex/codec-stack/03-delta-codecs). Part of the split of #18229.

Summary

Adds the two packing transform codecs to the package-private codec runtime in
pinot-segment-local org.apache.pinot.segment.local.io.codec:

  • T64CodecDefinition — frame-of-reference (FOR) bit-packing over fixed 64-value blocks for
    SV INT/LONG. Each block stores [baseline][bitWidth][bit-packed deltas] inside a
    [flag][count] frame. Partial last blocks still emit full 64-slot packed payloads (zero-filled
    tail) so payload size scales with block count.
  • GorillaCodecDefinition — Gorilla-style XOR compression (Facebook VLDB 2015) for SV
    INT/LONG: verbatim first value, then per-value 0 bit for repeats or
    1 + window metadata + meaningful bits, with window reuse between consecutive XOR deltas.

Both are CodecKind.TRANSFORM stages with preservesTypedValueLayout() == false: they consume a
typed value array and emit a headered byte frame, so the existing CodecPipelineValidator
structural rule automatically enforces that they must be the last transform (only compression
stages may follow). No validator changes were needed — the rule generalizes from the synthetic
PACKING handler introduced in PR 02.

Both are registered in CodecRegistry.DEFAULT (names T64 / GORILLA are frozen on-disk
contract strings). Both codecs are corrupt-segment defensive: header flag/count validation runs
before any count-based short-circuit, bit widths and window (leading, width) pairs are
range-checked, and decodeInto enforces dst-capacity bounds.

Tests

  • T64CodecDefinitionTest — round-trips for empty/single/all-equal/partial-block/max-range
    inputs; a parameterized sweep over every bitWidth 1..32 (INT) and 1..64 (LONG); an on-disk
    byte-layout pin test; encoded-size-within-maxEncodedSize bound checks; corrupt-segment
    defenses (bad flag, bad bitWidth, negative count, frame/column type mismatch).
  • GorillaCodecDefinitionTest — round-trips for repeat/window-reuse/explicit-window transitions,
    boundary values, timestamp/counter-like sequences; corrupt-segment defenses including
    window-reuse-before-explicit.
  • CodecPipelineValidatorTest — real-codec packing-rule scenarios through
    CodecRegistry.DEFAULT: DELTA,T64,LZ4 / DELTADELTA,GORILLA,ZSTD(3) / DELTA,T64 accepted;
    compression-after-packing (T64,ZSTD(3), GORILLA,SNAPPY, T64,LZ4,GZIP) accepted;
    T64,DELTA and T64,GORILLA rejected; type restrictions and no-arg enforcement;
    preservesTypedValueLayout() contract pinned to false for both.
  • CodecRegistryTestT64/GORILLA now resolve from the default registry (case-insensitive).
  • TableConfigUtilsTest — a mis-ordered T64,DELTA,LZ4 codecSpec is still gate-rejected at
    config time with codecSpec is not supported yet for column: intCol.

Deliberately excluded (carried by later PRs)

  • V7 raw forward index format (writer/reader that actually executes codec pipelines against
    segment chunks) — next PR in the stack (05, from local codec-split/07-v7-format).
  • Opening the feature gate (config/creator/reader/reload surfaces accepting codecSpec,
    reload change-detection via canonical specs) — final PR (from local codec-split/08-reload-enable).
  • Any public SPI exposure: both codecs are package-private inside io.codec; the only
    cross-package entry point remains CodecPipelineExecutor.

Why master stays safe with only this merged

The codecSpec feature gate is untouched and still closed: every config/creator/reader/reload
surface continues to throw codecSpec is not supported yet for column: %s, so no production
path can reach these codecs. The new classes are package-private additions to io.codec that are
only reachable through CodecPipelineExecutor, which nothing in production instantiates yet.
No on-disk format, wire protocol, or public API changes.

Verification

./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl pinot-segment-local -am
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local \
  -Dtest=T64CodecDefinitionTest,GorillaCodecDefinitionTest,CodecPipelineValidatorTest,CodecRegistryTest,DeltaCodecRoundTripTest,CodecPipelineExecutorTest \
  -Dsurefire.failIfNoSpecifiedTests=false
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local -Dtest=TableConfigUtilsTest \
  -Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format -pl pinot-segment-local
./mvnw checkstyle:check license:check -pl pinot-segment-local

Stack (split of #18229)

  1. Fix forward index reader context lifecycle #19281 — Fix forward index reader context lifecycle (merged)
  2. Fix delta forward index chunk caching #19282 — Fix delta forward index chunk caching (merged)
  3. Add codec spec DSL and configuration plumbing #19284 — Codec spec DSL and configuration plumbing
  4. Add bounded codec runtime and compression handlers #19285 — Bounded codec runtime and compression handlers
  5. Add DELTA and DELTADELTA transform codecs #19305 — DELTA and DELTADELTA transform codecs
  6. Add T64 and GORILLA packing transform codecs #19306 — T64 and GORILLA packing transform codecs ← this PR
  7. Add V7 raw forward index format for codec pipelines #19307 — V7 raw forward index format
  8. Support codecSpec on segment reload and enable codecSpec end to end #19308 — Reload/rewrite support, gate removal (enabling PR)
  9. Add codec pipeline integration tests and design doc #19309 — Integration tests and design doc

Each PR is based on its predecessor's branch; GitHub retargets the next PR as each merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant