Skip to content

Add DELTA and DELTADELTA transform codecs - #19305

Open
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/02-runtimefrom
xiangfu0/codex/codec-stack/03-delta-codecs
Open

Add DELTA and DELTADELTA transform codecs#19305
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/02-runtimefrom
xiangfu0/codex/codec-stack/03-delta-codecs

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Stacked on #19285 (xiangfu0/codex/codec-stack/02-runtime). Part of the split of #18229.

Summary

Adds the first two transform codecs to the package-private codec runtime in
org.apache.pinot.segment.local.io.codec:

  • BaseDeltaCodecDefinition — shared structure for the delta family. Both codecs are
    CodecKind.TRANSFORM stages restricted to INT/LONG stored types, and both are
    typed-layout-preserving (preservesTypedValueLayout() == true): they map count
    column-typed values to the same count same-width values with no per-frame header, so they can
    be chained ahead of another transform (or each other) and maxEncodedSize is the identity.
    Two's-complement wrap-around on encode/decode is an intentional, test-locked contract:
    overflow-adjacent sequences round-trip bit-for-bit through the wrapping -/+ arithmetic.
  • DeltaCodecDefinition (DELTA) — first value verbatim, then successive deltas.
  • DeltaDeltaCodecDefinition (DELTADELTA) — first value, first delta, then delta-of-deltas
    (good for near-regular timestamp intervals).

Both are registered in CodecRegistry.DEFAULT, so specs like DELTA,LZ4, DELTADELTA,ZSTD(3),
and DELTA,DELTADELTA,LZ4 now parse, validate, and execute through CodecPipelineExecutor.
The names DELTA/DELTADELTA are frozen on-disk contracts (stored verbatim in V7 segment
headers by a later PR in the stack) and must never be renamed.

Tests

  • DeltaCodecRoundTripTest (new) — executor-driven round trips for DELTA, DELTADELTA,
    DELTA,LZ4, DELTADELTA,LZ4 over INT and LONG datasets: empty, single-element, two-element
    (DELTADELTA boundary), constant, monotonic, negative/sign-changing, and overflow-adjacent
    (MIN_VALUE next to MAX_VALUE) sequences; both the allocating decode path and the bounded
    decode-into (segment-reader) path. Also a fail-closed corrupt-input case: a frame whose byte
    length is not a whole number of elements is rejected on both encode and bounded decode.
  • CodecPipelineValidatorTest — real-transform scenarios against CodecRegistry.DEFAULT:
    valid delta pipelines, typed-layout chaining (including DELTA,DELTA and mixed order),
    transform-after-compression rejected, INT/LONG-only type check, argument rejection, and the
    typed-value-layout contract assertions for both codecs.
  • CodecRegistryTest — DELTA/DELTADELTA now resolve from DEFAULT (case-insensitive); the
    parent PR's absence assertion for unregistered codec names is left untouched.
  • CompressionCodecCorruptInputTest — the multi-stage inner-frame-guard test now uses
    DELTA,SNAPPY, tightening the scratch bound to exactly the four-byte declared chunk size.
  • TableConfigUtilsTest — the codecSpec feature gate still rejects a well-formed
    DELTADELTA,LZ4 config on a supported LONG column with
    "codecSpec is not supported yet for column: longCol", proving registration of the runtime
    codecs did not open the config surface.

Deliberately excluded (carried by later PRs in the stack)

  • T64 and GORILLA packing transforms — next slice (04). They are the codecs with
    preservesTypedValueLayout() == false; until then the packing-placement rules remain covered
    by the synthetic PACKING handler in CodecPipelineValidatorTest.
  • V7 raw forward index format (writer/reader that persists the canonical spec in segment
    headers) — slice 05.
  • Opening the feature gate (config/creator/reader/reload wiring, reload change-detection via
    CodecPipelineExecutor.getCanonicalSpec()) — final slice.

Why master stays safe with only this merged

The codec runtime remains package-private and unreachable from production paths: the only public
entry point is CodecPipelineExecutor, and every config surface still throws
"codecSpec is not supported yet for column: %s" (re-asserted in this PR's TableConfigUtilsTest
addition). Registering DELTA/DELTADELTA in the closed registry adds dead-until-enabled code with
full unit coverage; no segment format, wire protocol, or config behavior changes.

Verification

./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl pinot-segment-local -am
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local \
  -Dtest=DeltaCodecRoundTripTest,CodecPipelineValidatorTest,CodecRegistryTest,CodecPipelineExecutorTest,CompressionCodecCorruptInputTest \
  -Dsurefire.failIfNoSpecifiedTests=false
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local -Dtest=TableConfigUtilsTest \
  -Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format -pl pinot-segment-local
./mvnw checkstyle:check license:check -pl pinot-segment-local

Stack (split of #18229)

  1. Fix forward index reader context lifecycle #19281 — Fix forward index reader context lifecycle (merged)
  2. Fix delta forward index chunk caching #19282 — Fix delta forward index chunk caching (merged)
  3. Add codec spec DSL and configuration plumbing #19284 — Codec spec DSL and configuration plumbing
  4. Add bounded codec runtime and compression handlers #19285 — Bounded codec runtime and compression handlers
  5. Add DELTA and DELTADELTA transform codecs #19305 — DELTA and DELTADELTA transform codecs ← this PR
  6. Add T64 and GORILLA packing transform codecs #19306 — T64 and GORILLA packing transform codecs
  7. Add V7 raw forward index format for codec pipelines #19307 — V7 raw forward index format
  8. Support codecSpec on segment reload and enable codecSpec end to end #19308 — Reload/rewrite support, gate removal (enabling PR)
  9. Add codec pipeline integration tests and design doc #19309 — Integration tests and design doc

Each PR is based on its predecessor's branch; GitHub retargets the next PR as each merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant