Add codec pipeline framework for raw forward index encoding

[xiangfu0]

Summary

This PR adds a composable codecSpec DSL for raw forward indexes at:

fieldConfigList[].indexes.forward.codecSpec

It deliberately keeps two on-disk paths:

• A single compression invocation that maps exactly to an existing ChunkCompressionType (LZ4, SNAPPY, GZIP, ZSTD, or ZSTD(3)) stays on the existing raw forward-index format.
• Any transform, compression chain, or non-default compression option such as ZSTD(5) uses a self-describing fixed-byte V7 format. This V7 path is currently limited to RAW single-value INT and LONG columns.

Valid pipelines have this shape:

zero or more typed-layout-preserving transforms
  -> at most one packing transform
  -> zero or more compression stages

Encode runs left-to-right and decode runs right-to-left. Examples include CODEC(DELTA,T64,LZ4) and multiple compression stages. The eight built-ins are DELTA, DELTADELTA, T64, GORILLA, ZSTD, LZ4, SNAPPY, and GZIP.

Design details: docs/design/codec-pipeline-v7.md

Configuration

{
  "fieldConfigList": [
    {
      "name": "ts",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "codecSpec": "CODEC(DELTADELTA,LZ4)"
        }
      }
    },
    {
      "name": "userId",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "codecSpec": "ZSTD(3)"
        }
      }
    }
  ]
}

codecSpec and the legacy compressionCodec are mutually exclusive. The legacy getter and enum remain supported because other column shapes and codec families still depend on them.

V7 format and safety

V7 stores the canonical codec spec in the index header and uses the pair (version 7, codec-pipeline magic) as its format discriminator. The integer version alone is not sufficient because the existing legacy fixed-byte writer accepts arbitrary version tags greater than or equal to 4, including 7.

The reader and executor fail closed on malformed segment data. They validate the fixed header, format magic, codec-spec size, chunk geometry, document/chunk counts, offset-table extents, per-chunk payload bounds, exact decoded sizes, codec frame completion, and composed allocation/work limits before large allocations or native reads. Intermediate direct buffers and reader contexts are released deterministically.

Compatibility and migration

Concern	Behavior
Existing compressionCodec configs	Supported
Legacy fixed-byte versions 2, 3, and arbitrary tags >= 4	Supported, including legacy version 7
Legacy-compatible single-compression codecSpec	Uses the existing raw segment format
Any codecSpec on pre-1.6 config consumers	Not supported; upgrade controllers, servers, and relevant segment builders first
V7-requiring specs on old servers	Not supported; upgrade every possible segment reader before enabling them
ForwardIndexReader.getCodecSpec()	Added as a backward-compatible default SPI method
Config transitions	Legacy-to-V7, V7-to-legacy, V7 codec changes, and legacy compression changes trigger the appropriate rewrite

CompressionCodecMigrator translates supported legacy configs. Reload parity coverage verifies that legacy DELTA and DELTADELTA segments retain every value through migration and rewrite. Golden fixtures generated before this change lock legacy fixed-byte v6/v7 readability, and the rollback test executes a V7-to-legacy rewrite and reopens the resulting index.

For a rolling upgrade, upgrade every component that validates or consumes table configs before enabling any codecSpec. For V7-requiring specs, every server that can load the table's segments must also have the V7 reader. To downgrade, first restore a legacy-representable config and reload segments so V7 indexes are rewritten.

Validation

Local validation on commit 9d12b82c55 rebased on apache/master (388bc64d5c):

• Focused codec, config, compatibility, migration, corruption, and handler reactor: 340 tests passed.
• CodecPipelineIntegrationTest: 75 tests passed across SSE and MSE, including multi-chunk boundary assertions.
• TableConfigUtilsTest: 78 tests passed after correcting the CI assertion to verify both the established wrapper and the semantic root cause.
• spotless:apply, checkstyle:check, license:format, and license:check passed for pinot-segment-spi, pinot-segment-local, and pinot-integration-tests.
• Final configuration/backward-compatibility, correctness/concurrency/performance, and architecture/API review passes found no remaining P0-P2 issues.

All existing review threads are resolved, and all 12 GitHub Actions checks passed on the amended head. Human approval is still requested.

[Copilot]

Pull request overview

Adds an initial “codec pipeline” framework for raw (no-dict) single-value forward index encoding, introducing a DSL (DELTA, ZSTD(N), CODEC(...)) and a new self-describing on-disk format (writer/reader v7) that persists the canonical codec spec in the file header. This is wired through ForwardIndexConfig.codecSpec and the forward-index creator/reader factories, with tests covering parsing, validation, and INT/LONG round-trips.

Changes:

• Introduce codec DSL AST + parser in pinot-segment-spi, plus ForwardIndexConfig.codecSpec (mutually exclusive with compressionCodec) and writer-version forcing.
• Add codec registry/validator/executor in pinot-segment-local and implement v7 fixed-byte chunk writer/reader that stores the canonical spec in the header.
• Wire new creator/reader paths via ForwardIndexCreatorFactory and ForwardIndexReaderFactory, and add comprehensive unit tests.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/index/ForwardIndexConfigTest.java	Adds tests for codecSpec JSON round-trip, equality, and mutual-exclusion/ordering guards.
pinot-segment-spi/src/test/java/org/apache/pinot/segment/spi/codec/CodecSpecParserTest.java	New unit tests for codec DSL parsing/canonicalization and invalid specs.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java	Adds codecSpec, forces raw writer version 7 when set, updates equals/hashCode and Builder guards.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/codec/CodecSpecParser.java	Implements structural (phase-1) recursive-descent parser for the DSL.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/codec/CodecPipeline.java	New AST node representing an ordered pipeline with canonical spec rendering.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/codec/CodecOptions.java	Marker interface for typed, validated per-codec options.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/codec/CodecKind.java	Enum classifying codecs as TRANSFORM vs COMPRESSION for pipeline rules.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/codec/CodecInvocation.java	New AST node representing a single codec invocation + args.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/codec/CodecDefinition.java	SPI interface describing codec parsing/validation/canonicalization.
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/codec/CodecContext.java	Context object for per-column type validation during pipeline validation.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/forward/CodecPipelineForwardIndexTest.java	Integration tests for v7 writer/reader round-trip, header spec storage, factory dispatch, partial chunk.
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/io/codec/CodecPipelineValidatorTest.java	Tests pipeline validation rules (ordering, type checks, unknown codecs, arg ranges).
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/forward/FixedByteChunkSVForwardIndexReaderV7.java	New v7 reader that reads canonical spec from header and decodes chunks via executor.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexReaderFactory.java	Dispatches fixed-width SV version-7 raw indexes to the new v7 reader.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java	Creates codec-pipeline raw forward index creators when codecSpec is present and supported.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/SingleValueFixedByteCodecPipelineIndexCreator.java	New creator wiring v7 writer + executor for INT/LONG SV raw forward indexes.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/writer/impl/FixedByteChunkForwardIndexWriterV7.java	New v7 writer that embeds canonical spec in header and writes variable-size encoded chunks with long offsets.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/codec/ZstdCodecDefinition.java	Adds ZSTD codec definition with typed options and canonicalization.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/codec/DeltaCodecDefinition.java	Adds DELTA transform codec definition with INT/LONG validation and canonicalization.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/codec/CodecRegistry.java	Introduces immutable default registry + mutable registry for tests/future plugins, with reserved keyword protection.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/codec/CodecPipelineValidator.java	Validates structural rules and per-codec context compatibility for pipelines.
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/io/codec/CodecPipelineExecutor.java	Executes the validated pipeline per chunk (DELTA + ZSTD v1 wiring) and produces canonical spec.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 11.43740% with 1719 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.35%. Comparing base (968bf3c) to head (aa402ff).

Files with missing lines	Patch %	Lines
...ocal/segment/index/loader/ForwardIndexHandler.java	14.13%	242 Missing and 7 partials ⚠️
...segment/local/io/codec/GorillaCodecDefinition.java	0.00%	192 Missing ⚠️
...not/segment/local/io/codec/T64CodecDefinition.java	0.00%	177 Missing ⚠️
.../forward/FixedByteChunkSVForwardIndexReaderV7.java	0.00%	132 Missing and 1 partial ⚠️
...riter/impl/FixedByteChunkForwardIndexWriterV7.java	0.00%	124 Missing ⚠️
.../segment/local/io/codec/CodecPipelineExecutor.java	0.00%	121 Missing ⚠️
...ot/segment/local/io/codec/GzipCodecDefinition.java	0.00%	112 Missing ⚠️
...ment/local/io/codec/DeltaDeltaCodecDefinition.java	0.00%	99 Missing ⚠️
.../segment/local/utils/CompressionCodecMigrator.java	0.00%	80 Missing ⚠️
...ot/segment/local/io/codec/ZstdCodecDefinition.java	0.00%	69 Missing ⚠️
... and 19 more

❗ There is a different number of reports uploaded between BASE (968bf3c) and HEAD (aa402ff). Click for more details.

HEAD has 4 uploads less than BASE

Flag	BASE (968bf3c)	HEAD (aa402ff)
unittests	2	1
java-25	6	5
temurin	6	5
unittests2	1	0

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #18229      +/-   ##
============================================
- Coverage     67.10%   57.35%   -9.76%     
+ Complexity     1424        7    -1417     
============================================
  Files          3457     2683     -774     
  Lines        219288   161042   -58246     
  Branches      34865    26488    -8377     
============================================
- Hits         147148    92359   -54789     
- Misses        60378    60832     +454     
+ Partials      11762     7851    -3911     

Flag	Coverage Δ
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	57.35% <11.43%> (-9.76%)	⬇️
lane-a	100.00% <ø> (ø)
lane-b	0.00% <ø> (ø)
temurin	57.35% <11.43%> (-9.76%)	⬇️
unittests	57.34% <11.43%> (-9.76%)	⬇️
unittests1	57.34% <11.43%> (-0.47%)	⬇️
unittests2	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 10 comments.

[xiangfu0]

Found one high-confidence correctness issue; see inline comment.

[Copilot]

Pull request overview

Copilot reviewed 40 out of 40 changed files in this pull request and generated 6 comments.

[xiangfu0]

Found one high-confidence backward-compatibility issue; see inline comment.

[Copilot]

Pull request overview

Copilot reviewed 52 out of 52 changed files in this pull request and generated 3 comments.

[xiangfu0]

Codec chaining now supported (revisits the earlier single-transform constraint)

Following up on the earlier discussion where we restricted pipelines to a single transform: the pipeline now supports full chaining of the form

N value-preserving transforms → at most one packing transform → N compressions

Examples that are now valid and round-trip end-to-end (covered by CodecPipelineForwardIndexTest):

• CODEC(DELTA, DELTADELTA, LZ4) — chained value transforms + compression
• CODEC(DELTA, T64, LZ4) — delta → frame-of-reference bit-pack → compress
• CODEC(DELTA, LZ4, ZSTD(3)) — multiple chained compressions

Why multi-transform is now safe (not the earlier corruption case)

The original concern was that CODEC(DELTA, DELTA) would silently corrupt, because each transform embedded a per-frame [flag][count] header and assumed raw column-typed input — so feeding one transform's headered output into the next was undefined.

That's fixed at the root: DELTA/DELTADELTA are now header-less, value-preserving passthrough transforms — they map a typed value array to a same-width typed value array, with the element type taken from the column context and the value count from the buffer length. So a following transform receives exactly what it expects, and the chain composes correctly. This is enforced, not just hoped for:

• New SPI contract CodecDefinition.isValuePreserving() — true for DELTA/DELTADELTA, false for packing transforms (T64/GORILLA) and compressions.
• CodecPipelineValidator walks the stages tracking a "typed-value domain". A TRANSFORM may only appear while still in that domain; a packing transform (T64/GORILLA, whose bit-packed output is no longer a typed value array) or any compression ends it. So a packing transform must be the last transform, and transforms must precede compressions.

Still rejected (by necessity)

• CODEC(T64, DELTA, …) / CODEC(T64, GORILLA) — a packing transform's output isn't a typed value array, so nothing typed may follow it.
• CODEC(ZSTD(3), DELTA) — a transform can't consume compressed bytes.

Wire formats: DELTA/DELTADELTA are now header-less (passthrough); T64/GORILLA keep their self-framed formats and remain terminal among transforms. Design doc (§2, §5, §12) updated. Happy to adjust the model if you'd prefer to keep the stricter constraint.

[Jackie-Jiang]

Can we split it into smaller PRs? It is too large for review.

I think we can first introduce the DSL and wiring of the config.
For the DSL, given it is already under _codecSpec, can we make it a list of invocation without the CODEC wrapper?

[Jackie-Jiang]

Also, why is variable length value reader not a goal?

[xiangfu0]

Thanks, Jackie. I have started splitting this into GitHub-native stacked PRs:

• Add codec spec DSL and configuration plumbing #19284 introduces the DSL and ForwardIndexConfig.codecSpec wiring first. The outer CODEC(...) wrapper is removed; the syntax is now an ordered invocation list such as DELTA,ZSTD(3).
• Add bounded codec runtime and compression handlers #19285 adds only the bounded internal runtime and the four compression handlers. It remains inert because all layer-1 gates stay closed.
• Fix forward index reader context lifecycle #19281 and Fix delta forward index chunk caching #19282 extract two independent correctness prerequisites so they can be reviewed and merged separately.

The V7 reader/writer, activation, and transforms will remain separate follow-up stack layers rather than returning to one large PR.

Variable-length values are deferred from the first new-format slice, not a permanent non-goal. A single legacy-compatible compression spec can eventually reuse the existing variable-byte readers and writers. General chained or non-default codec pipelines need an explicit variable-width envelope because the current proposed V7 layout assumes fixed entry width, derives decoded bytes from document count times width, and uses fixed-width random access. Numeric transforms such as DELTA and packing transforms also remain INT/LONG-specific. I will keep variable-width SV/MV pipeline support as a linked follow-up after the fixed-width format contract is reviewed rather than silently freezing the current fixed-width assumptions into its future design.

[xiangfu0]

Closing this large umbrella as superseded by the reviewable split requested in the Aug 17 review.

Complete codec stack

Merged prerequisites:

• Fix forward index reader context lifecycle #19281 — Fix forward-index reader context lifecycle
• Fix delta forward index chunk caching #19282 — Fix legacy DELTA/DELTADELTA chunk caching

Native GitHub stack (review and merge parent-first):

1. Add codec spec DSL and configuration plumbing #19284 — Codec spec DSL and configuration plumbing
2. Add bounded codec runtime and compression handlers #19285 — Bounded codec runtime and compression handlers
3. Add DELTA and DELTADELTA transform codecs #19305 — DELTA and DELTADELTA transform codecs
4. Add T64 and GORILLA packing transform codecs #19306 — T64 and GORILLA packing transform codecs
5. Add V7 raw forward index format for codec pipelines #19307 — V7 raw forward index format
6. Validate codecSpec and support codec-aware forward-index reloads #19308 — Reload/rewrite support and feature activation
7. Add codec pipeline integration tests and design doc #19309 — Integration tests and design documentation

Chain: #19284 → #19285 → #19305 → #19306 → #19307 → #19308 → #19309.

All open branches were restacked onto current master on Aug 20. Each child uses its predecessor's branch as its GitHub base, so GitHub will retarget the next PR as each parent merges. This PR remains available as the original design and review history.