Support codecSpec on segment reload and enable codecSpec end to end

[xiangfu0]

Stacked on #19307 (xiangfu0/codex/codec-stack/05-v7-format). Part of the split of #18229.

Summary

This is the enabling PR of the series: it removes every remaining codecSpec feature-gate site and
replaces the gates with real behavior — semantic table-config-time validation, reload/rewrite support in
ForwardIndexHandler, and rollback back to the legacy formats.

Reload / rewrite (ForwardIndexHandler)

shouldChangeRawCompressionType is replaced by shouldRewriteRawForwardIndex, which handles every
combination of existing on-disk format × new config:

Existing on disk	New config	Decision
Legacy raw (compression C1)	no codecSpec, compression C2	rewrite iff C1 != C2 (unchanged legacy behavior)
Legacy raw (compression C1)	codecSpec mapping to legacy C2	rewrite iff C1 != C2 (stays in legacy format)
Legacy raw	codecSpec needing V7 (transform/chain/non-default options)	rewrite to V7 codec-pipeline format
V7 (stored canonical spec S1)	codecSpec needing V7, canonical S2	rewrite iff S1 != S2
V7	codecSpec mapping to a legacy compression	rewrite back to the legacy raw format
V7	no codecSpec, explicit legacy compressionCodec (incl. PASS_THROUGH)	rewrite back to the legacy format (rollback path — isLegacyRevertTargetForFixedByteSv)
V7	no codecSpec, no compressionCodec	no-op

Change detection compares canonical specs on both sides: the stored side is already canonical (V7
headers persist CodecPipelineExecutor.getCanonicalSpec()), and the configured side is canonicalized by
running it through CodecPipelineExecutor.create(...) — never toDslString() or the raw config string —
so semantically identical specs (e.g. differently-spelled defaults) never trigger a rewrite, and an
unchanged config is always a no-op.

V7 detection is positive (reader.getCodecSpec() != null), not inferred from a null compression type; a
legacy raw reader returning a null ChunkCompressionType now fails loudly instead of silently skipping.

Gate removal (all remaining sites)

• ForwardIndexType.validate — gate replaced by validateCodecSpec(...), run only for enabled forward
  indexes: parses/validates the pipeline via the public CodecPipelineExecutor.create entry point
  (unknown codec, transform ordering, per-codec type compatibility) and enforces the V7 writer's shape
  constraints (single-value INT/LONG) for specs the legacy raw formats cannot serve. Precise errors
  replace the generic gate message.
• ForwardIndexType.shouldCreateIndex — gate removed; codecSpec configs flow to the creator factory
  (wired in the previous PR).
• ForwardIndexType.createMutableIndex — gate removed. Matching the original Add codec pipeline framework for raw forward index encoding #18229 behavior, the
  mutable (consuming) forward index builds the standard in-memory format and deliberately ignores
  codecSpec: the spec applies when the consuming segment is converted/committed to an immutable
  segment. Realtime tables with a codecSpec consume normally.
• ForwardIndexHandler.computeOperations — the fail-fast guard from the previous PR is replaced by the
  real decision logic above.
• OpenStructIndexType / OpenStructColumnSplitter — kept as a deliberate rejection, reworded from
  the temporary gate message to "codecSpec is not supported for OPEN_STRUCT key: %s". The original
  Add codec pipeline framework for raw forward index encoding #18229 never wired codecSpec into OPEN_STRUCT per-key child columns (the splitter builds its own child
  forward configs with a fixed LZ4 raw compression and would silently discard a per-key codecSpec), so an
  explicit rejection is the honest behavior rather than silent config loss.

Docs

• FieldConfig.CompressionCodec constants get "prefer codecSpec" guidance in the new comma-list syntax
  (SNAPPY→"SNAPPY", ZSTANDARD→"ZSTD(3)", DELTA→"DELTA,LZ4" with the on-disk-semantics
  caveat, etc.); getCompressionCodec() documents that it remains the only way to express
  MV_ENTRY_DICT and the CLP family.
• TableConfigUtils comment updated to the final behavior (codecSpec is validated on the resolved
  ForwardIndexConfig via IndexType.validate, so noDictionaryColumns-resolved RAW columns work).

Tests

• ForwardIndexHandlerTest: V7 no-op on unchanged canonical spec; V7 spec-change rewrite; V7
  compression-only spec rewritten back to legacy raw; V7→legacy compressionCodec rollback with full
  value verification; legacy→equivalent codecSpec no-op; legacy→different compression-only codecSpec
  rewrite; legacy DELTA/DELTADELTA → V7 DELTA,LZ4/DELTADELTA,LZ4 migration reload preserving all
  values across chunks.
• TableConfigUtilsTest: gate-rejection test replaced by testCodecSpecValidation — accepts valid specs
  (compression-only on any RAW column incl. STRING/MV, transform chains on SV INT/LONG,
  noDictionaryColumns-resolved RAW), rejects invalid ones with precise errors (unknown codec, transform
  after packing transform, V7-requiring specs on MV or non-INT/LONG columns).
• ForwardIndexTypeTest: mutable-index gate test replaced — a codecSpec config now builds the standard
  mutable forward index for realtime consumption.
• ForwardIndexCreatorFactoryTest: shouldCreateIndex gate test flipped to acceptance.
• OpenStructIndexTypeTest / OpenStructColumnSplitterTest: assertions updated to the permanent
  rejection message.

Deliberately excluded

• CompressionCodecMigrator and its tests (present in the original Add codec pipeline framework for raw forward index encoding #18229) are dropped from the series
  entirely — legacy DELTA/DELTADELTA columns migrate through the normal reload path instead, covered
  by testLegacyTransformMigrationReloadPreservesValues.
• codecSpec on OPEN_STRUCT per-key child columns is explicitly rejected (see above); no later PR in this
  series adds it.

The final slice (integration tests + design doc) follows this PR but adds no production code.

Why master stays safe with only this merged

This PR is the gate-opener, so unlike the earlier slices it does activate the feature — safety comes
from the pipeline being complete underneath it:

• All runtime pieces (DSL, validator, executor, transforms, V7 format, creator/reader wiring) merged in
  the previous five PRs; this PR only removes the gates and adds the reload logic.
• No existing config changes behavior: columns without codecSpec follow the exact pre-series code
  paths (the legacy compression-change comparison is byte-for-byte the old logic), and reload of a
  segment with an unchanged config is a no-op.
• Rollback is first-class: reverting a table config from codecSpec to a legacy compressionCodec
  (or an equivalent compression-only spec) rewrites segments back to the legacy formats readable by
  pre-V7 servers.
• Realtime ingestion is unaffected: the mutable index ignores codecSpec; it only applies at segment
  conversion.
• Mixed-version caution still applies as designed for the series: V7 segments are only produced when an
  operator explicitly configures a V7-requiring codecSpec, and older servers cannot read them — the
  standard guidance (finish the server rollout before adopting V7-requiring specs) is unchanged from
  Add codec pipeline framework for raw forward index encoding #18229.

Verification

./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl pinot-spi,pinot-segment-spi,pinot-segment-local -am
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local \
  -Dtest=ForwardIndexHandlerTest,TableConfigUtilsTest,ForwardIndexTypeTest,CodecPipelineForwardIndexTest,OpenStructIndexTypeTest,OpenStructColumnSplitterTest,ForwardIndexCreatorFactoryTest \
  -Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format -pl pinot-spi,pinot-segment-local
./mvnw checkstyle:check license:check -pl pinot-spi,pinot-segment-local

Result: all tests pass. The stack root is synced with master, so this branch includes the #19282
chunk-caching fix that testLegacyTransformMigrationReloadPreservesValues depends on (verified: a
523-test batch across the codec suite at the chain tip runs green, including both migration variants
and ForwardIndexHandlerContextTest).

Stack (split of #18229)

1. Fix forward index reader context lifecycle #19281 — Fix forward index reader context lifecycle (merged)
2. Fix delta forward index chunk caching #19282 — Fix delta forward index chunk caching (merged)
3. Add codec spec DSL and configuration plumbing #19284 — Codec spec DSL and configuration plumbing
4. Add bounded codec runtime and compression handlers #19285 — Bounded codec runtime and compression handlers
5. Add DELTA and DELTADELTA transform codecs #19305 — DELTA and DELTADELTA transform codecs
6. Add T64 and GORILLA packing transform codecs #19306 — T64 and GORILLA packing transform codecs
7. Add V7 raw forward index format for codec pipelines #19307 — V7 raw forward index format
8. Support codecSpec on segment reload and enable codecSpec end to end #19308 — Reload/rewrite support, gate removal (enabling PR) ← this PR
9. Add codec pipeline integration tests and design doc #19309 — Integration tests and design doc

Each PR is based on its predecessor's branch; GitHub retargets the next PR as each merges.

[xiangfu0]

Pushed aafe8c6fd7 correcting the CompressionCodec Javadoc this PR adds. Two issues, both found while assessing whether compressionCodec could be deprecated in favour of codecSpec:

1. The DELTA/DELTADELTA guidance described a migration that is not possible. The previous wording ("For new SV INT/LONG columns, prefer codecSpec=..." / "migrating to codecSpec changes on-disk semantics") implied users have a working compressionCodec: DELTA config to migrate from. They do not: both constants are declared (false, false), so ForwardIndexType.validateForwardIndexEnabled rejects them on the dictionary branch (isApplicableToDictEncodedIndex()) and on the raw branch (isApplicableToRawIndex()) alike. I verified this empirically rather than by inspection — a table config with compressionCodec: DELTA on a RAW single-value INT column, the exact shape the codec targets, fails validation:

IllegalStateException: Compression codec: DELTA is not applicable to raw column: intCol

Same for DELTADELTA and for LONG. Note this makes TableConfigUtils.validateGorillaCompressionCodecIfPresent (the SV / INT-LONG checks for DELTA and DELTADELTA) effectively unreachable on the accept path — the applicability check fires first. That is pre-existing and out of scope here, but worth knowing.

The Javadoc now states plainly that these values are rejected by table-config validation and points to codecSpec="DELTA,LZ4" as the supported route, rather than framing it as a migration.

2. The six constants with no codecSpec equivalent had no Javadoc at all, which read as an oversight next to the four annotated ones. PASS_THROUGH, MV_ENTRY_DICT and the CLP family now each say so explicitly and why:

• PASS_THROUGH — no equivalent; the DSL has no identity codec and rejects a blank spec, and this is the default for METRIC columns, so it stays the only way to state "uncompressed".
• MV_ENTRY_DICT — no equivalent; codecSpec applies only to RAW forward indexes.
• CLP family — no equivalent; whole-index formats for STRING columns rather than chunk codecs, validated against stored type instead of the applicability flags.

I also fixed the malformed {@link CLPForwardIndexCreatorV2) reference in that comment (mismatched brace/paren, and those classes are not reachable from pinot-spi anyway) by using code spans.

Net effect: the guidance now covers all 12 constants and matches what validation actually does. Comment-only change, no behavior difference.

Verification: pinot-spi compiles; IndexCombinationValidationTest, TableConfigUtilsTest, ForwardIndexTypeTest all green (169 tests); spotless/checkstyle/license clean. #19309 has been rebased onto the new head so the chain stays intact.