Skip to content

Add codec pipeline integration tests and design doc - #19309

Open
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/06-reload-enablefrom
xiangfu0/codex/codec-stack/07-integration-docs
Open

Add codec pipeline integration tests and design doc#19309
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/06-reload-enablefrom
xiangfu0/codex/codec-stack/07-integration-docs

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Stacked on #19308 (xiangfu0/codex/codec-stack/06-reload-enable). Part of the split of #18229.

Summary

  • CodecPipelineIntegrationTest (pinot-integration-tests, custom/ suite): end-to-end
    coverage of indexes.forward.codecSpec on an offline table. Every supported codec spec gets its
    own INT and LONG raw column, all populated with identical values, so every codec must read back
    the same aggregates, filter counts, and point lookups on both query engines (SSE + MSE):
    • Single-stage compression: LZ4, ZSTD(3), SNAPPY, GZIP (legacy-compatible raw writer path)
    • Transform + compression chains: DELTA,LZ4, DELTA,ZSTD(3), DELTADELTA,LZ4 (V7 writer path)
    • Packing transforms: T64, GORILLA, T64,LZ4, GORILLA,ZSTD(3) (V7 writer path)
    • Per-codec SUM/filter checks, point lookups across real V7 chunk boundaries
      (targetDocsPerChunk=256 against 500-row input files), a cross-codec same-row consistency
      query, coexistence with a dictionary-encoded STRING column, and a RAW STRING column using a
      compression-only codecSpec (ZSTD(3)) alongside V7 transform columns in the same segment.
  • docs/design/codec-pipeline-v7.md: design doc for the codec pipeline framework — DSL
    grammar (matching CodecSpecParser), architecture, V7 on-disk format and reader-side corruption
    defenses, built-in codec catalog, configuration and validation, user manual (enable / migrate /
    roll back / mixed-version safety / common errors), threading, performance, backward
    compatibility, test-coverage map, and future work. Placement under docs/design/ is noted as
    subject to reviewer preference.

All spec strings use the wrapper-less comma-list DSL introduced by this series (DELTA,T64,LZ4,
ZSTD(3)). The doc mentions the removed CODEC(...) wrapper only to document that the parser
rejects it with a precise error.

Exclusions relative to the original #18229

  • CompressionCodecMigrator (and its tests) were dropped from the series; the design doc refers
    to future migration tooling instead and keeps the manual compressionCodeccodecSpec
    migration table.
  • No test in this slice depends on migration helpers; all table configs are built directly with
    indexes.forward.codecSpec.

Safety

Test-and-docs-only slice: no production code changes. The integration test only exercises code
paths introduced by the earlier slices of this series (03–06). No wire formats, configs, or public
APIs are touched.

Verification

./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl pinot-integration-tests -am
./mvnw -q test -Ppinot-fastdev -pl pinot-integration-tests \
  -Dtest=CodecPipelineIntegrationTest -Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format checkstyle:check license:check -pl pinot-integration-tests

Stack (split of #18229)

  1. Fix forward index reader context lifecycle #19281 — Fix forward index reader context lifecycle (merged)
  2. Fix delta forward index chunk caching #19282 — Fix delta forward index chunk caching (merged)
  3. Add codec spec DSL and configuration plumbing #19284 — Codec spec DSL and configuration plumbing
  4. Add bounded codec runtime and compression handlers #19285 — Bounded codec runtime and compression handlers
  5. Add DELTA and DELTADELTA transform codecs #19305 — DELTA and DELTADELTA transform codecs
  6. Add T64 and GORILLA packing transform codecs #19306 — T64 and GORILLA packing transform codecs
  7. Add V7 raw forward index format for codec pipelines #19307 — V7 raw forward index format
  8. Support codecSpec on segment reload and enable codecSpec end to end #19308 — Reload/rewrite support, gate removal (enabling PR)
  9. Add codec pipeline integration tests and design doc #19309 — Integration tests and design doc ← this PR

Each PR is based on its predecessor's branch; GitHub retargets the next PR as each merges.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/06-reload-enable branch from 9737c0c to aafe8c6 Compare August 19, 2026 09:51
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 88e1d40 to 026ac0b Compare August 19, 2026 09:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant