Add codec pipeline integration tests and design doc - #19309
Open
xiangfu0 wants to merge 1 commit into
Open
Conversation
This was referenced Aug 19, 2026
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/06-reload-enable
branch
from
August 19, 2026 09:51
9737c0c to
aafe8c6
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 19, 2026 09:51
88e1d40 to
026ac0b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #19308 (
xiangfu0/codex/codec-stack/06-reload-enable). Part of the split of #18229.Summary
CodecPipelineIntegrationTest(pinot-integration-tests,custom/suite): end-to-endcoverage of
indexes.forward.codecSpecon an offline table. Every supported codec spec gets itsown INT and LONG raw column, all populated with identical values, so every codec must read back
the same aggregates, filter counts, and point lookups on both query engines (SSE + MSE):
LZ4,ZSTD(3),SNAPPY,GZIP(legacy-compatible raw writer path)DELTA,LZ4,DELTA,ZSTD(3),DELTADELTA,LZ4(V7 writer path)T64,GORILLA,T64,LZ4,GORILLA,ZSTD(3)(V7 writer path)(
targetDocsPerChunk=256against 500-row input files), a cross-codec same-row consistencyquery, coexistence with a dictionary-encoded STRING column, and a RAW STRING column using a
compression-only
codecSpec(ZSTD(3)) alongside V7 transform columns in the same segment.docs/design/codec-pipeline-v7.md: design doc for the codec pipeline framework — DSLgrammar (matching
CodecSpecParser), architecture, V7 on-disk format and reader-side corruptiondefenses, built-in codec catalog, configuration and validation, user manual (enable / migrate /
roll back / mixed-version safety / common errors), threading, performance, backward
compatibility, test-coverage map, and future work. Placement under
docs/design/is noted assubject to reviewer preference.
All spec strings use the wrapper-less comma-list DSL introduced by this series (
DELTA,T64,LZ4,ZSTD(3)). The doc mentions the removedCODEC(...)wrapper only to document that the parserrejects it with a precise error.
Exclusions relative to the original #18229
CompressionCodecMigrator(and its tests) were dropped from the series; the design doc refersto future migration tooling instead and keeps the manual
compressionCodec→codecSpecmigration table.
indexes.forward.codecSpec.Safety
Test-and-docs-only slice: no production code changes. The integration test only exercises code
paths introduced by the earlier slices of this series (03–06). No wire formats, configs, or public
APIs are touched.
Verification
Stack (split of #18229)
Each PR is based on its predecessor's branch; GitHub retargets the next PR as each merges.