Skip to content

[core][mosaic] Preserve projected raw reads - #9086

Open
jianguotian wants to merge 8 commits into
apache:masterfrom
jianguotian:fix/raw-file-split-read-projection
Open

[core][mosaic] Preserve projected raw reads#9086
jianguotian wants to merge 8 commits into
apache:masterfrom
jianguotian:fix/raw-file-split-read-projection

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

RawFileSplitRead can be reused with different projected RowType values, but its cached FormatReaderMapping entries are keyed only by file schema and format. Reconfiguring the reader can therefore reuse a stale projection. In addition, lazily-created file readers need to retain the output type selected when each reader is created.

This PR:

  • invalidates cached format mappings only when withReadType receives a different RowType;
  • captures the current output type when a reader is created;
  • passes that same type through the mapping builder, lazy file-reader suppliers, and DataFileRecordReader;
  • preserves existing Manifest partition-column restoration when partition fields are requested.

The core change is format-neutral. It does not add Direct Bundle or writer behavior.

Tests

Added coverage for:

  • changed projections not sharing cached format mappings;
  • equal RowType instances reusing the existing mapping;
  • physical Mosaic files containing only business columns;
  • full, business-only, and partition-only projections;
  • row tracking and existing-versus-future lazy reader stability.

Focused validation:

mvn -B -ntp \
  -pl paimon-common,paimon-core,paimon-mosaic -am \
  -Dtest=ColumnarRowIteratorTest,RawFileSplitReadTest,PartitionColumnsOutsidePhysicalMosaicTest \
  -Dsurefire.failIfNoSpecifiedTests=false test
  • ColumnarRowIteratorTest: 1 passed
  • RawFileSplitReadTest: 2 passed
  • PartitionColumnsOutsidePhysicalMosaicTest: 2 passed
  • Checkstyle, Spotless, Maven Enforcer, and git diff --check: passed

@jianguotian jianguotian closed this Aug 7, 2026
@jianguotian jianguotian reopened this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant