Skip to content

[flink] Fix limit pushdown for lake union read#3433

Open
QuakeWang wants to merge 2 commits into
apache:mainfrom
QuakeWang:fix-datalake-limit
Open

[flink] Fix limit pushdown for lake union read#3433
QuakeWang wants to merge 2 commits into
apache:mainfrom
QuakeWang:fix-datalake-limit

Conversation

@QuakeWang
Copy link
Copy Markdown
Contributor

Purpose

LIMIT pushdown previously always used the Fluss-only limit scan path. For datalake-enabled FULL scans, this skipped the lake source, so union reads could miss records that had already been tiered to the lake.

Brief change log

  • Avoid the Fluss-only limit scan path when a FULL scan needs to read from the lake.
  • Let the normal Flink source path perform lake + Fluss union reads in that case.
  • Add a Paimon IT covering LIMIT reads from an expired Fluss partition whose data still exists in Paimon.

Tests

  • ./mvnw -pl fluss-flink/fluss-flink-common,fluss-lake/fluss-lake-paimon -am -DskipTests -DfailIfNoTests=false test-compile

API and Format

Documentation

QuakeWang added 2 commits June 5, 2026 11:00
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant