Skip to content

[FLINK-40334][table] Insert-only input should stay unmaterialized under FORCE - #28930

Open
gustavodemorais wants to merge 1 commit into
apache:masterfrom
confluentinc:FLINK-40334
Open

[FLINK-40334][table] Insert-only input should stay unmaterialized under FORCE#28930
gustavodemorais wants to merge 1 commit into
apache:masterfrom
confluentinc:FLINK-40334

Conversation

@gustavodemorais

Copy link
Copy Markdown
Contributor

What is the purpose of the change

FLINK-38928 dropped the guard that skipped the sink upsert materializer for insert-only input (here), because DO ERROR and DO NOTHING need the operator to compare inserts that share a primary key. Without it, an insert-only query into a sink with a primary key gets a SinkUpsertMaterializer and a keyed shuffle whenever upsert materialization is forced. This restores the guard so that only DO ERROR and DO NOTHING keep materializing insert-only input.

Short version: this makes so we generate the same job for the same compiled plan again after the regression 3ecbbd1523f#diff-a3e03102c64370ff75d10affae859b211715603dc4dc81bc6c25bf83f1161adcL191

Brief change log

  • Restore the insert-only guard around the sink materializer and the keyed shuffle in CommonExecSink.
  • Pass the conflict strategy into createSinkTransformation and add detectsDuplicateKeys.
  • Replace isErrorOrNothingConflictStrategy in StreamExecSink with detectsDuplicateKeys.
  • Note: the decision stays at translation time because plans compiled before the clause existed persist requireUpsertMaterialize = true, so suppressing the operator in the planner would make it appear on restore.

Verifying this change

  • TableSinkTest
  • SinkSemanticTests
  • TableSinkRestoreTest

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): yes - removes an operator from insert-only pipelines under forced upsert materialization
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes - a job re-planned after this change drops the materializer and its keyed state, so the topology shifts on restore
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

2.1.220 (Claude Code) with Opus 5

@flinkbot

flinkbot commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

…er FORCE

FLINK-38928 dropped the guard that skipped the sink upsert materializer for
insert-only input, because DO ERROR and DO NOTHING need the operator to compare
inserts that share a primary key. Without it, an insert-only query into a sink
with a primary key gets a SinkUpsertMaterializer and a keyed shuffle whenever
upsert materialization is forced, even though there are no changes to reconcile.

Restore the guard and pass the conflict strategy into the sink transformation so
that only DO ERROR and DO NOTHING keep materializing insert-only input.

The decision stays at translation time on purpose. Plans compiled before the
clause existed persist requireUpsertMaterialize = true for this case, so
suppressing the operator in the planner instead would make it appear on restore.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants