[fix](pipeline) Make fragment cancellation idempotent - #67236
Open
HappenLee wants to merge 1 commit into
Open
Conversation
A query timeout can wake every pending pipeline task. Each task then closes with the same timeout status and calls PipelineFragmentContext::cancel() before the fragment closed-task count is incremented. This repeats the full task dump and instance logs for every task, producing quadratic log amplification. Add an atomic fragment-level gate after notify_close() so cancellation side effects run only once while preserving recursive CTE close handling. Add a unit test that verifies repeated timeout cancellation emits each fragment-level diagnostic once. Issue Number: None Test: GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh -j 48 --run --filter=PipelineTaskTest.TEST_FRAGMENT_CANCEL_IS_IDEMPOTENT
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
yiguolei
reviewed
Aug 27, 2026
| std::atomic<int> _total_tasks = 0; | ||
| // Multiple tasks can observe the same query cancellation and call cancel concurrently. | ||
| // Run fragment-level cancellation side effects only once. | ||
| std::atomic_bool _cancelled = false; |
Contributor
There was a problem hiding this comment.
不要增加这种标记位
一个fragment的status 是由atomic status 这种来标记的。
单独的标记位，会引入问题，比如is cancelld，然后，后面可能还会增加一个string 来存储reason 这种东东
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
When a query is cancelled, every pending pipeline task can be awakened and closed with the same error.
TaskScheduler::close_task()callsPipelineFragmentContext::cancel()before incrementing the fragment closed-task count, so each task can repeat the fragment-wide timeout dump, instance logs, query cancellation, stream cancellation, and dependency unblocking. For a fragment with many tasks, the full task dump is therefore amplified roughly quadratically.This change adds an atomic one-shot gate after
notify_close(). The first active cancellation retains all existing behavior; later calls return before fragment-level logging and side effects. Keepingnotify_close()before the gate preserves recursive CTE close handling and the already-closed removal path.Release note
Prevent repeated BE pipeline timeout dumps when a query is cancelled.
Check List (For Author)
Test
GLIBC_COMPATIBILITY=OFF ./run-be-ut.sh -j 48 --run --filter=PipelineTaskTest.TEST_FRAGMENT_CANCEL_IS_IDEMPOTENTBehavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)