fix(pyamber): wait out a pause before completing the worker

[aglinxinyuan]

What changes were proposed in this PR?

Root cause. MainLoop._process_end_channel ends with self.complete(), which transits the worker to COMPLETED. Nothing between that call and the last control check above it drains control, but the coordinator pauses on its own schedule — so a PauseWorker command can land in that window and leave the worker PAUSED. PAUSED -> COMPLETED is forbidden by the transition graph, so transit_to raised InvalidTransitionException, which killed the main loop thread:

main_loop.py:674  _process_end_channel -> self.complete()
main_loop.py:305  complete()           -> state_manager.transit_to(COMPLETED)
state_manager.py:80                    -> InvalidTransitionException:
                                          Cannot transit from PAUSED to COMPLETED

Before:  pause lands before complete() -> PAUSED -> COMPLETED -> InvalidTransition, thread dies
After:   pause lands before complete() -> wait for Resume -> RUNNING -> COMPLETED

The worker now waits the pause out before completing, which is what the Scala runtime already does: DPThread's input selection only picks control channels while pauseManager.isPaused, so a paused Scala worker never advances to completion.

The wait is guarded on pause_manager.is_paused() rather than draining control unconditionally. _check_and_process_control blocks while the data lane is disabled, and backpressure disables that lane too (DisableType.DISABLE_BY_BACKPRESSURE) — an unguarded drain here would park a worker that is merely backpressured. Guarded, the happy path is untouched: not paused, loop body never runs.

Deliberately not changed: the transition graph. Adding PAUSED -> COMPLETED would be the smaller diff, but WORKER_STATE_TRANSITIONS mirrors Scala's WorkerStateManager (PAUSED -> Set(RUNNING)) by contract, and the two must agree.

Note for #5913: its stated cause is now stale. It attributes the flake to a 100-row smallCsvScanOpDesc finishing before the pause lands and prescribes a larger source; #5915 already did that, and the spec has used slowRegionSourceOpDesc(numTuple = 30, delaySeconds = 0.25) since. The surviving race is this one, in pyamber, and is independent of how long the source runs.

Any related issues, documentation, discussions?

Closes #5913

How was this PR tested?

New regression test, written first and confirmed to fail red against unmodified source with the exact production traceback (Cannot transit from PAUSED to COMPLETED at _process_end_channel -> complete()).

test_pause_landing_before_completion_defers_it_until_resume makes the race deterministic by pausing inside all_ports_completed() — the last call before complete(), with no control check in between, i.e. exactly the production window. It pins both directions:

• negative — while paused: the main loop thread stays alive, the worker sits in PAUSED, and no WorkerExecutionCompleted is announced;
• positive — after Resume: the worker reaches COMPLETED and announces it.

From amber/:

• python -m pytest src/test/python/core/runnables/test_main_loop.py — 35 passed (34 pre-existing + the new one).
• The new test run 5x consecutively — passed every time (it is a race regression, so repetition matters).
• ruff check src/main/python src/test/python && ruff format --check src/main/python src/test/python — clean.

Pre-existing local failures in test_iceberg_document.py, test_tuple.py::test_hash and test_expression_evaluator.py were confirmed identical with this change stashed, so they are unrelated to it.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch	Analysis
✅ release/v1.2	Change detected on this branch — label added; this fix is queued to backport here. Requested review from @xuang7.

Auto-label run.

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.99%. Comparing base (310ab88) to head (d1e2829).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #7624   +/-   ##
=========================================
  Coverage     88.99%   88.99%           
  Complexity     4348     4348           
=========================================
  Files          1178     1178           
  Lines         46836    46838    +2     
  Branches       5226     5226           
=========================================
+ Hits          41681    41683    +2     
  Misses         3416     3416           
  Partials       1739     1739           

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)		Carriedforward from 310ab88
agent-service	98.62% <ø> (ø)		Carriedforward from 310ab88
amber	84.95% <ø> (ø)		Carriedforward from 310ab88
computing-unit-managing-service	60.38% <ø> (ø)		Carriedforward from 310ab88
config-service	77.31% <ø> (ø)		Carriedforward from 310ab88
file-service	68.90% <ø> (ø)		Carriedforward from 310ab88
frontend	90.57% <ø> (ø)		Carriedforward from 310ab88
notebook-migration-service	78.89% <ø> (ø)		Carriedforward from 310ab88
pyamber	97.56% <100.00%> (+<0.01%)	⬆️
workflow-compiling-service	57.89% <ø> (ø)		Carriedforward from 310ab88

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 310ab88 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	436	0.266	22,162/28,905/28,905 us	🔴 +8.8% / 🔴 +77.2%
🔴	bs=100 sw=10 sl=64	891	0.544	109,553/152,198/152,198 us	🔴 +12.8% / 🔴 +40.7%
🔴	bs=1000 sw=10 sl=64	1,098	0.67	911,194/1,003,346/1,003,346 us	🔴 +5.2% / 🟢 -8.5%

Baseline details

Latest main 310ab88 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	436 tuples/sec	453 tuples/sec	775.33 tuples/sec	-3.8%	-43.8%
bs=10 sw=10 sl=64	MB/s	0.266 MB/s	0.276 MB/s	0.473 MB/s	-3.6%	-43.8%
bs=10 sw=10 sl=64	p50	22,162 us	20,377 us	12,743 us	+8.8%	+73.9%
bs=10 sw=10 sl=64	p95	28,905 us	31,124 us	16,310 us	-7.1%	+77.2%
bs=10 sw=10 sl=64	p99	28,905 us	31,124 us	18,926 us	-7.1%	+52.7%
bs=100 sw=10 sl=64	throughput	891 tuples/sec	921 tuples/sec	1,001 tuples/sec	-3.3%	-11.0%
bs=100 sw=10 sl=64	MB/s	0.544 MB/s	0.562 MB/s	0.611 MB/s	-3.2%	-10.9%
bs=100 sw=10 sl=64	p50	109,553 us	107,259 us	101,399 us	+2.1%	+8.0%
bs=100 sw=10 sl=64	p95	152,198 us	134,890 us	108,206 us	+12.8%	+40.7%
bs=100 sw=10 sl=64	p99	152,198 us	134,890 us	118,195 us	+12.8%	+28.8%
bs=1000 sw=10 sl=64	throughput	1,098 tuples/sec	1,108 tuples/sec	1,026 tuples/sec	-0.9%	+7.0%
bs=1000 sw=10 sl=64	MB/s	0.67 MB/s	0.677 MB/s	0.626 MB/s	-1.0%	+7.0%
bs=1000 sw=10 sl=64	p50	911,194 us	899,001 us	996,304 us	+1.4%	-8.5%
bs=1000 sw=10 sl=64	p95	1,003,346 us	953,842 us	1,042,531 us	+5.2%	-3.8%
bs=1000 sw=10 sl=64	p99	1,003,346 us	953,842 us	1,074,934 us	+5.2%	-6.7%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,458.34,200,128000,436,0.266,22162.09,28905.47,28905.47
1,100,10,64,20,2245.35,2000,1280000,891,0.544,109552.70,152198.16,152198.16
2,1000,10,64,20,18217.01,20000,12800000,1098,0.670,911194.07,1003345.94,1003345.94