test: add regression test for Actor.exit() deadlock from event listeners - #1061
Open
vdusek wants to merge 5 commits into
Open
test: add regression test for Actor.exit() deadlock from event listeners#1061vdusek wants to merge 5 commits into
vdusek wants to merge 5 commits into
Conversation
vdusek
added
adhoc
github-actions
Bot
added
the
tested
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1061 +/- ##
===========================================
+ Coverage 56.63% 92.17% +35.54%
===========================================
Files 51 51
Lines 3247 3247
===========================================
+ Hits 1839 2993 +1154
+ Misses 1408 254 -1154
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
vdusek
marked this pull request as ready for review
July 21, 2026 11:06
Pijukatel
reviewed
Jul 21, 2026
Contributor
Author
|
maybe we can close this in favor of apify/crawlee-python#2088 , let's see |
vdusek
added a commit
to apify/crawlee-python
that referenced
this pull request
Aug 5, 2026
…rom within a listener (#2088) Calling `EventManager.wait_for_all_listeners_to_complete()` - or closing the manager via `__aexit__` - from *within* an event listener deadlocked. The listener runs in a task that is itself registered in `_listener_tasks`, so the wait ends up awaiting the very task that is awaiting it. Under a close timeout this cycle degrades further into a `RecursionError`. Changes in `EventManager`: - Listener tasks currently blocked in `wait_for_all_listeners_to_complete()` are tracked in `_waiting_listener_tasks` and excluded from the wait, so listener waiters never await themselves or each other. A caller that is not a listener is outside the cycle and still awaits every listener, waiting ones included. - The wait no longer wraps the gather in an inner task. The one-tick defer that task provided is now explicit: `emit` only schedules the listener wrappers, and each registers its listener task once it starts running, so the wait yields before snapshotting `_listener_tasks`. This also drops the `Event listener raised an exception.` log line, which duplicated the ERROR the listener wrapper already logs with the traceback. - The listener wrapper's `finally` uses `set.discard()` instead of `set.remove()`, since `__aexit__` may have already cleared the task set while the listener was mid-flight (avoids a spurious `KeyError`). Regression tests cover waiting from within a listener, several listeners waiting at once, closing the manager from within a listener, and waiting from outside while a listener is itself waiting. This unblocks apify/apify-sdk-python#1061, where `Actor.exit()` is called from inside an event listener (e.g. an `ABORTING` handler) - with this fix the SDK can drop its `_detach_current_listener_task` workaround. *✍️ Drafted by Claude Code*
…ee lock crawlee's EventManager now handles this deadlock upstream (apify/crawlee-python#2088), so the SDK-side workaround is redundant. The lockfile is bumped to crawlee 1.9.1b4, which contains the fix; the regression test passes without any SDK-side code changes. The declared crawlee constraint in pyproject.toml stays >=1.8.0,<2.0.0 until crawlee ships a stable release with the fix.
vdusek
changed the title
vdusek
changed the title
vdusek
changed the title
vdusek
changed the title
vdusek
marked this pull request as draft
August 5, 2026 11:58
Contributor
Author
|
Merge it with Crawlee 1.9.1 in the lockfile |
vdusek
marked this pull request as ready for review
August 6, 2026 15:47
# Conflicts: # uv.lock
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Actor.exit()/Actor.fail()called from within an event listener used to deadlock into aRecursionError, since the cleanup path waited on the very listener task that called it.Fixed upstream in crawlee's
EventManager: apify/crawlee-python#2088 (merged). On Python 3.11 it still deadlocks, becauseasyncio.wait_forthere wraps the awaited coroutine in a separate task, which defeats crawlee's self-wait detection. This is a minor edge case, so we're not adding an SDK-side workaround just for Python 3.11.This PR:
Actor.exit()called from anABORTINGlistener, skipped on Python 3.11.uv.lock(not the declaredpyproject.tomlconstraint) tocrawlee==1.9.1b4, so CI exercises the fix.Follow-up: bump the declared
crawleeconstraint inpyproject.tomlonce a stable release with the fix ships.✍️ Drafted by Claude Code