Claude/async arax pathfinder heartbeat 5ahuhf - #137
Open
maximusunc wants to merge 3 commits into
Open
Conversation
The pathfinding search ran via asyncio.to_thread, so its GIL-bound work
competed with the event loop -- and TASK_LIMIT was 100, meaning up to 100
concurrent searches. The heartbeat is a coroutine on that same loop pinging
every 5s against a 15s freshness window, so under load a live worker read as
dead: reclaim_orphaned could XCLAIM a still-running task to a peer and
duplicate it, and a stall past 60s tripped LoopWatchdog into os._exit(1).
Restructure to match aragorn_score / arax_rank:
- arax_pathfinder_task is a sync process-pool entrypoint taking only the
query/response ids. It loads, searches, rehydrates, and saves entirely in a
spawned child, so neither the search nor the message encode touches the
parent's loop or heap. rehydrate becomes sync (httpx.Client) since it now
runs in the child.
- process_task delegates to the shared run_task_lifecycle via a pool closure,
replacing the hand-rolled copy that used tracer.start_span instead of
start_as_current_span (so httpx spans never nested under the task span) and
never recorded exceptions on the span.
- poll_for_tasks builds a ProcessPoolManager sized by resolve_pool_workers and
passes max_workers to get_tasks; TASK_LIMIT drops 100 -> 10.
Failures now reach the lifecycle instead of being swallowed. The three
query-graph validation failures returned `message, 500`, a value the caller
discarded, so the task was wrapped up as a success with no message ever
written to response_id -- leaving the next worker to KeyError on it. Search
failures were saved as a non-TRAPI {"status": "error"} blob that then flowed
down the workflow. Both now raise and route to finish_query with ERROR.
Also:
- The blocked-concept list is fetched once at startup via
ensure_http_files_dataset (temp file + atomic rename) rather than with a
per-task requests.get racing on the same relative path; children parse it
once and memoize, as they do the BiolinkHelper descendants.
- Add "arax.pathfinder" to PER_STREAM_MIN_IDLE_SEC. It was absent, so its
reclaim floor was the 30s fast default; 360s sits above pool_task_timeout_sec.
- Dockerfile gets PYTHONFAULTHANDLER=1 so a child hung during import still
dumps a stack when ProcessPoolManager SIGABRTs it.
- Fix rehydrate's no-op `.replace("query", "rehydrate")` on an already-correct
URL, and its references to a possibly-unbound `res` in the error handlers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H51TTxq1F6o6yZSP1C86MZ
Codecov Report❌ Patch coverage is
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.