[agentserver] Gate resilient TaskManager on the explicit enablement flag - #48591
Open
Nathandrake229 wants to merge 7 commits into
Open
[agentserver] Gate resilient TaskManager on the explicit enablement flag#48591Nathandrake229 wants to merge 7 commits into
Nathandrake229 wants to merge 7 commits into
Conversation
Make the durable-response/task subsystem strictly opt-in via `set_resilient_tasks_enabled` (auto-set by `resilient_background` / `steerable_conversations`). Previously every responses host wrapped all `store=true` responses in internal resilient tasks and therefore forced the boot-time recovery scan, even for a plain host that opted into nothing. Core (_base.py): - Construct the TaskManager (and run startup recovery) ONLY when the switch is enabled. When off, no manager is installed; `get_task_manager()` raises `TaskManagerNotInitialized` and callers degrade to non-durable in-process execution. A plain host pays nothing: no manager, no task-store call. Responses: - ResilientResponseOrchestrator auto-enables the switch when `resilient_background` or `steerable_conversations` is set, so resilient deployments keep full Row 1/2/3 recovery. - _start_resilient_background swallows `TaskManagerNotInitialized` at the single outer catch and runs the handler in-process (non-durable). Removed the hosted fail-loud branch and the now-unused `_is_hosted_environment` helper. Tests/samples: - Updated the opt-in gate tests for gated construction. - Rewrote the resilient-start-failure contract test: no-manager now swallows and runs in-process (real task-store start failures with a manager present still fail loud as platform errors). - Conformance `_test_handler` enables the switch so Row 2 (resilient_background=False) still validates mark-failed-on-crash. - Invocations resilient samples (research, multiturn, langgraph) enable the switch explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Emit a one-time WARNING at responses host startup when the durable-response subsystem is off (store=true responses are non-durable across an ungraceful crash), and an INFO when it is enabled, so operators are not surprised by the plain non-durable default. Reflects the final resolved state (after the resilient orchestrator's resilient_background/steerable auto-enable). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Nathandrake229
marked this pull request as ready for review
August 14, 2026 08:36
Nathandrake229
requested review from
Ravi Pidaparthi (RaviPidaparthi),
Shiva S (Shivakishore14),
Ankit Sinha (ankitbko) and
Pranav Pandit (vangarp)
as code owners
August 14, 2026 08:36
|
Azure Pipelines: Successfully started running 1 pipeline(s). 9 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Makes resilient task management strictly opt-in while preserving in-process execution for plain response hosts.
Changes:
- Gates
TaskManagerconstruction and recovery behind the enablement switch. - Auto-enables resilience for configured response hosts and updates fallback behavior.
- Updates resilient samples and tests for explicit enablement.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
.../core/_base.py |
Gates task-manager lifecycle. |
.../tests/tasks/test_task_manager_optin.py |
Updates gate tests. |
.../hosting/_routing.py |
Logs resilience state. |
.../hosting/_resilient_orchestrator.py |
Auto-enables configured resilience. |
.../hosting/_orchestrator.py |
Adds non-durable fallback. |
.../test_resilient_start_failure.py |
Revises fallback contract tests. |
.../resilience_contract/_test_handler.py |
Enables durable conformance testing. |
.../resilient_research/app.py |
Explicitly enables recovery. |
.../resilient_multiturn/app.py |
Explicitly enables recovery. |
.../resilient_langgraph/app.py |
Explicitly enables recovery. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+327
to
+329
| # recoverable. Declaring a durable task (``@task`` / | ||
| # ``@multi_turn_task``) does NOT implicitly turn the subsystem on; | ||
| # an app that wants durable tasks / recovery must set the switch. |
Comment on lines
+3990
to
+3994
| except TaskManagerNotInitialized: | ||
| # No resilient-task subsystem is installed in this process — the | ||
| # host did not enable resilient tasks | ||
| # (``set_resilient_tasks_enabled``), so recovery/durability is | ||
| # opt-out here. SWALLOW the signal and run the handler in-process: |
Comment on lines
142
to
143
| def test_no_manager_background_runs_in_process(self) -> None: | ||
| client = _build_client() |
Narrow the responses auto-enable of the resilient-tasks switch to `resilient_background` only; `steerable_conversations` no longer implicitly enables the durable subsystem. Recovery is tied to resilient_background alone; a steerable host that wants durability sets resilient_background=True (or the switch explicitly). The two-switch UX is a known rough edge to smooth over post-Public-Preview. Steering conformance handler now enables the switch explicitly, since steering (multi-turn input queuing) needs the TaskManager regardless of resilient_background. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.
Suppressed comments (2)
sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/_base.py:331
- This changes the public switch from controlling only recovery to controlling whether
TaskManagerexists, but the exported API documentation still promises the opposite._enablement.py:4-17,35-40,55-59,docs/tasks-guide.md:118-152, and the current CHANGELOG entry at lines 42-46 all say declared tasks work without this switch and that the manager is always constructed. Users following those docs will now getTaskManagerNotInitialized; update the public docstrings, task guide/examples, and release notes together with this behavior change.
if _resilient_tasks_enabled():
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_resilient_orchestrator.py:449
- This contradicts the PR's stated contract that either
resilient_backgroundorsteerable_conversationsauto-enables the subsystem. As written, a host configured only withsteerable_conversations=Trueleaves the switch off, while the new routing warning also tells operators that steering enables durability. Include the steering option in this gate so the implementation and advertised behavior agree.
if options.resilient_background:
…cations samples - Rewrite set_resilient_tasks_enabled/_enablement docs to describe the switch as the opt-in gate for TaskManager construction (not just the recovery scan), and that a declared @task no longer implicitly enables it. - Update core README + tasks-guide to show the required opt-in, and core/responses CHANGELOGs with the breaking-change note. - Fix _start_resilient_background docstring: TaskManagerNotInitialized is now swallowed (in-process fallback) regardless of hosting; only real task-start failures raise/tag platform errors. - Rewrite the swallow contract test to enter the ASGI lifespan with the switch off, assert no TaskManager is installed, and poll GET to completion. - Revert the 3 invocations resilient sample edits (unrelated api.md whitespace consistency drift); sample opt-in to be handled as a follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_routing.py:523
- This warning currently tells operators that
steerable_conversationsenables durability, but the orchestrator only flips the switch forresilient_background. As written, following this guidance still leaves the manager disabled and responses non-durable. Align the enablement condition and this operator-facing message.
"Enable durability via resilient_background / steerable_conversations, or "
"set_resilient_tasks_enabled(True)."
sdk/agentserver/azure-ai-agentserver-responses/tests/contract/test_resilient_start_failure.py:34
- The “manager present” start-failure tests still build a default host without ever enabling this switch. After the new gate, no manager is installed; the tests only pass because
_start_failsmocksstart_resilient()above the manager lookup and raises unconditionally. Add a switch-on fixture to those three tests (and verifyget_task_manager()succeeds) so they actually cover the promised manager-present/task-store-failure path.
from azure.ai.agentserver.core.tasks import (
TaskManagerNotInitialized,
resilient_tasks_enabled,
set_resilient_tasks_enabled,
)
sdk/agentserver/azure-ai-agentserver-responses/tests/e2e/resilience_contract/_steering_handler.py:85
- Explicitly enabling the switch here masks the stated requirement that
steerable_conversations=Trueauto-enables the task subsystem. The steering conformance test will continue passing even when that production wiring is absent, as it is in this diff. Let the option perform the enablement and assert the manager is installed during lifespan instead.
# Steering (mid-turn input queuing) is implemented by the multi-turn task
# primitive and needs the TaskManager. Since only ``resilient_background`` now
# auto-enables the subsystem (steerable does not), enable it explicitly so the
# steering conformance is valid even when CONFORMANCE_RESILIENT_BACKGROUND=false.
set_resilient_tasks_enabled(True)
| # switch) constructs no ``TaskManager``: ``store=true`` work degrades to | ||
| # non-durable in-process execution (``_start_resilient_background`` | ||
| # swallows ``TaskManagerNotInitialized``). | ||
| if options.resilient_background: |
This comment has been minimized.
This comment has been minimized.
…Initialized message - The switch-off contract tests asserted get_task_manager() raises, but the process-global TaskManager singleton can leak from a prior test in a shared pytest process (CI), causing DID-NOT-RAISE. The _switch_off fixture now also snapshots/resets/restores the manager singleton (set_task_manager(None)), so the no-manager path is exercised deterministically. - Make the TaskManagerNotInitialized message actionable: it now names set_resilient_tasks_enabled(True) and the responses resilient_background=True option instead of a generic 'ensure resilient tasks are enabled'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (4)
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_resilient_orchestrator.py:449
- The PR contract says either
resilient_backgroundorsteerable_conversationsauto-enables the subsystem, and the new operator warning makes the same promise. This condition excludes steerable-only hosts, so their lifespan constructs noTaskManagerand mid-turn input falls back to independent in-process work instead of the multi-turn task queue. Includesteerable_conversationsin this gate.
if options.resilient_background:
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_routing.py:511
- This mode is sampled during host construction, but
set_resilient_tasks_enabledis documented to take effect any time before lifespan startup. For example, constructing the host and then enabling the switch logsDISABLEDeven though_base.pylater constructs the manager; clearing it after construction causes the inverse. Emit this announcement from lifespan startup after the gate is resolved so operators see the mode that actually runs.
if resilient_tasks_enabled():
sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/tasks/_manager.py:304
get_task_manager()only knows that_manageris absent; it cannot conclude that the switch is disabled. This message is false when resilience was enabled but the lifespan has not started yet (or initialization failed), and can send users toward re-enabling an already-enabled switch. Keep the distinction between enablement and initialization in the guidance.
"TaskManager not initialized: the resilient task subsystem is not enabled. "
"Durable tasks and crash recovery are opt-in — call "
"set_resilient_tasks_enabled(True) before host startup (e.g. at import time), "
"or, for the responses protocol, construct the host with "
"ResponsesServerOptions(resilient_background=True)." # pylint: disable=implicit-str-concat
sdk/agentserver/azure-ai-agentserver-responses/tests/e2e/resilience_contract/_steering_handler.py:85
- This explicit switch call masks the advertised auto-enablement contract for
steerable_conversations: the conformance server passes even when the option fails to enable a manager, which is exactly the current implementation defect. Exercise the steerable-only configuration without manually setting the switch (and assert that a manager is installed) so this behavior cannot regress.
# Steering (mid-turn input queuing) is implemented by the multi-turn task
# primitive and needs the TaskManager. Since only ``resilient_background`` now
# auto-enables the subsystem (steerable does not), enable it explicitly so the
# steering conformance is valid even when CONFORMANCE_RESILIENT_BACKGROUND=false.
set_resilient_tasks_enabled(True)
This comment has been minimized.
This comment has been minimized.
…ake) The switch-off contract tests asserted get_task_manager() raises after entering the lifespan, but that depended on process-global manager/flag state (mutated by 1400+ other tests) and on which core build performs manager construction — making it fail deterministically in CI while passing locally. Force the no-manager condition explicitly via set_task_manager(None) inside the running lifespan so the responses SWALLOW path is exercised robustly. Core's gating of manager construction is covered separately by the core opt-in tests. Verified in a Linux (WSL) full-suite run: both tests pass among 1436 others. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_resilient_orchestrator.py:442
steerable_conversations=Trueis omitted from the gate, even though steering selects the multi-turn primitive and its mid-turn queuing occurs insideMultiTurnTask.start(). Withresilient_background=False, no manager is constructed, the call falls back before that primitive can queue input, and steering no longer works as advertised. This also contradicts the PR description and the startup message in_routing.py; enable the subsystem when either option is set.
# NOTE: only ``resilient_background`` auto-enables the subsystem —
# ``steerable_conversations`` intentionally does NOT. Recovery is tied to
# ``resilient_background`` alone; a steerable host that wants durability
# must set ``resilient_background=True`` (or call
# ``set_resilient_tasks_enabled(True)`` explicitly). The two-switch UX is
sdk/agentserver/azure-ai-agentserver-responses/azure/ai/agentserver/responses/hosting/_orchestrator.py:4010
TaskManagerNotInitializeddoes not necessarily mean deliberate opt-out:_base.pycatches manager import/construction failures while the enablement flag remains true. An explicitly resilient deployment can therefore reach this branch and silently run non-durably. Only fall back whenresilient_tasks_enabled()is false; if it is true, preserve the fail-loud platform-error behavior.
except TaskManagerNotInitialized:
# No resilient-task subsystem is installed in this process — the
# host did not enable resilient tasks
# (``set_resilient_tasks_enabled``), so recovery/durability is
# opt-out here. SWALLOW the signal and run the handler in-process:
Comment on lines
+4004
to
4010
| logger.info( | ||
| "Resilient task subsystem not enabled for response %s; running handler " | ||
| "in-process (non-durable). Enable via set_resilient_tasks_enabled(True) " | ||
| "for crash recovery.", | ||
| ctx.response_id, | ||
| ) | ||
| record.execution_task = asyncio.create_task(fallback_runner()) |
Contributor
[Pilot] PR Pipeline Failure AnalysisA CI pipeline failed on this pull request. Here is an automated analysis of what went wrong and how to get the build green. What failedThe python - pullrequest pipeline (build #6703685) failed in the Build Analyze stage on a pylint validation check for the
The Recommended next steps
Raw pipeline analysis (azsdk ci analyze)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make the durable-response / resilient-task subsystem strictly opt-in via the existing
set_resilient_tasks_enabledswitch (auto-set whenresilient_backgroundorsteerable_conversationsis configured).Previously, every responses host wrapped all
store=trueresponses in internal resilient tasks (responses_resilient_one_shot/responses_resilient_multi_turn) to provide the Row 2/3 "crashed →failed" guarantee. Because those internal primitives register durable tasks, every responses host — even a plain one that opted into nothing — triggered the boot-time recovery scan (a blocking task-storelist()+ credential-token acquisition). This PR decouples plain responses usage from the task subsystem entirely.Behavior
store=truepathresilient_background/steerable_conversations)in_progressA plain responses host now pays nothing: no manager, no task-store call, no boot scan. Its
store=trueresponses run as plain in-process work — the only thing given up is the crash-only mark-failed guarantee it never opted into.Changes
Core (
_base.py)TaskManager(and run startup recovery) only when the switch is enabled. When off, no manager is installed;get_task_manager()raisesTaskManagerNotInitializedand callers degrade to non-durable in-process execution.Responses
ResilientResponseOrchestratorauto-enables the switch whenresilient_backgroundorsteerable_conversationsis set (explicit resilience opt-in ⇒ subsystem on), so resilient deployments keep full Row 1/2/3._start_resilient_backgroundswallowsTaskManagerNotInitializedat the single outer catch and runs the handler in-process. Removed the hosted fail-loud branch and the now-unused_is_hosted_environmenthelper. Real task-store start failures (manager present) still fail loud as platform errors.Tests / samples
_test_handlerenables the switch so Row 2 (resilient_background=False) still validates mark-failed-on-crash.Validation
os.killpg) — validated in Linux CI, skipped on Windows dev.Relationship to #48585
#48585 gated only the blocking boot scan (manager always constructed, no swallowing needed) but regressed Row 1/2/3 Path C (skipping Layer 1 while one-shot recovery depends on it). This PR instead makes the switch the true single source of truth for the whole subsystem, preserving Row 1/2/3 for opted-in hosts while fully decoupling plain hosts.