Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@

### Breaking Changes

- The resilient task subsystem is now **strictly opt-in**. `AgentServerHost`
constructs the `TaskManager` only when resilient tasks are enabled via
`set_resilient_tasks_enabled(True)` (or a protocol option that maps to it,
e.g. the responses `resilient_background`). Previously the manager was always
constructed and a declared `@task` / `@multi_turn_task` implicitly enabled the
startup recovery scan. Now, declaring a task does **not** turn the subsystem
on: with the switch off, `get_task_manager()` raises `TaskManagerNotInitialized`
and `.run()` / `.start()` cannot run a task. Existing apps that rely on `@task`
must call `set_resilient_tasks_enabled(True)` (before host startup) to keep
durable tasks and crash recovery. Plain servers that use no tasks are
unaffected and continue to pay nothing.
- Removed `TaskMetadata`, `TaskContext.metadata`, and `TaskRun.metadata`.
Durable application state now belongs in an explicit `FoundryStateStore`
and no longer shares task lifecycle PATCHes or lease renewal. Typed task
Expand Down
11 changes: 10 additions & 1 deletion sdk/agentserver/azure-ai-agentserver-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,17 @@ python my_agent.py

The `@task` decorator builds crash-resilient agents that survive container restarts, OOM kills, and redeployments. Task state is persisted to a task store, enabling automatic recovery and multi-turn suspend/resume patterns.

The resilient task subsystem is **opt-in**: call `set_resilient_tasks_enabled(True)` (before host startup, typically at import time) so the framework constructs the `TaskManager` and runs crash recovery. Without it, `.run()` / `.start()` raise `TaskManagerNotInitialized`.

```python
from azure.ai.agentserver.core.tasks import task, TaskContext
from azure.ai.agentserver.core.tasks import (
set_resilient_tasks_enabled,
task,
TaskContext,
)

# Opt in to the durable task subsystem (required for @task to run).
set_resilient_tasks_enabled(True)

@task(name="process_document")
async def process_document(ctx: TaskContext[dict]) -> dict:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -311,38 +311,30 @@ async def _lifespan(_app: Starlette) -> AsyncGenerator[None, None]: # noqa: RUF

# --- Resilient task manager initialization ---
#
# The TaskManager is CONSTRUCTED unconditionally (whenever the
# resilient tasks module is importable). Construction is cheap and
# makes NO task-store calls — it only builds in-memory state (an
# idle provider client, empty routing tables, a lease-owner string).
# This keeps ``get_task_manager()`` working so a ``@task``-based app
# can run tasks without hitting ``TaskManagerNotInitialized``,
# while paying zero network cost until a task is actually used.
# The resilient ``TaskManager`` is CONSTRUCTED ONLY when resilient
# tasks were explicitly enabled via
# ``set_resilient_tasks_enabled(True)``. Recovery (and the durable
# task subsystem as a whole) is strictly opt-in: a host that does
# not set the switch pays nothing — no manager, no provider client,
# no task-store call, no recovery loop.
#
# The network-backed startup RECOVERY SCAN (a blocking hosted
# task-store ``list()`` plus ``DefaultAzureCredential`` token
# acquisition, which would otherwise gate server readiness) — and
# the periodic recovery loop it spawns — run when EITHER holds:
# (1) at least one durable task was declared (``@task`` /
# ``@multi_turn_task``, tracked in ``_REGISTERED_DESCRIPTORS``)
# — an app that uses tasks gets recovery automatically, OR
# (2) resilient tasks were explicitly enabled via
# ``set_resilient_tasks_enabled(True)`` — a force-enable that
# starts the recovery loop even before any task is declared,
# so a task declared later is picked up by the loop.
# A plain server that neither declares a task nor sets the switch
# (e.g. an invocations-only host) makes no task-store call at
# startup.
#
# NOTE (deferred): if the switch is OFF and no task is declared at
# startup, the recovery loop is not started, so a task declared
# LATER in that lifetime will run but its prior-crash orphans are
# not scanned until the next restart. Fully closing that requires a
# lazy manager-start on first late registration; tracked as future
# work (the manager cannot be made fully async, only lazily
# started).
# When the switch is OFF the manager is NOT installed, so
# ``get_task_manager()`` raises ``TaskManagerNotInitialized``.
# Callers that route through the manager (e.g. the responses
# protocol's ``store=true`` path) are expected to SWALLOW that
# signal and degrade to non-durable, in-process execution — the
# response still runs and persists, it simply is not crash-
# recoverable. Declaring a durable task (``@task`` /
# ``@multi_turn_task``) does NOT implicitly turn the subsystem on;
# an app that wants durable tasks / recovery must set the switch.
Comment thread
Nathandrake229 marked this conversation as resolved.
task_manager = None
try:
if _resilient_tasks_enabled():
# Resilient tasks were explicitly enabled. Construct the manager
# and run startup recovery. If EITHER fails, durability was
# explicitly requested, so fail the lifespan (fail-fast at boot)
# rather than start a server that would silently run store=true
# work non-durably. This guarantees that in a running deployment
# "enabled" always implies a live manager.
from .tasks._manager import ( # pylint: disable=import-outside-toplevel
TaskManager,
set_task_manager,
Expand All @@ -354,20 +346,31 @@ async def _lifespan(_app: Starlette) -> AsyncGenerator[None, None]: # noqa: RUF
shutdown_grace_seconds=_read_task_manager_shutdown_grace(),
)
set_task_manager(task_manager)

if _resilient_tasks_enabled() or _has_registered_tasks():
try:
await task_manager.startup()
logger.info("TaskManager initialized with startup recovery")
else:
logger.info(
"TaskManager initialized (recovery deferred; enabled=%s, tasks_declared=%s)",
_resilient_tasks_enabled(),
_has_registered_tasks(),
)
except ImportError:
pass # resilient module not available
except Exception: # pylint: disable=broad-exception-caught
logger.warning("Failed to initialize TaskManager", exc_info=True)
except BaseException:
# ``set_task_manager`` above already installed the manager
# globally, but startup failed and the lifespan will now
# abort before the shutdown block runs. Best-effort tear the
# partially-started manager down and clear the singleton so a
# failed/half-initialized manager is not left visible through
# ``get_task_manager()``; then re-raise to fail the lifespan.
try:
await task_manager.shutdown()
except Exception: # pylint: disable=broad-exception-caught
logger.warning("Error shutting down TaskManager after startup failure", exc_info=True)
set_task_manager(None)
task_manager = None
raise
logger.info("TaskManager initialized with startup recovery")
else:
logger.info(
"TaskManager NOT initialized (resilient tasks disabled; enable via "
"set_resilient_tasks_enabled(True)). Durable tasks and crash recovery "
"are inactive; store=true work degrades to non-durable execution. "
"tasks_declared=%s",
_has_registered_tasks(),
)

yield

Expand Down
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
# ---------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# ---------------------------------------------------------
"""Explicit force-enable switch for the resilient task recovery scan.
"""Explicit opt-in switch for the resilient task subsystem.

The resilient ``TaskManager`` is always constructed by ``AgentServerHost`` (a
cheap, in-memory object that makes no task-store calls), so ``get_task_manager``
and ``.run()`` / ``.start()`` work regardless of this switch. What this switch
affects is only the network-backed **startup recovery scan** (a hosted
task-store ``list()`` plus credential-token acquisition) and the periodic
recovery loop it spawns.
The resilient ``TaskManager`` is constructed by ``AgentServerHost`` **only when
this switch is on**. It is the single source of truth for the durable task
subsystem: when the switch is off, no ``TaskManager`` is installed, so
``get_task_manager()`` raises
:class:`~azure.ai.agentserver.core.tasks.TaskManagerNotInitialized` and
``.run()`` / ``.start()`` cannot run a task. When on, the manager is
constructed and the startup crash-recovery scan (plus the periodic recovery
loop) runs.

That recovery runs at startup when EITHER a durable task has been declared
(``@task`` / ``@multi_turn_task``) OR this switch is on. So an app that uses
tasks gets recovery automatically; this switch is a **force-enable** that
starts the recovery loop even before any task is declared (useful when tasks
are registered lazily after startup — the running loop then picks them up).
Recovery — and durable tasks as a whole — is therefore strictly opt-in. Merely
declaring a durable task (``@task`` / ``@multi_turn_task``) does **not** turn
the subsystem on; you must set this switch (directly, or via a protocol option
that maps to it, e.g. the responses ``resilient_background`` server option).

The switch is process-global and defaults to ``False``. It is intentionally
decoupled from any ``AgentServerHost`` instance so it can be flipped
independently at import time, e.g.::
The switch is process-global and defaults to ``False``. Set it before
``AgentServerHost`` lifespan startup (typically at import time)::

from azure.ai.agentserver.core.tasks import set_resilient_tasks_enabled

Expand All @@ -32,18 +32,22 @@

@experimental
def set_resilient_tasks_enabled(value: bool = True) -> None:
"""Force-enable (or clear) the resilient task recovery scan process-wide.
"""Opt in to (or clear) the resilient task subsystem process-wide.

Setting this to ``True`` starts the startup recovery scan + periodic
recovery loop even when no durable task is declared at startup. It does
NOT gate the ``TaskManager``'s existence: ``.run()`` / ``.start()`` work
whether or not this is set — this only controls automatic crash recovery.
This gates whether ``AgentServerHost`` constructs the ``TaskManager`` at
lifespan startup. Setting it to ``True`` constructs the manager, runs the
startup crash-recovery scan, and starts the periodic recovery loop. When it
is ``False`` no manager is installed: ``get_task_manager()`` raises
:class:`~azure.ai.agentserver.core.tasks.TaskManagerNotInitialized` and a
durable task cannot run (callers such as the responses ``store=true`` path
swallow that and degrade to non-durable in-process execution).

Must be called before ``AgentServerHost`` lifespan startup (typically at
import time) to take effect. Defaults to enabling when called with no
argument.

:param value: ``True`` to force-enable recovery, ``False`` to clear.
:param value: ``True`` to enable the resilient task subsystem, ``False`` to
clear.
:type value: bool
"""
global _RESILIENT_TASKS_ENABLED # pylint: disable=global-statement
Expand All @@ -52,11 +56,11 @@ def set_resilient_tasks_enabled(value: bool = True) -> None:

@experimental
def resilient_tasks_enabled() -> bool:
"""Return whether the recovery scan was explicitly force-enabled.
"""Return whether the resilient task subsystem is enabled.

Note this reflects only the switch — recovery also runs automatically when
a durable task is declared, so a ``False`` return does not mean recovery is
off, nor that tasks are unavailable.
This is the authoritative gate: a ``False`` return means no ``TaskManager``
is constructed and durable tasks / crash recovery are inactive (declaring a
task does NOT change this — the subsystem is opt-in).

:return: ``True`` if :func:`set_resilient_tasks_enabled` turned it on.
:rtype: bool
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -297,8 +297,11 @@ def get_task_manager() -> TaskManager:
)

raise TaskManagerNotInitialized(
"TaskManager not initialized. Ensure resilient tasks "
"are enabled on the AgentServerHost." # pylint: disable=implicit-str-concat
"TaskManager not initialized: the resilient task subsystem is not enabled. "
"Durable tasks and crash recovery are opt-in - call "
"set_resilient_tasks_enabled(True) before host startup (e.g. at import time), "
"or, for the responses protocol, construct the host with "
"ResponsesServerOptions(resilient_background=True)." # pylint: disable=implicit-str-concat
)
return _manager

Expand Down
34 changes: 17 additions & 17 deletions sdk/agentserver/azure-ai-agentserver-core/docs/tasks-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,31 +115,31 @@ What this primitive deliberately does **not** do:

### Enabling resilient tasks

The resilient `TaskManager`'s **startup recovery scan** — a network round-trip
to the hosted task store that reclaims tasks left in-flight by a crashed prior
instance — runs at startup when **either** of the following holds:

1. at least one durable task has been declared (`@task` / `@multi_turn_task`) —
an app that uses tasks gets recovery automatically, **or**
2. it was explicitly force-enabled via `set_resilient_tasks_enabled(True)`.

The force-enable is useful when tasks are registered *lazily* (declared after
startup): it starts the periodic recovery loop up front so a task declared
later is still recovered.
The resilient task subsystem is **opt-in**. The `TaskManager` is constructed —
and its **startup recovery scan** (a network round-trip to the hosted task store
that reclaims tasks left in-flight by a crashed prior instance) plus the
periodic recovery loop run — **only when the switch is on**:

```python
from azure.ai.agentserver.core.tasks import set_resilient_tasks_enabled

set_resilient_tasks_enabled(True) # force-enable recovery before any task
set_resilient_tasks_enabled(True) # opt in BEFORE host startup (e.g. at import)
```

Use `resilient_tasks_enabled()` to read the current switch state.

The `TaskManager` itself is always constructed (a cheap, in-memory object that
makes no task-store calls until a task is used), so `get_task_manager()` and
`.run()` / `.start()` work regardless of the switch. A server that neither
declares a task nor sets the switch (e.g. an invocations-only host) simply
skips the startup recovery scan and pays none of its latency.
Merely declaring a durable task (`@task` / `@multi_turn_task`) does **not** turn
the subsystem on. When the switch is off, no `TaskManager` is installed:
`get_task_manager()` raises `TaskManagerNotInitialized` and `.run()` / `.start()`
cannot run a task. A server that does not set the switch (e.g. an
invocations-only host) constructs no manager and pays none of the recovery-scan
latency.

> **Protocol note:** the responses protocol exposes a `resilient_background`
> server option that maps to this switch — constructing a
> `ResponsesAgentServerHost(options=ResponsesServerOptions(resilient_background=True))`
> enables the subsystem automatically, so responses apps typically don't call
> `set_resilient_tasks_enabled` directly.

### One-shot

Expand Down
Loading
Loading