httpware ships a resilience suite under httpware.middleware.resilience, composed via the standard middleware chain (Seam A). It is pure stdlib — no optional extra.
Retry (and AsyncRetry) is a retry middleware backed by a Finagle-style RetryBudget — a token bucket that caps the proportion of traffic spent on retries so a degraded backend cannot be amplified into a retry storm. RetryBudget is a single thread-safe class shared by both worlds: all mutations go through a threading.Lock, so state is never torn. "Safe" here means no corruption, not non-blocking — when one budget is shared across a (sync Client, AsyncClient) pair, a sync thread holding the lock can briefly block the event-loop thread's acquisition. The critical section is intentionally tiny to bound that latency. Backoff between attempts uses full-jitter.
The decision logic — status/method eligibility, streaming-body refusal, exhaustion, Retry-After handling, budget accounting, and the backoff delay — lives once in a stateless private _RetryPolicy.decide, the retry analog of how the circuit breaker keeps its transition logic in one shared state object. Retry and AsyncRetry are thin loop drivers over that policy: they own the attempt loop, the terminal call, and the sleep, and differ only in await next vs next and asyncio.sleep vs time.sleep. decide returns the delay to sleep before the next attempt, or raises the terminal exception (with its PEP 678 note and event already emitted); because it runs inside the wrapper's except block, exception chaining behaves as a direct raise. _RetryPolicy holds the immutable config plus the shared RetryBudget; per-attempt state stays as wrapper locals, so one instance is safe across the concurrent requests it serves.
Bulkhead / AsyncBulkhead is a concurrency limiter. AsyncBulkhead uses asyncio.Semaphore with a bounded acquire wait; sync Bulkhead uses threading.Semaphore. A sync instance cannot share with an async one. Both are sharable across clients (one instance = one shared concurrency pool).
AsyncCircuitBreaker and sync CircuitBreaker are a classic consecutive-failure circuit breaker: the circuit opens after failure_threshold consecutive counted failures, fast-fails while OPEN, admits one probe after reset_timeout (HALF_OPEN), and closes again after success_threshold consecutive probe successes; a probe failure re-opens it. A counted failure is a NetworkError, an httpware TimeoutError, or a StatusError whose status_code is in the effective failure set (default: all 5xx, 500–599); 4xx including 429 count as successes, and any other exception type propagates unchanged without affecting circuit state. When the breaker refuses a request — OPEN, or HALF_OPEN with the single probe slot already taken — it raises CircuitOpenError and never forwards to next; the error's retry_after carries the seconds until the next probe will be admitted, or None when a concurrent probe is already in flight. A breaker instance is sharable across clients (one shared circuit); a sync instance cannot be shared with an async one.
Both AsyncCircuitBreaker and CircuitBreaker expose a read-only state property that returns a public CircuitState enum (CLOSED/OPEN/HALF_OPEN), importable from httpware, for health checks and introspection. The property is a raw read of the stored state: because the OPEN→HALF_OPEN transition is lazy (it fires on the next request after reset_timeout elapses, not on a clock tick), state continues to report OPEN until a request is actually admitted as the probe — reading the property never triggers the transition.
The classic consecutive-failure mode is the default and unchanged. An opt-in time-based failure-rate mode is available: set failure_rate_threshold (a float in (0, 1]) to switch. In rate mode the circuit opens when the observed failure rate over a rolling window_seconds window (default 30.0 s) meets or exceeds the threshold, but only once minimum_calls outcomes have been observed in that window (default 20). The presence of failure_rate_threshold is the sole mode switch: when it is set, the breaker is in rate mode and failure_threshold is ignored (setting both is not an error — rate mode wins). window_seconds and minimum_calls are validated at construction in both modes even though they are inert in classic mode, so an invalid value is rejected eagerly regardless of mode. Half-open recovery (reset_timeout, success_threshold, the single-probe admission) is identical to classic mode. The event names (circuit.opened, circuit.rejected, circuit.half_open, circuit.closed) are the same in both modes; in rate mode the circuit.opened event carries extra attributes — failure_rate, failure_rate_threshold, window_seconds, observed_calls — and its message is "circuit opened — failure rate threshold reached". The rate-flavored circuit.opened is emitted only on the initial trip from CLOSED: a HALF_OPEN probe failure re-opens the circuit through the shared half-open path (a probe failure is a distinct event from a rate/threshold trip), so that re-open's circuit.opened carries the classic failure_threshold / failures attributes (with failures = 1) and the "circuit opened — failure threshold reached" message in both modes. This is intentional; the event name is the stable contract, and a consumer keys off the present attribute set rather than assuming a fixed shape.
AsyncTimeout is an async-only middleware that bounds the total wall-clock for the whole inner pipeline (most importantly across an AsyncRetry loop, whose attempts and backoff sleeps httpx2 cannot bound). It is not a per-call timeout — httpx2's connect/read/write/pool timeouts are the right tool for a single outbound call, and AsyncTimeout does not duplicate them. It rejects a non-finite or non-positive timeout at construction, and on expiry raises httpware TimeoutError. There is no sync Timeout: a sync total-deadline cannot interrupt a blocking call mid-flight, and httpx2 already covers sync per-call timeouts. Sync callers configure httpx2's timeouts directly.
The recommended (documented, not enforced) composition order is AsyncTimeout → AsyncCircuitBreaker → AsyncBulkhead → AsyncRetry → terminal. With the breaker outside retry, an open circuit short-circuits the entire retry loop and the breaker counts one outcome per fully-exhausted retry sequence rather than per attempt.
Retry, Bulkhead, CircuitBreaker, and AsyncTimeout emit operational events via stdlib logging records on dedicated loggers (httpware.retry, httpware.bulkhead, httpware.circuit_breaker, httpware.timeout) and — when opentelemetry-api is installed — OpenTelemetry span events on the active span via trace.get_current_span().add_event(...). The circuit-breaker and timeout event names (circuit.opened, circuit.rejected, circuit.half_open, circuit.closed, timeout.exceeded) join retry.* / bulkhead.* as the stable observability surface; renames are breaking changes.
Every event's url attribute is redacted before emission: user:pass@ userinfo is stripped and the values of known-sensitive query/fragment parameters (api_key, access_token, token, secret, password, …) are masked with REDACTED, so secrets in request URLs do not reach logs or telemetry. Redaction happens once at the _emit_event emission boundary — covering both the log record and the span event — so it cannot be bypassed by a new emit site, and non-secret URLs are left unchanged. The same redact_url sanitizer backs StatusError messages and repr (see Errors).