fix(compiler): retry transient LLM API timeouts with bounded backoff

[sebastianbraun25]

Problem

During batch document ingestion (openkb add), ~15-20% of planned concepts and entities fail to generate with transient timeout errors:

litellm.Timeout: AnthropicException Timeout - Gateway Timeout

This occurs under sustained high-concurrency load (5 parallel requests over ~150 seconds) when multiple documents trigger concept/entity generation in the same batch run. Despite the error, the document is marked [OK] and added with incomplete concept/entity coverage, reducing knowledge base quality.

Root Cause

The Anthropic API gateway experiences overload after ~6-7 concurrent tasks sustained for 150+ seconds. This is a transient, recoverable server-side phenomenon (not a client-side timeout misconfiguration). The same request succeeds on retry because the gateway recovers within exponential backoff windows (2-4 seconds).

Solution

Implemented hybrid retry mechanism combining LiteLLM's built-in retry capability with selective exception filtering:

• Retryable errors (automatically retried up to 2 times with exponential backoff):
  • Timeout / Gateway Timeout
  • RateLimitError / 429 Too Many Requests
  • ConnectionError
  • ServiceUnavailableError / 5xx API errors
• Non-retryable errors (failed immediately):
  • ValueError, TypeError (client-side bugs)
  • AuthenticationError, BadRequestError (permanent auth/validation failures)
  • TruncatedResponseError (output already truncated; retry won't fix it)
  • Unknown/generic errors (conservative: don't retry)

Changes

1. Added _should_retry_exception() function (~60 LOC)

   • Classifies exceptions as retryable (True) or permanent (False)
   • Checks exception type name against hardcoded allowlist
   • Safely handles exceptions without standard attributes

2. Modified _llm_call() function (~30 LOC changes)

   • Passes num_retries=2 kwarg to litellm.completion()
   • Wraps call in try/except to log transient vs. permanent failures
   • LiteLLM handles exponential backoff internally (base 2)

3. Modified _llm_call_async() function (~30 LOC changes)

   • Passes num_retries=2 kwarg to litellm.acompletion()
   • Same logging strategy as sync variant
   • Preserves async execution semantics

4. Added comprehensive unit tests (17 test cases in tests/test_compiler_retry.py)

   • Tests each retryable exception type (Timeout, RateLimit, Connection, ServiceUnavailable)
   • Tests each non-retryable type (ValueError, TypeError, Auth, BadRequest, Truncation)
   • Tests conservative fallback (unknown exceptions not retried)
   • Tests that num_retries (not retries) is forwarded to litellm.completion/acompletion, and that an explicit caller-supplied num_retries isn't overridden

Behavior

• Transient error + successful retry: LiteLLM logs "retrying..." internally; operation succeeds; audit log shows: "LLM [X] failed with transient error (retries applied by litellm)"
• Transient error + all retries exhausted: Exception re-raised after 3 attempts; audit log shows full traceback
• Permanent error: Fails immediately without retry; audit log shows: "LLM [X] failed with permanent error (no retry)" + traceback

Configuration

No configuration changes required. Retry behavior is fixed:

• Retries: 2 (= 3 total attempts: original + 2 retries)
• Backoff: exponential, base 2 (LiteLLM default): 2s, 4s between attempts
• Client-side timeout: None (defers to LiteLLM/provider defaults)

Testing

• ✅ 17 unit tests for exception filtering and kwarg forwarding (all pass)
• ✅ Mypy type checking: clean
• ✅ Ruff linting & formatting: clean
• ✅ Backward compatible: no API/config changes

Expected Impact

• Reduces incomplete concept/entity coverage in batch runs from ~80-85% to ~95%+ (empirically observed transient nature of failures)
• No performance regression (retries only on error; backoff delays occur anyway during gateway recovery)
• No operational burden (fixed parameters; no monitoring/tuning needed)

Update: fixed wrong LiteLLM kwarg name (retries → num_retries)

The initial implementation passed retries=2 to litellm.completion()/acompletion(). LiteLLM does not recognize a bare retries kwarg as an internal control parameter (only num_retries/max_retries) — it silently falls through as an unrecognized provider request-body field. Against most providers this is harmlessly dropped, but strict-mode proxies (e.g. custom enterprise Anthropic gateways) reject the request outright:

litellm.BadRequestError: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"retries: Extra inputs are not permitted"}}

This effectively broke every LLM call (and therefore every add) for anyone behind such a proxy — a regression more severe than the original timeout issue this PR set out to fix. Renamed to num_retries in both _llm_call() and _llm_call_async(), and added regression tests asserting the exact kwarg forwarded to litellm.completion/acompletion.

Issues

• Resolves fix(compiler): retry transient LLM API timeouts with bounded backoff #229

[sebastianbraun25]

Withdrawing this PR for now.

Why

Root-cause investigation (in downstream production use) points to these Gateway
Timeout failures correlating with response duration/size rather than random
transient load: the pages that fail do so deterministically on every
attempt (including after a delayed retry sweep tried as a follow-up to this
PR), and they are specifically the largest existing concept/entity pages in
the KB (requiring the model to regenerate a correspondingly large merged
completion). Other, smaller "update" calls in the very same batch run
succeeded reliably.

This is consistent with a fixed-duration timeout enforced by an intermediary
(a corporate/self-hosted LLM gateway or proxy in front of the provider, common
in enterprise deployments of OpenKB) rather than genuinely transient upstream
load — a client-side retry policy does not address that, and simply retrying
the exact same slow request reproduces the exact same timeout. Anyone running
OpenKB behind a similar proxy would likely hit this same wall first, so
shipping a retry-only fix risks masking the real issue rather than solving it.

Not a rejection of the approach

The underlying fix (using LiteLLM's recognized num_retries kwarg instead
of the unrecognized retries) may still be independently correct and worth
having — it fixes a real, separate bug (a request field silently dropped, or
rejected outright by strict-mode proxies) that has nothing to do with the
timeout-duration issue above. The branch is kept on our fork (not deleted) in
case it's worth reactivating once the timeout-duration root cause has its own
fix (e.g. streaming responses, or bounding how large a single page rewrite can
grow), or independently of it.

Closing #229 with the same reasoning.