Skip to content

feat: warn and confirm at resume when active provider/model differs from session's last writer#232

Open
michaeljabbour wants to merge 1 commit into
mainfrom
feat/resume-provider-mismatch-warning
Open

feat: warn and confirm at resume when active provider/model differs from session's last writer#232
michaeljabbour wants to merge 1 commit into
mainfrom
feat/resume-provider-mismatch-warning

Conversation

@michaeljabbour

Copy link
Copy Markdown

Summary

Sessions that switch providers mid-conversation can persist provider-specific content (e.g. Anthropic thinking-block signatures) that bricks the session when replayed against a different provider (Anthropic 400s on invalid signatures). This PR adds the resume-time guardrail: warn — and, interactively, require confirmation — when the provider/model about to handle a resumed session differs from whichever provider/model last wrote its metadata.

Addresses microsoft-amplifier/amplifier-support#208 — implements Option A (warn at resume); Option B's provider-edge normalization is delivered by provider-anthropic#72.

Investigation note (why this PR carries two halves)

The originally-planned amplifier-core PR was skipped. Investigation of amplifier-core found it persists no session files — the kernel only emits events, and its DESIGN_PHILOSOPHY.md explicitly excludes provider selection and logging policy from the kernel. metadata.json — including the existing model field — is written entirely by this repo (app-cli).

So this PR carries both halves:

  1. Write-side mechanism: persist the active provider identity into session metadata alongside the existing model (main.py, both the interactive _save_session() closure and execute_single()'s single-shot save block). Both derive it from get_effective_config_summary().provider_module — the same function the read-side check uses to compute the active provider — so write-side and read-side always agree on source and shape.
  2. Read-side policy: at the single chokepoint every resume surface passes through (create_initialized_session, after transcript/cost restore and before the first session.execute()), compare the active provider/model against whichever provider/model last wrote the session's metadata.

Subtraction over addition: no new amplifier-core coupling, no new file, no new subsystem — just one field added at an existing write site, and one check inserted at the one existing chokepoint every resume path already funnels through.

Behavior

  • Match, or no prior metadata at all: completely silent. Zero behavior change, zero console output, for every session that isn't affected.
  • Mismatch + tty: print a yellow warning ("session was last written by X/Y — now resuming with A/B", plus a dim line on why it matters), then require explicit confirmation via click.confirm (offloaded through asyncio.to_thread, matching the existing KeyboardInterrupt-handler pattern in main.py so the event loop is never blocked). Declining cleans up the already-created session and raises click.Abort — no half-initialized session is left behind.
  • Mismatch + non-tty (CI, piped input, scripted flows): print the warning and continue automatically. Scripted flows must never hang waiting for input that can't arrive. JSON output modes are unaffected — the warning goes through the same console whose .file is already redirected to stderr before create_initialized_session runs in JSON mode, so it can never contaminate JSON stdout.
  • Feature-detection fallback: pre-existing sessions (saved before this PR) have no provider field in their metadata. For those, comparison falls back to model-only. No amplifier-core version bump required — metadata is read directly off disk via SessionStore, not through any core API.
  • Normalization: provider identity may appear as a module id (provider-anthropic) or a bare name (anthropic); both sides are normalized before comparison so this never produces a false-positive warning.
  • Audit trail (optional, included): accepted/warned-through mismatches are appended to a provider_mismatches list in metadata, mirroring the existing _record_bundle_override() pattern.

Where it's wired in

create_initialized_session() in session_runner.py is the single chokepoint every resume surface passes through:

  • amplifier run --resume <id> (commands/run.py)
  • amplifier session resume / amplifier resume / amplifier continue (commands/session.py, via _prepare_resume_context())

The check is gated on config.is_resume and runs once, after Step 7.5 (cost restore) and before Step 10 (approval provider registration) — i.e. before the first session.execute() can run.

Testing

TDD: tests were written first and asserted the exact required behaviors before implementation.

  • tests/test_session_runner.py — new TestNormalizeProviderIdentity, TestWarnOnResumeProviderMismatch, and TestProviderMismatchCheckWiredIntoChokepoint classes (15 new tests) covering: tty mismatch + confirm, tty mismatch + decline (abort + cleanup), non-tty mismatch (warn-only, no confirm), exact match (silent), missing-provider-field fallback (both differing and matching model), provider-form normalization (no false positive), no-prior-metadata silence, best-effort silence on metadata load failure, and that the check is gated on is_resume at the create_initialized_session chokepoint.
  • tests/test_resume_provider_metadata_write.py — new file (2 tests) covering the write side: execute_single()'s saved metadata includes provider matching the resolved config, and existing metadata fields (name, description, ...) survive the save alongside it.

Baseline vs. final: baseline (clean origin/main) is 42 pre-existing failures / 1031 passed / 2 skipped / 1 xfailed (plus 6 test modules that fail to even collect due to an unrelated amplifier_foundation version/export mismatch in the environment — not introduced by this PR). After this change: the same 42 pre-existing failures (identical set, diffed line-for-line) / 1048 passed (+17 new tests, all green) / 2 skipped / 1 xfailed. No regressions; no pre-existing failures were fixed or absorbed into this diff.

Manual E2E sanity check (per the verification recipe): constructed a fake session directory with a metadata.json recording provider-anthropic/claude-sonnet-4-5, and called _warn_on_resume_provider_mismatch() directly against an "active" config resolving to provider-openai/gpt-5, exercising all three branches (tty+accept, tty+decline, non-tty) against a real rich.Console — confirmed the exact rendered warning text, confirm-gating, clean abort with session.cleanup() invoked exactly once on decline, and silent pass-through behavior when nothing has changed.

Part of the cross-provider resume hardening set

Tracked on microsoft-amplifier/amplifier-support#208. Sibling PRs:

…rom session's last writer

Wave 2 of the cross-provider resume hardening set. Investigation found
amplifier-core persists no session files (kernel emits events only; its
DESIGN_PHILOSOPHY.md explicitly excludes provider selection/logging policy
from the kernel) -- metadata.json (including the existing 'model' field)
is owned entirely by this repo. The originally-planned amplifier-core PR
is therefore skipped; this PR carries both halves:

1. Write-side mechanism: persist the active 'provider' identity into
   session metadata alongside the existing 'model' (main.py, both the
   interactive _save_session() closure and execute_single()'s save block).
   Both derive it from get_effective_config_summary().provider_module --
   the same function the read-side check uses -- so write and read agree.

2. Read-side policy: at the single chokepoint every resume surface passes
   through (create_initialized_session, after transcript/cost restore and
   before the first session.execute()), compare the active provider/model
   against whichever provider/model last wrote the session's metadata.
   - Match, or no prior metadata: completely silent, zero behavior change.
   - Mismatch + tty: warn, then require explicit confirmation
     (click.confirm offloaded via asyncio.to_thread, matching the existing
     KeyboardInterrupt-handler pattern). Decline cleans up the
     already-created session and raises click.Abort -- no half-initialized
     session left behind.
   - Mismatch + non-tty (CI/piped/scripted): warn-only, proceed
     automatically -- scripted flows must never hang on input.
   - Feature-detected fallback: pre-existing sessions (no 'provider' field)
     compare model-only. No amplifier-core version bump required --
     metadata is read directly off disk.
   - Provider identity is normalized (module id vs. bare name) before
     comparison to avoid false positives.
   - Accepted/warned-through mismatches are recorded to a
     'provider_mismatches' audit trail, mirroring _record_bundle_override().

Addresses microsoft-amplifier/amplifier-support#208 -- implements Option A
(warn at resume). Option B's provider-edge normalization is delivered by
provider-anthropic#72.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant