Resync session tools when backend health changes

[premctl]

Summary

A vMCP session's advertised tool catalog is derived once at registration, and the existing resync machinery (#5748) only re-derives it when a connected backend itself emits notifications/tools/list_changed. A backend flipping unhealthy → healthy (or the group's backend set changing) emits nothing — the health monitor detected the transition but nothing consumed it — so already-connected sessions kept serving the stale catalog until they reconnected, even though capability aggregation already health-filters the backend set. This is PR 1 of the two-PR split agreed with @aponcedeleonch in #5786: passthrough mode only; the optimizer-mode catalog rebuild is the follow-up PR.

• pkg/vmcp/health: add Monitor.OnChange(fn ChangeListener) (also on health.Reporter), fired when a backend's advertisability flips — a transition across the healthy/degraded ⇄ unhealthy/unknown/unauthenticated partition, detected at the statusTracker.RecordSuccess/RecordFailure transition points, mirroring filterHealthyBackends' inclusion rule — and when UpdateBackends adds/removes backends. Delivery is debounced to the monitor's check interval (leading edge immediate, in-window changes coalesced into one trailing delivery carrying a monotonic generation), so a flapping backend or multi-backend partition cannot storm listeners. Listeners run off the health-check path; Stop waits for in-flight deliveries.
• pkg/vmcp/server: Serve subscribes via the core-owned monitor. The server keeps a registry of each live session's KindTools resync worker — the same per-session coalescing worker the backend-notification path builds, so identity/forwarded-header capture, the liveness guard, capability-cache invalidation, replace semantics, and the SDK's automatic downstream notifications/tools/list_changed emission are all shared — and triggers each on delivery. Sessions register on successful registration, deregister on server-observed termination paths, and are pruned lazily when a triggered resync finds them gone (TTL expiry / SDK-initiated DELETE end sessions without server involvement).
• Passthrough-only gate: with the optimizer enabled the fan-out is a no-op (the advertised find_tool/call_tool meta-tools don't change on a health flip; rebuilding their backing index is PR 2).
• docs/arch/10-virtual-mcp-architecture.md: new "Health-driven tools resync" section.

Part of #5786 (PR 1 of 2 — do not auto-close; PR 2 covers optimizer mode)

Type of change

• Bug fix
• New feature
• Refactoring (no behavior change)
• Dependency update
• Documentation
• Other (describe):

Test plan

• Unit tests (task test) — full suite green with race detection: debounce semantics (leading/trailing coalescing, stop cancels pending delivery, generations strictly increasing); statusTracker advertisability-transition matrix; monitor fires OnChange on fail→recover and on UpdateBackends add/remove, and stays quiet in steady state; server fan-out resyncs a live session (gains recovered tool, drops failed tool, invalidates capability cache), coalesces a 10-delivery burst into two re-derivations, no-ops in optimizer mode, and prunes dead sessions
• E2E tests — new virtualmcp_health_list_changed_test.go spec run against a Kind cluster (kind-setup-e2e + operator-deploy-local): a Legacy session initialized while a backend is broken receives notifications/tools/list_changed on its standalone SSE stream when the backend recovers, then lists and successfully calls the recovered backend's tool on the same session without reconnecting
• Linting (task lint-fix)
• Manual testing (describe below)

Does this introduce a user-facing change?

Yes. Connected vMCP client sessions (passthrough mode) now receive notifications/tools/list_changed and an updated tools/list when a backend recovers/fails health checks or when the group's backend set changes, instead of serving the registration-time snapshot until reconnect.

Implementation plan

Approved implementation plan (agreed in #5786, PR 1 scope)

• pkg/vmcp/health/monitor.go: add type ChangeListener func(gen uint64) and func (m *Monitor) OnChange(fn ChangeListener). Fire it from the statusTracker transition points (healthy⇄unhealthy) and from UpdateBackends (add/remove), debounced so a flapping backend doesn't storm sessions. Callback on Monitor rather than a channel, per the thread.
• Core/aggregation: re-aggregate against the new healthy set and invalidate the cached merge so it doesn't keep serving the stale aggregation.
• Server + session registry: re-apply each live session's advertised tools post-init (add new, remove gone), once the session's notification channel is live — not via the registration-time setSessionToolsDirect bypass as-is, per the thread's note on why that bypass exists.
• Emit notifications/tools/list_changed to clients after applying, gated on the server advertising tools.listChanged.
• Concurrency: single writer of the catalog view; sessions apply under their existing session lock; skip sessions already up to date; an in-flight call_tool never resolves against a mismatched route.
• PR 1 = passthrough mode only; PR 2 = optimizer-mode find_tool/call_tool catalog rebuild.

Special notes for reviewers

The agreed design predates #5748/#5969 landing, and several of its steps are now provided by machinery that didn't exist when the thread discussion happened — this PR wires the health monitor into that machinery rather than re-implementing those steps:

• "Re-run AggregateCapabilities and atomically swap the catalog + routing table, bump a generation" — the core is now stateless (P2.5 Move AS runner, status reporter, optimizer, health monitor under Serve #5443/P3.2 Reduce server.New body to the wrapper #5445: it health-filters and aggregates per call; "the core filters, Serve caches"), so there is no stored catalog to swap and no background re-aggregation to run. A background re-aggregation would in fact be wrong today: the capability cache and outbound backend auth are keyed on per-identity context (Consume backend notifications in vMCP and propagate list_changed #5748), which a background subscriber doesn't have — only the per-session workers carry the captured identity. The atomicity concern is inherently satisfied (each call derives one consistent aggregation+routing view), and the cache serves no stale merge (its key includes the backend-ID set, which a health flip changes; the shared resync path additionally invalidates it).
• "Apply via the existing setSessionToolsDirect path" / "gate on advertising tools.listChanged" — Consume backend notifications in vMCP and propagate list_changed #5748 already built the correct post-init apply path (resyncSessionTools: REPLACE semantics, so removals propagate, unlike setSessionToolsDirect's registration-time merge) and already flipped WithToolCapabilities(true); the SDK emits the notification to each session whose tool store changes, so no explicit SendNotificationToAllClients call is needed (it would duplicate).
• Generation counter — kept as the ChangeListener payload (monotonic, coalesced by debouncing) for correlation; the "skip an already-applied session" role is subsumed by the per-session workers' dirty-flag coalescing.
• The debounce window is the monitor's own CheckInterval (default 30s, same default as the status-reporting interval the thread referenced) — transitions are detected at check cadence, so this is the natural window, and it avoids threading the server-layer reporting interval into the core-owned monitor.
• Registry lifecycle: sessions deregister eagerly on every termination path the server observes — including SDK-initiated HTTP DELETE, via a thin SessionIdManager wrapper (pruneOnTerminateSessionIDManager), since that path otherwise reaches the session manager without passing through server code. Only TTL expiry is pruned lazily (worker's liveness guard on the next fan-out); such an entry retains the worker closure (SDK session + captured identity/headers) until then, which the registry doc comment states explicitly.
• No eager cache purge is needed on a fan-out with zero live sessions: the capability cache is keyed on (identity, forwarded headers, filtered backend-ID set), so a health flip changes the key and any session registering after the flip re-sweeps by construction; pre-flip entries age out via TTL. The per-session resync path still purges before re-deriving (shared Consume backend notifications in vMCP and propagate list_changed #5748 behavior, needed there because a backend's content can change under an unchanged backend set).
• UpdateBackends property changes (URL/transport) intentionally do not notify — membership-only, per the agreed scope; noted in the architecture doc.
• E2E covers unhealthy→healthy on a live session; the healthy→unhealthy direction (tool removal) exercises the same replace path and is covered by the server unit tests.
• Size: 437 changed lines across 8 files excluding tests/docs — marginally over the 400-line guideline. A meaningful share is doc comments on the new concurrency surfaces; splitting the monitor half from the server half would leave neither independently useful, so I kept the scope agreed in the issue thread as one PR. Happy to split if preferred.

Generated with Claude Code

[amirejaz]

Really thorough PR — thanks for adapting the agreed design to the machinery that landed after the #5786 discussion (reusing the #5748/#5969 per-session resync workers rather than re-implementing the re-apply/notify steps is exactly right, and the "Special notes for reviewers" mapping made this easy to verify). Correctness and lifecycle hold up on my read: no synchronous re-entry into the monitor locks from notify() so Stop can't deadlock, eager deregistration on every server-observed termination path plus the lazy liveness-guard prune, and advertisable matching filterHealthyBackends' inclusion rule.

A couple of small, non-blocking items (inline). Nothing here needs to hold the PR up if you'd rather defer.

One more optional note: the tools resync worker is registered (healthResync.add) just before injectCoreSessionCapabilities runs, so a health flip in that narrow window could run SetSessionTools (REPLACE) concurrently with registration's setSessionToolsDirect (MERGE). Both derive from the live health-filtered core view and the SDK store is internally locked, so it self-heals on the next fan-out — and registering early is a deliberate "don't miss a change" choice — but a one-line comment acknowledging the overlap would help the next reader.

[amirejaz]

Non-blocking: this is now a second hand-kept copy of the healthy/degraded/"" inclusion rule that also lives in filterHealthyBackends (pkg/vmcp/core/core_vmcp.go). The "keep the two in sync" comment is good, but a silent divergence here would either fire OnChange for a transition the catalog doesn't observe or (worse) suppress one it does — with no test likely to catch it. Worth extracting a single shared predicate (e.g. exported from this package or on vmcp.BackendHealthStatus) that both call, so they can't drift. Cheap now, and the coupling is only going to get more load-bearing in PR2.

[amirejaz]

Optional: since this is a no-op in optimizer mode, the healthResync.add at registration also serves no purpose there — those entries are only reclaimed via a terminate path or a backend-driven resync, so an optimizer-mode session that TTL-expires without ever emitting a backend list_changed lingers in the registry (holding its worker closure + captured identity/headers) until then. Bounded and low-severity, but skipping the add when optimizerFactory != nil would avoid it and read more clearly. If you're keeping it because PR2 will need the registration regardless, a one-line note to that effect would do.

[codecov]

Codecov Report

❌ Patch coverage is 95.65217% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.49%. Comparing base (aa3f5b3) to head (ad69d8e).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/vmcp/health/change_notifier.go	95.34%	1 Missing and 1 partial ⚠️
pkg/vmcp/health/status.go	83.33%	2 Missing ⚠️
pkg/vmcp/server/server.go	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6196      +/-   ##
==========================================
- Coverage   72.51%   72.49%   -0.03%     
==========================================
  Files         739      741       +2     
  Lines       76719    76822     +103     
==========================================
+ Hits        55634    55692      +58     
- Misses      17107    17164      +57     
+ Partials     3978     3966      -12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.