feat(ext): add route selector extension point and adaptive strategy - #611
Conversation
Extensions can now steer load balancing for virtual models. A new ext.RouteSelector registers through the registry's single selector slot and is consulted by redirects using the new "adaptive" strategy; it picks among the currently viable targets (catalog-supported, with rate-limit capacity) and observes every upstream attempt — primaries, retries, and failovers — through the llmclient hooks, so it can score targets by real traffic. Session affinity, capacity filtering, failover chains, and retries all remain core's responsibility. Without a registered selector — or when the selector declines, answers outside the pool, or panics (contained and logged) — the adaptive strategy behaves exactly like round_robin, so configs stay portable between core and extended builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds the ChangesAdaptive routing
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant ExtensionRegistry
participant App
participant ProviderFactory
participant VirtualModelService
participant RouteSelector
participant Dashboard
ExtensionRegistry->>App: expose registered selector
App->>ProviderFactory: install attempt hooks
App->>VirtualModelService: install selector
VirtualModelService->>RouteSelector: select from RouteRequest
ProviderFactory->>RouteSelector: report attempt lifecycle
App->>Dashboard: publish supported strategies
Dashboard->>App: load runtime strategy configuration
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@config/config.example.yaml`:
- Line 64: Update the strategy comment near the strategy configuration example
to describe the routing extension as optional rather than required, while
preserving that adaptive is valid without a selector and falls back to
round_robin.
In `@ext/registry_test.go`:
- Around line 88-102: Convert TestRegistryRouteSelectorSingleSlot and
TestRouteTargetQualified into table-driven tests covering unset, replacement,
and nil-reset selector behavior plus route-target qualification. Reuse each
case’s inputs and expected outputs through subtests, preserving the current
assertions and behavior.
In `@internal/app/app.go`:
- Around line 117-127: Update the recovery handler inside routeSelectorHooks so
it performs no extension calls, including selector.Name(), and logs only fixed,
non-sensitive metadata such as the observation event. Remove the recovered panic
value from slog.Error while preserving panic containment and the existing error
message.
In `@internal/virtualmodels/adaptive_test.go`:
- Around line 82-88: Expand the request-translation assertions in the relevant
test to compare the complete req.Candidates slice against expected
ext.RouteCandidate metadata, including Weight, OutputPerMtok, InputPerMtok,
provider/model fields, and nil versus populated optional prices. Convert the
coverage to table-driven tests so multiple candidate and parameter-mapping cases
are validated, while preserving the existing request translation behavior.
In `@internal/virtualmodels/adaptive.go`:
- Around line 20-25: Update the deferred recovery handler around Select so panic
reporting does not call the extension-provided selector.Name(); use a constant
label or panic-safe helper while preserving the resolvedTarget{} false fallback.
Add a regression test covering both Select and Name panicking and verify
round-robin fallback is returned without another panic.
- Around line 22-23: Update the panic recovery logging around the route selector
fallback to stop including the raw recovered value r. Remove the panic field or
replace it with a non-sensitive panic classification, while preserving the
selector, source, and fallback behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: fbb3af64-6418-4421-a897-87bf4dde574b
📒 Files selected for processing (14)
config/config.example.yamlconfig/virtualmodels.goext/ext.goext/registry.goext/registry_test.goext/route.gointernal/admin/handler_virtualmodels.gointernal/app/app.gointernal/virtualmodels/adaptive.gointernal/virtualmodels/adaptive_test.gointernal/virtualmodels/balancer.gointernal/virtualmodels/service.gointernal/virtualmodels/types.gointernal/virtualmodels/validation.go
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
What T-Rex did
|
Panic recovery around selector calls no longer re-enters extension code: the selector's name is captured panic-safe at install time, so a Name that panics mid-recovery can no longer escape the handler and abort the request instead of falling back. The recovered value is no longer logged — it is extension-controlled and could carry request data. Candidate pricing is now defensively copied: selectors received the catalog's own *float64 values, so a misbehaving extension could mutate the prices that cost routing and cost calculation later read. The per-attempt observation contract is corrected to match behavior: hooks fire once per upstream call, and transport-level retries inside the provider client are aggregated into that call's single outcome (final status, duration spanning retry backoff). Moving observation inside the client retry loop would change hook semantics for every consumer (metrics, health tracking) and is out of scope here. Also: registry selector-slot tests are table-driven (incl. nil reset), the delegation test asserts the full candidate set, a regression test covers the double-panic path, and the example config no longer implies the adaptive strategy requires an extension. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed the review findings in 189beca:
#612 has been rebased on top. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/virtualmodels/adaptive_test.go`:
- Around line 119-135: Expand TestBalancer_AdaptiveCandidatePricingIsCopied into
a table-driven isolation test that mutates both InputPerMtok and OutputPerMtok
for every selector candidate, then verifies the corresponding pricing pointers
for every catalog model remain unchanged. Use the existing candidate and catalog
model symbols to cover all candidates and both pricing fields, while preserving
the current defensive-copy assertion behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 10507680-ad9f-47ea-a5ff-fc0f9baa0dc0
📒 Files selected for processing (7)
config/config.example.yamlext/registry_test.goext/route.gointernal/app/app.gointernal/virtualmodels/adaptive.gointernal/virtualmodels/adaptive_test.gointernal/virtualmodels/service.go
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Follow-up from the second review pass in b3c07a4: the pricing-isolation test now mutates every pricing pointer of every candidate and verifies all catalog prices stay intact. |
The strategy dropdown was hardcoded to round_robin and cost, so the new adaptive strategy was unreachable from the UI even on deployments with a route selector registered — and hardcoding would have offered it on plain core, where it silently falls back to round robin. /admin/runtime/config now carries VIRTUAL_MODEL_STRATEGIES, the comma- separated strategies this deployment supports; adaptive appears only when a route-selector extension is registered. The backend owns the values, the dashboard owns the labels (unknown values render as their raw value), and the editor keeps the edited row's current strategy selectable even when the deployment no longer offers it, so opening an adaptive virtual model on a plain-core dashboard no longer renders a blank select that could silently rewrite the strategy on save. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Added in 454b7cc (discussed with the team): the strategy dropdown is now server-driven. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/dashboard/src/pages/models/virtualModelsLogic.js`:
- Around line 171-175: Update the adaptive label in STRATEGY_OPTION_LABELS at
web/dashboard/src/pages/models/virtualModelsLogic.js lines 171-175 to describe
target selection as delegated to the configured route-selector extension,
without promising health, latency, or cost inputs. Apply the same generic
extension-defined wording in
web/dashboard/src/pages/models/VirtualModelEditor.svelte lines 47-49.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 4cb4e178-95eb-4e78-8333-da9006334974
⛔ Files ignored due to path filters (3)
internal/admin/dashboard/static/dist/assets/index-BOEWOpVo.jsis excluded by!**/dist/**internal/admin/dashboard/static/dist/assets/index-jnxLFx0i.jsis excluded by!**/dist/**internal/admin/dashboard/static/dist/index.htmlis excluded by!**/dist/**
📒 Files selected for processing (9)
internal/admin/handler.gointernal/app/app.gointernal/app/app_test.gointernal/virtualmodels/adaptive_test.goweb/dashboard/src/lib/stores/runtimeConfig.svelte.jsweb/dashboard/src/pages/models/VirtualModelEditor.svelteweb/dashboard/src/pages/models/virtualModels.svelte.jsweb/dashboard/src/pages/models/virtualModelsLogic.jsweb/dashboard/tests/models-virtual-models.test.js
The label promised health/latency/cost routing, but core's contract only delegates target choice to the registered extension — describe that, not one extension's policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Addressed in fe6a0e6: the adaptive label and help text now describe the contract ("target chosen by the registered routing extension") rather than one extension's policy. |
# Conflicts: # internal/admin/dashboard/static/dist/index.html # web/dashboard/src/pages/models/VirtualModelEditor.svelte
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/dashboard/src/pages/models/VirtualModelEditor.svelte`:
- Around line 50-52: Update the adaptive help-text condition in
VirtualModelEditor to check the server-provided supported-strategy list for
adaptive support instead of vm.vmStrategyOptions(). Preserve the existing
explanatory text and rendering behavior, including compatibility with edited
unsupported strategy values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 4bd5d9c5-0785-4ff4-8f66-1826b4a6e263
⛔ Files ignored due to path filters (2)
internal/admin/dashboard/static/dist/assets/index-CXC5vqx6.jsis excluded by!**/dist/**internal/admin/dashboard/static/dist/index.htmlis excluded by!**/dist/**
📒 Files selected for processing (1)
web/dashboard/src/pages/models/VirtualModelEditor.svelte
The help condition used the dropdown options, which include an edited row's unsupported value for compatibility — so an adaptive virtual model opened on a deployment without a route selector still showed help claiming a registered extension picks targets. Gate on the server-provided strategy list instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
CodeRabbit's help-text finding fixed in 094d1d1: the adaptive help sentence now gates on the server-provided strategy list ( |
# Conflicts: # internal/admin/dashboard/static/dist/index.html
…normalization normalizeDashboardRuntimeConfig rebuilt the response field-by-field and dropped VirtualModelStrategies, so /admin/runtime/config never exposed the strategy list and the dashboard could not offer adaptive. Carry the field through and assert it in the handler-level runtime-config test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Summary
Adds a routing extension point to the public
extAPI so extension builds can steer load balancing for virtual models:ext.RouteSelector(ext/route.go):Selectpicks among currently viable targets of a load-balanced virtual model;OnAttemptStart/OnAttemptEndobserve every upstream attempt (primaries, retries, failovers) via the llmclient hooks, so selectors score targets from real traffic. Registered through a single slot on the registry (RegisterRouteSelector).adaptiveload-balancing strategy for virtual models: delegates target choice to the registered selector. Candidates carry provider/model, configured weight, and registry pricing.internal/app/app.go): selector hooks attach before providers are created (same rule as the health tracker); the virtual-models service gets the selector alongside the capacity probe.Safety / portability
round_robinbehavior. Configs usingadaptivestay portable between plain core and extended builds.Tests
ext: registry single-slot semantics,RouteTarget.Qualified().virtualmodels: selector delegation (request shape incl. pricing), fallback matrix (no selector / decline / out-of-pool answer / panic), single-target bypass.make test-race).🤖 Generated with Claude Code
Summary by CodeRabbit