Inference section — user-configured model connections

[JArmandoAnaya]

This feature spans the kernel, the wire, the server, the CLI, MCP and the UI. It sits between
the ModelProvider port that declares how a model is asked for a prediction (#418) and the work
that surfaces model provenance and confidence in the review loop (#417).

It carries deferred-needs-input because the two open questions at the bottom block UI placement
and persistence design. They do not block the kernel work.

Summary

VisionSet never bundles or auto-downloads inference models. Auto-labeling operates
against model connections the user creates: a new top-level configuration surface
where connections to local or remote models are created, set up, tested, and managed.
The auto-labeling feature itself is always present (not an optional extra); only the
local runtime ships as visionset[local-inference].

Ratified direction (Armando, 2026-08-07), recorded in full on the port issue (#418):

• ModelProvider adapters are instantiated from user-created connections, never from a
  bundled default.
• VisionSet never downloads weights at install time; download is an explicit user
  action during local-connection setup, with the size shown beforehand.
• Nothing may preclude cloud execution later — a hosted adapter is just another
  connection type against the same port.

The user journey (design target)

Empty state. Centered card: "Connect a model to enable auto-labeling", primary CTA
Add connection. Meta line stating the policy as a value proposition: "VisionSet
never downloads models on its own — you choose what runs and where."

List. Compact rows (density principle): connection name · type chip (Local /
HTTP) · model identity (grounding-dino-base @ <revision>) · status (Ready /
Not set up / Unreachable), semantic token + text, never color alone. Per-row
overflow: edit, test, delete.

Create — two steps. Step 1: pick type. Step 2 by type:

• Local: HF model id + revision (suggested default; download size shown before
  confirming), device, precision. If local-inference is not installed, the form stays
  usable and shows the exact command (pip install "visionset[local-inference]") —
  never a bare disabled control (design principle 9). Creating the connection downloads
  nothing; the row lands at Not set up with a Download weights action.
• HTTP: endpoint URL, optional credential, and a Test button that makes a real
  call and reports what answered. The endpoint contract is defined by this project —
  the same contract a future hosted adapter implements, and anyone may implement it.
  Third-party inference protocols are not adopted as the contract.

Download weights runs as a background job on the existing infrastructure (SQLite
queue in visionset.db, workers in the FastAPI lifespan, /background-jobs routes,
observable progress).

Delete states its blast radius accurately: annotation provenance stores model
identity denormalized (id + revision copied at write time, no FK to the
connection), so deleting a connection never breaks provenance. Dialog copy: "Annotations
keep their model provenance; only this configuration is removed."

Scope by layer

Kernel (visionset.kernel) — one home, one spelling:

• New aggregate InferenceConnection: id, name, type (local | http), model
  reference (id + revision), per-type parameters, setup state.
• The ModelProvider port (ModelProvider: declare the kernel port for autolabeling — local and cloud adapters both ordinary #418) is unchanged. Resolving a connection to an adapter happens in
  the composition root, outside the kernel, so the kernel knows the configuration and the
  protocol and never torch or transformers.
• The annotation write gate and the provenance work (Model-produced annotations: provenance and confidence surfaced in the review loop #417) are untouched, because a connection
  only determines who infers.

Persistence:

• New workspace-scoped table (like background_job: no project_id), narrow port
  methods only.

Wire + server:

• ConnectionOut with allowed_actions derived from the same transition tables the
  service enforces (standing contract): download_weights only for local + not-set-up;
  test only for HTTP; delete always, with the blast-radius message.
• Routes under /inference/connections. Weight download responds 202 pointing at its
  background job. Regenerate and commit openapi.json.

CLI + MCP (SDK-first parity):

• visionset inference list|create|test|download with --json.
• Equivalent MCP tools — an agent must be able to make a workspace auto-label-ready
  without a browser.

UI:

• Screens in ui-core, navigation as callbacks, render allowed_actions only, existing
  semantic status tokens — zero new colors.
• Annotation-workspace suture: when auto-labeling is offered in the editor with no
  connections configured, principle 10 forbids forcing navigation out of the editor or
  losing work — same pattern as in-editor quick class creation, or minimally a
  state-preserving link. Design explicitly, do not improvise at implementation time.

Open questions — Armando's, do not decide (deferred-needs-input)

1. Navigation placement. A top-level Inference section re-opens the closed-rail rule set
   by the routes-and-composition work (app: routes and composition — deliberately thin; all domain logic lives in ui-core (the enterprise rule) #58), which fixed the rail at "Home, Projects, avatar —
   nothing else". Two options are prepared rather than picked: (a) a new rail entry
   Inference; (b) another placement that leaves the app: routes and composition — deliberately thin; all domain logic lives in ui-core (the enterprise rule) #58 rule intact. Whichever is chosen is
   recorded as a superseding decision on that rule in the same turn it is confirmed.
2. Credential storage. HTTP-connection credentials would land in plain SQLite inside
   the workspace. Local-first makes this defensible, but it ships only as an explicit
   decision, never a silent default. Options to prepare: plain storage documented as
   such; OS keyring integration; env-var indirection.

Non-goals

• No model marketplace/registry — out of scope for this distribution.
• No training or fine-tuning surfaces.
• No bundled default connection of any kind.

[JArmandoAnaya]

Slice 2 (UI) is blocked on open question 1

Recording what is verified at main (8b38eed) so the placement decision can be made
against the current state rather than re-derived.

The placement question has no recorded answer

No superseding decision exists, neither on this issue nor on the routes-and-composition work
(#58) that set the closed-rail rule, and deferred-needs-input is still set here. That rule is
live in two places, both citing #58:

• DESIGN.md § Layout — "dark left rail (logo, collapse toggle, Home, Projects,
  account avatar at the bottom — nothing else; app: routes and composition — deliberately thin; all domain logic lives in ui-core (the enterprise rule) #58)"
• DESIGN.md § Navigation rules — "The rail is for top-level destinations only … what
  keeps the rail the four things ## Layout gives it."

So a rail change edits DESIGN.md, not only the sitemap in the
information-architecture skill: that sitemap enumerates routes and tabs, and rail
membership is not in it. Whichever option is taken lands in both documents, with the
supersession recorded here and on #58 in the same turn — this issue's own rule.

Options prepared, not picked

(a) New rail entry Inference. Rail goes from two destinations to three. Costs the
DESIGN.md rule above, a sitemap entry, a PARENT row in frontend/app/src/routes.tsx,
and the supersession on #58. Matches the scope of the object: connections are
workspace-scoped, and the rail is the only workspace-level surface that exists.

(b) The account slot becomes a workspace menu, with Inference an item inside it.
Keeps the rail's membership list literally intact — the slot is already there, currently
holding the one control that forgets the token. Costs a new menu pattern and puts a
frequently-visited configuration surface one click deeper than (a), behind a control
users read as identity rather than as configuration.

(c) A project tab. Recorded so it is not re-proposed rather than because it is
viable: connections carry no project_id by design (workspace-scoped table, as
specified above), and navigation maps 1:1 to domain objects — a project tab would state
a scope the object does not have.

There is no existing settings or workspace-configuration area to place it in; every
product route today is under /projects or /jobs.

Two wire gaps found while verifying, independent of the placement question

1. No download-size surface. Decision D1 of the interactive auto-labeling work (Interactive auto-labeling — click-to-suggest in the editor canvas #424)
   requires the local-connection form to pre-fill "with its download size shown" before
   confirming. Nothing on the wire reports a
   model's size: neither ConnectionCreate nor ConnectionOut carries one, and no route
   answers one. A client cannot compute it. Either the wire gains it, or the pre-fill
   ships without the size and Interactive auto-labeling — click-to-suggest in the editor canvas #424 stays open on that half.

2. No test action. The journey above specifies a per-row Test for HTTP
   connections and a Test button in the create form. ConnectionAction is
   download_weights | update | delete, and there is no
   /inference/connections/{id}/test. The capability is unshipped, so it cannot be
   rendered from allowed_actions, and computing it client-side is not an option.

Everything else the screens need is already present, shipped in slice 1 (#430) with the
InferenceConnection entity, its persistence, wire, REST and CLI: ConnectionPage (items +
total), setup_state (not_set_up | ready), the three actions above, and
POST /inference/connections/{id}/download answering at its background job.

Already in place and waiting

The editor's no-connection panel, specified as decision D6 of #424 and shipped with the
interactive suggest tool (#451), takes its destination as an optional callback, threaded
AnnotationPage → JobScreen → Workspace → SuggestPanel. frontend/app passes none, so
the panel renders its explanation and no control. Wiring it is one callback in routes.tsx
once there is a route to name.