A second architecture answers a point prompt

docker/compose.yaml

@@ -185,6 +185,20 @@ services:
       # port open to every interface would hand the workspace to the LAN. The two
       # lines belong together and neither is safe alone.
       VISIONSET_UI_SESSION: always
+      # Forwarded from whoever ran `docker compose`, empty when they have none.
+      # Some published weights are served only to accounts that have been granted
+      # them, and the refusal for fetching one without that names `HF_TOKEN` as
+      # the remedy. Without this line the remedy is unreachable from this stack:
+      # the reader sets the variable on the host, the container never sees it, and
+      # the same refusal comes back with nothing about it changed. A refusal
+      # naming a remedy nobody in that position can take is the mistake
+      # `NotAWorkspace` is the standing example of.
+      #
+      # It carries no default and belongs to no mode: a stack without the
+      # inference runtime never fetches anything, and one with it fetches ungated
+      # weights exactly as before, because the hub client treats an empty value
+      # as no token at all.
+      HF_TOKEN: ${HF_TOKEN:-}
     # Raw uvicorn rather than `visionset server`, and deliberately so: that
     # command also serves the compiled bundle at /app, which in development is
     # vite's job on :5173 — and building that bundle is exactly what this stack

docs/architecture/backend/inference.md

@@ -148,9 +148,21 @@ Third-party inference and labeling-ecosystem frameworks are not taken as runtime
 dependencies; external work in that space is design reference only. A candidate
 that exists solely inside one is excluded on that ground alone.
 
-**Copyleft, business-source and custom-gated licenses are never bundled and never
-curated as defaults.** At most they are reachable through the **Custom model...**
-path a user types themselves, at that user's own compliance risk.
+**Copyleft and business-source licenses are never bundled and never curated.** At
+most they are reachable through the **Custom model...** path a user types
+themselves, at that user's own compliance risk.
+
+**A gated model may be curated, and may never be the default** (2026-08-15,
+superseding the narrower rule this replaces, which excluded gated licenses
+alongside copyleft ones). Nothing is bundled either way: weights are fetched by
+the user, from the publisher, after that user has accepted the publisher's terms
+themselves, which is the same act whether the id came from a curated list or was
+typed in. What curation adds is that the requirement is *stated* - the entry
+carries the sentence and the page where access is requested, and the form shows
+both before a download is offered. Leaving such a model out does not spare anybody
+its terms; it means the people who want it find the id elsewhere and type it in,
+having read nothing. The default entry must stay one that anybody can fetch, so
+that opening the form is never itself a refusal.
 
 **`transformers`-native is strongly preferred**, because it is one integration
 surface. A candidate published only as a raw `.pt` or `.onnx` checkpoint needs its
@@ -159,8 +171,10 @@ entry however small the model is.
 
 Curated entries live in
 [`inferenceCatalog.ts`](../../../frontend/ui-core/src/screens/inferenceCatalog.ts),
-which records a model id, a pinned revision and a download size per entry and
-asserts the license once, in the module's prose, for the whole list. Pinning the
+which records a model id, a pinned revision and a download size per entry. The
+module's prose asserts the license for the list and names its one exception by
+hand; an entry that carries a requirement of its own carries it as data, in
+`access`, because that is the half the interface has to render. Pinning the
 license per entry the way the revision is pinned is what would make a family-level
 assumption impossible to write by accident.
 
@@ -169,6 +183,7 @@ assumption impossible to write by accident.
 | Family (`model_type`) | Capability | Curated checkpoints | License |
 | --- | --- | --- | --- |
 | `sam2`, `sam2_video` | `point_suggest` | `facebook/sam2.1-hiera-{tiny,small,base-plus,large}` | Apache-2.0 |
+| `sam3_video` | `point_suggest` | `facebook/sam3` | SAM License, access granted by request |
 | `grounding-dino` | `text_detect` | `IDEA-Research/grounding-dino-{tiny,base}` | Apache-2.0 |
 | `mm-grounding-dino` | `text_detect` | none | - |
 
@@ -254,15 +269,54 @@ question and not a family entry.
 
 ### The gate a gated model hits
 
-`facebook/sam3` is a capability step-change - concept segmentation from text or
-exemplar prompts - and it is **gated**: the hub reports `gated: manual` and a
-`license: other`, and its `config.json` cannot be read without accepting terms.
+**Both blockers recorded here are now cleared, and `facebook/sam3` ships as a
+curated `point_suggest` entry.** What follows describes how, because the shape is
+what any future gated candidate inherits.
+
+The hub reports `gated: manual` and a `license: other`, and the repository's
+`config.json` cannot be read without being granted access. The license question is
+answered by the gated-model rule above: nothing is bundled, the user fetches from
+the publisher after accepting the publisher's terms, and the requirement is stated
+in the form before a download is offered.
+
+The fetch question turned out to need no mechanism at all. The download job reaches
+the hub through its standard client, which resolves a token from the environment
+on its own, so an authenticated fetch is `HF_TOKEN` being set where the server or
+the CLI runs and nothing in a workspace. What was missing was not the capability
+but the *sentence*: an unauthenticated fetch raised an error whose text opened with
+a status line and a request id, and re-raising it carried both to a reader. That is
+now translated into the remedy.
+
+Two measurements worth keeping, because both contradict the assumption a reader
+would otherwise make. **Reading a size needs no access** - the hub answers
+`model_info` for a gated repository unauthenticated - which is what lets the size
+line work before anybody has asked for anything. And **a gated repository answers
+401 rather than 403**, the same status a repository that does not exist answers, so
+the two are told apart by the client's exception class and never by the code. A
+branch on the number would send somebody with a typo to request access to a model
+nobody publishes.
+
+Point prompts run through the tracker half of the architecture, which agrees with
+SAM 2 signature for signature; the concept segmentation the model is named for is
+not integrated, and the entry declares `point_suggest` alone.
+
+**Which name a connection resolves on is the trap here, and it cost a revision.**
+The repository publishes one artifact carrying the whole architecture, and the
+family is read from the top of its config:
 
-Two separate blockers, and only the second is this build's. The license is not one
-the third rule above permits bundling or curating. And the download job has no
-notion of an authenticated fetch - no hub token, no terms acceptance - so a gated
-revision is unreachable regardless of its license. That is the gate any future
-gated candidate would hit, recorded here rather than as current debt.
+```
+Sam3VideoConfig            model_type = sam3_video          <- what resolves
+|-- detector_config        model_type = sam3                <- concepts and words
++-- tracker_config         model_type = sam3_tracker_video  <- the promptable half
+```
+
+Only `sam3_video` belongs in `SEGMENTER_FAMILIES`. `sam3` is the concept detector,
+and admitting it would route a text model to the point adapter. A register built
+by reasoning from the `transformers` class names produces `sam3` and
+`sam3_tracker`, neither of which any published checkpoint declares, and the
+capability is then unreachable while every test agrees with the guess that built
+it. The rule this restates: a family entry is a string read out of a config, and
+the test that holds it names the configs it was read from.
 
 ## Related
 

docs/inference.md

@@ -92,6 +92,11 @@ to the plain file - keeps projects, connections and downloaded weights; only the
 and `--build` is not optional on any of those switches. That stack is for development and changes
 nothing about the wheel above.
 
+`HF_TOKEN` is forwarded from whoever runs `docker compose`, so fetching weights that have to be
+asked for (below) works the same way inside the stack as outside it - export it in the shell you
+bring the stack up in. It is empty when you have none, which is what every ungated model already
+expects.
+
 ## Knowing what a download costs, before agreeing to it
 
 A decision you cannot see the price of is not a decision, so the size is readable on its own,
@@ -115,6 +120,10 @@ publishing two serialisations of the same tensors really does cost both, and a f
 of them would understate what lands on your disk. A revision the listing cannot fully size is
 refused rather than estimated.
 
+`facebook/sam3` is the live example of that and the largest entry in the curated list: it publishes
+its weights as both a checkpoint and safetensors, so about 6.9 GB arrives to install a model of
+about 3.4 GB. The number shown is the one that describes your disk.
+
 It needs the `local-inference` extra, because the size is read with the same client that would do
 the fetching. Without it you get `LOCAL_INFERENCE_UNAVAILABLE` and the install command.
 
@@ -151,6 +160,26 @@ error on the job. There is no half-ready state to recover from because there is
 one could be written. Ask again: an interrupted transfer resumes from what it had, and each file
 that arrives is checked against the size the hub published for it before it is put in place.
 
+**Some models have to be asked for first.** A publisher can put its weights behind an access gate,
+where the files are served only to accounts that have been granted them. `facebook/sam3` is one:
+its weights are under Meta's SAM License and access is granted by request. Two steps, once per
+machine rather than once per connection - ask for access on the model's own page, then put a token
+from that account in the environment the server or the CLI runs in:
+
+```bash
+export HF_TOKEN=hf_…
+```
+
+The token is read by the hub client itself, so there is nothing to configure in a workspace and
+nothing stored in one. A download attempted without it fails with a sentence naming both halves of
+the remedy rather than an HTTP status. Reading a model's **size** needs no token and no access:
+that is what lets the connection form tell you what a download costs before you decide whether to
+go and ask for it.
+
+The gate is on fetching, never on running. Once the files are in a workspace's `models/`, the model
+loads and answers clicks with no token present at all - which is what makes a workspace you copy to
+another machine still work there.
+
 **Asking again checks rather than repeats.** `download_weights` stays available once a connection
 is `ready`, where the same call re-checks that the snapshot is still complete and fetches only what
 is missing - the browser labels it **Check for missing files** and puts it in the row's overflow
@@ -262,7 +291,7 @@ about to send? So a connection also declares what it can be asked for.
 
 | Capability | Means | Families |
 | --- | --- | --- |
-| `point_suggest` | Give me the thing under these points | the SAM 2 family |
+| `point_suggest` | Give me the thing under these points | the SAM 2 and SAM 3 families |
 | `text_detect` | Find everything these words name | the grounding-dino family |
 
 **Read from the model, never from its name.** The value comes from the `model_type` the
@@ -492,15 +521,18 @@ steps, because the two kinds share almost no fields: first where the model runs,
 form.
 
 - **Local** opens on a curated model, a `cpu` device and `fp32` precision. The model field is a
-  grouped list - the SAM 2.1 ladder under *Interactive segmentation*, Grounding DINO under
-  *Text-prompt detection* - showing each entry's download size and a line on what it is for, and
+  grouped list - the SAM 2.1 ladder and SAM 3 under *Interactive segmentation*, Grounding DINO
+  under *Text-prompt detection* - showing each entry's download size and a line on what it is for, and
   each one is pinned to a revision this build was checked against. **Custom model...** is the last
   entry and reveals the free model id and revision fields: the list guides, it does not restrict,
   and any model this build has an adapter for remains typeable. Device and precision are lists too,
   and the precision list follows the device, because half precision applies on CUDA only - so
   picking `mps` leaves `fp32` as the only precision offered. Underneath
   is what fetching that revision would cost - the size described above, read while you are still
-  deciding. If this machine has no `local-inference` extra the size cannot be read, and the form
+  deciding. An entry whose weights have to be asked for says so on the line above that one, with a
+  link to the page where access is requested, so the requirement is read while the model is being
+  chosen rather than met as a refused download later.
+  If this machine has no `local-inference` extra the size cannot be read, and the form
   says so, in the server's own words, with the install command. **It stays usable**: creating a
   connection downloads nothing, so not knowing the size is information rather than a barrier.
 - **HTTP** asks for the endpoint URL. There is no credential field; where a secret would live is

frontend/app/e2e/inference.spec.ts

@@ -365,3 +365,78 @@ test("a transfer and a re-read are two records on one row", async ({ page }) =>
   await expect(page.getByTestId("integrity-progress-prose")).toHaveText("3 of 9 files · 33%");
   await expect(page.getByTestId("download-progress")).toHaveCount(0);
 });
+
+test("a model whose weights have to be asked for says so before it can be downloaded", async ({
+  page,
+}) => {
+  // Here rather than only in `inference.test.tsx` because the claim is about what
+  // a person meets on the way to a download: the requirement has to be legible in
+  // the real dialog, above the real size line, before the control that would
+  // fetch anything exists. jsdom proves the conditional; this proves the journey.
+  await serveApi(page, () => connection("ready", null));
+  await page.route("**/api/inference/download-size*", (route) =>
+    route.fulfill({
+      status: 200,
+      json: {
+        model_id: "facebook/sam3",
+        model_revision: "3c879f39826c281e95690f02c7821c4de09afae7",
+        total_bytes: 6_895_093_624,
+        file_count: 12,
+      },
+    }),
+  );
+  await openInference(page);
+
+  await page.getByTestId("new-connection").click();
+  await page.getByTestId("choose-local").click();
+  // The form opens on a model anybody can fetch, so there is nothing to say yet.
+  await expect(page.getByTestId("model-access")).toHaveCount(0);
+
+  await page.getByTestId("connection-model").click();
+  await page.getByRole("option", { name: /facebook\/sam3/ }).click();
+
+  const access = page.getByTestId("model-access");
+  await expect(access).toBeVisible();
+  await expect(access).toContainText("SAM License");
+  await expect(access).toContainText("HF_TOKEN");
+  await expect(access.getByRole("link", { name: "Request access" })).toHaveAttribute(
+    "href",
+    "https://huggingface.co/facebook/sam3",
+  );
+});
+
+test("a model list taller than the window scrolls instead of running off it", async ({ page }) => {
+  // Layout under a real viewport, so it cannot live in `inference.test.tsx`:
+  // jsdom reports every height as zero and would pass against the implementation
+  // this replaced, which clipped the list and left the options past the bottom
+  // edge in the DOM, keyboard-reachable and unreachable with a pointer.
+  await serveApi(page, () => connection("ready", null));
+  await page.route("**/api/inference/download-size*", (route) =>
+    route.fulfill({
+      status: 200,
+      json: { model_id: "m", model_revision: "r", total_bytes: 1_200_000_000, file_count: 3 },
+    }),
+  );
+  // Short enough that the curated list cannot fit under its trigger whatever the
+  // catalog holds — the condition being tested, rather than a window size that
+  // happens to provoke it today.
+  await page.setViewportSize({ width: 1280, height: 600 });
+  await openInference(page);
+  await page.getByTestId("new-connection").click();
+  await page.getByTestId("choose-local").click();
+  await page.getByTestId("connection-model").click();
+
+  const listbox = page.getByRole("listbox");
+  await expect(listbox).toBeVisible();
+  const box = (await listbox.boundingBox())!;
+  expect(box.y + box.height).toBeLessThanOrEqual(600 + 1);
+
+  // The last entry is the one a clipped list loses, and it is reachable: Radix
+  // scrolls it into view, and clicking it selects it rather than hitting a
+  // control drawn past the bottom of the window.
+  const last = page.getByRole("option", { name: /Custom model/ });
+  await last.scrollIntoViewIfNeeded();
+  await expect(last).toBeVisible();
+  await last.click();
+  await expect(page.getByTestId("connection-custom-model")).toBeVisible();
+});

frontend/ui-core/src/annotator/SuggestPanel.tsx

@@ -179,9 +179,13 @@ const BLOCKER_COPY: Readonly<
   // Ranked *below* `not-ready` by `usableConnection`, and the copy relies on it:
   // this sentence is only ever read where something is downloaded and running,
   // so "the model you have" is a model that is genuinely here.
+  // The sentence names what the model has to *do* rather than which family does
+  // it. More than one architecture answers a point now, so a copy naming one of
+  // them was a second place the supported list lived — and the one that goes
+  // stale silently, because no build fails when prose falls behind a register.
   "not-capable": {
     title: "That model answers a different question",
-    body: "Suggesting a shape needs a model that takes points — the SAM 2 family. The connections that are ready here answer something else, so a click would come back refused.",
+    body: "Suggesting a shape needs a model that takes points and answers with a region. The connections that are ready here answer something else, so a click would come back refused.",
     action: "Set up a connection",
     tone: "warn",
   },