Add SAM 3 as an interactive point-segmentation provider alongside SAM 2

[JArmandoAnaya]

VisionSet's interactive auto-labeling turns a click in the annotator into a shape. A person
creates an inference connection naming a model and a pinned revision, downloads its weights, and
the suggest tool sends the points they click and draws back what the model proposes. Today the
only checkpoint family that answers a point prompt is SAM 2.

This adds SAM 3 (facebook/sam3) as a second driver of the point-prompt capability that already
exists. It is not a new capability: ModelCapability.POINT_SUGGEST is unchanged, the dashboard
gains no section, and the annotator gains no control. A connection pointed at SAM 3 becomes a
candidate for the suggest tool on exactly the same terms as a SAM 2 one.

Scope

Point-prompt inference only. SAM 3's open-vocabulary text and concept segmentation is deliberately
out of scope and is not designed here; the text-prompt capability the product already declares
(text_detect) is untouched, and so is the text-prompt adapter behind it. SAM 3.1 is out of scope:
it is a checkpoint-only release with no library integration to build against.

What the work is

The resolver reads a model's family from its own downloaded config and maps that family to a
capability, so registering a new driver is an entry in the segmenter family set rather than a new
adapter class. The point-prompt path in the library is Sam3TrackerModel with
Sam3TrackerProcessor, whose forward signature, encode method and mask post-processing are
identical to SAM 2's — which is why the existing local segmenter adapter serves both and only its
model-loading lines change.

Two things are genuinely new. The repository is gated: weights download only after a person has
requested access and authenticated, and the failure that arrives when they have not is currently
re-raised carrying the hub's raw HTTP body. That refusal has to become one sentence saying what
happened and one saying what to do, on the error contract every other refusal follows. And the
model is large enough that the size has to be visible before anyone commits to fetching it.

Licensing

The adapter code is VisionSet's own and ships in the Apache-2.0 core. The weights are under Meta's
SAM License and are downloaded by the user directly from Hugging Face after the user accepts the
gated-access terms; VisionSet never redistributes them.

Acceptance

• A connection pointed at facebook/sam3 downloads, reports the point-prompt capability, and
  answers a click in the annotator with a shape.
• Attempting the download without approved access produces a refusal naming the remedy, with no
  HTTP status code and no traceback in the text a person reads.
• Weights already in the local cache run with no token present: the gate is on download, never on
  execution.
• The access requirement is visible in the connection form before any action is taken.
• SAM 2 and the text-prompt adapter behave exactly as they did.

[JArmandoAnaya]

Decision (Armando, 2026-08-15): SAM 3 (facebook/sam3) joins VisionSet as an additional
driver of the existing interactive point-segmentation capability, alongside SAM 2. Scope
is point-prompt inference only — SAM 3's open-vocabulary text/concept segmentation is
deliberately out of scope and is not designed here; a future issue may revisit it.
Grounding DINO is untouched. SAM 3.1 is out of scope (checkpoint-only release, no
Transformers integration).

License treatment ratified: the SAM 3 adapter ships in the Apache-2.0 core. Our adapter
code is our own; the SAM-licensed weights are downloaded by the user directly from
Hugging Face after the user accepts Meta's gated-access terms — VisionSet never
redistributes them. This is the same pattern as any gated-model integration and does not
require the separate-plugin treatment reserved for AGPL models.

No fallback between models: if SAM 3 fails at runtime, the refusal names what happened
and may suggest SAM 2 — it never silently substitutes it.

— supersedes any prior direction

[JArmandoAnaya]

Measurements taken against transformers 5.14.1 — the version this repository's lockfile already
resolves — and against the hub's own metadata for facebook/sam3. They are recorded here because
three of them contradict what the task was written against, and the last two change what the work
has to do.

The point-prompt path is the tracker classes, not Sam3Model

Sam3Model and Sam3Processor are the text and concept path — the capability this issue puts out
of scope. Point prompts go through Sam3TrackerModel and Sam3TrackerProcessor. The library's own
auto-mapping is unambiguous about which is which:

CONFIG_MAPPING_NAMES:     sam3 -> Sam3Config              sam3_tracker -> Sam3TrackerConfig
PROCESSOR_MAPPING_NAMES:  sam3 -> Sam3Processor           sam3_tracker -> Sam3TrackerProcessor
MODEL_MAPPING_NAMES:      sam3 -> Sam3Model               sam3_tracker -> Sam3TrackerModel

The consequence is that AutoProcessor cannot be used for this adapter. The existing local
segmenter adapter loads its processor through AutoProcessor.from_pretrained, which resolves off
the repository's declared config type; if facebook/sam3 declares sam3, that call returns the
text processor, which does not accept input_points. The adapter has to name the tracker classes
explicitly.

The tracker is a straight port of SAM 2's image path

Every signature the adapter depends on is identical between Sam2Model and Sam3TrackerModel,
compared in the installed source rather than inferred from documentation:

What the adapter calls	SAM 2	SAM 3 tracker
get_image_embeddings(pixel_values, **kwargs) -> list[Tensor]	present	identical
forward(pixel_values, input_points, input_labels, input_boxes, input_masks, image_embeddings, multimask_output, …)	present	identical
post_process_masks(masks, original_sizes, mask_threshold, binarize, …)	present	identical
processor __call__ accepting original_sizes without images	yes	yes
outputs carrying pred_masks and iou_scores	yes	yes

This is what keeps the change small: the embedding cache, the decode path, the fp16 guard and the
mask-to-shape pipeline are all untouched, and only the two class names in the loader move.

The documentation page for the tracker lists get_image_features and its prose mentions
get_image_embeddings, which read as a contradiction. Both methods exist on both classes; the one
the adapter already calls is present and unchanged.

Reading the download size does not need access, and the size is 6.9 GB

model_info answers for a gated repository without a token. The size line in the connection form
therefore works before anyone has requested access, which is the right way round — a person can see
what a download costs before deciding to clear the gate for it.

The revision is 3c879f39826c281e95690f02c7821c4de09afae7, and the repository is 12 files totalling
6,895,093,624 bytes. That total deserves attention, because it is not the size of the model:

3,450,062,241  sam3.pt
3,439,938,512  model.safetensors
    3,642,073  tokenizer.json
      862,328  vocab.json
      524,619  merges.txt
       ... and 7 smaller files

The weights are published twice, once as a checkpoint and once as safetensors, and weight
fetching downloads every file in the revision. A SAM 3 connection therefore transfers about 6.9 GB
to use about 3.4 GB of it. That is a property of how the repository is published rather than a
defect here, but it is the number a person has to be shown, and it is roughly ten times the
balanced SAM 2 rung.

The gated failure arrives as a 401 carrying an HTTP body

Unauthenticated access raises GatedRepoError, whose text begins with a status line and a request
id:

401 Client Error. (Request ID: Root=1-6a8074a4-3bc3584f53a6fc7333db2091;f55535d6-…)

Cannot access gated repo for url https://huggingface.co/facebook/sam3/resolve/main/config.json.
Access to model facebook/sam3 is restricted. You must have access to it and be authenticated to access it. Please log in.

Weight fetching catches the base exception and re-raises it with that text interpolated, so today
this reaches a person verbatim. Replacing it is the one behavioural fix in this issue.

Two details matter when writing the translation. The status is 401, not the 403 a gated
repository is usually assumed to answer with. And GatedRepoError inherits from
RepositoryNotFoundError, so a handler that tests for the parent first silently swallows the gated
case and reports a missing repository instead — the same ordering trap the media adapter documents
for UnidentifiedImageError under OSError, and the persistence adapter documents for
IntegrityError under DatabaseError.

Still unmeasured

What facebook/sam3's config.json declares as its model_type is behind the gate, and it decides
which string the segmenter family set gains. The family is read from a model's own config rather
than guessed from its name, deliberately, so this one is left open rather than assumed until the
weights can be fetched.