feat(inference): run local models on the Apple Silicon GPU

docs/cli.md

@@ -37,7 +37,7 @@ visionset token create --name NAME
 visionset token list
 visionset token revoke NAME [--yes]
 visionset inference create NAME --type local|http --model ID --revision REV
-                               [--device cpu|cuda|cuda:N] [--precision fp16|fp32]
+                               [--device cpu|mps|cuda|cuda:N] [--precision fp16|fp32]
                                [--endpoint URL]
 visionset inference list
 visionset inference show|update|delete NAME_OR_ID

docs/inference.md

@@ -35,7 +35,7 @@ with WorkspaceService.open("./road-signs") as workspace:
         model_id="some/model",
         model_revision="abc123",
         device="cuda",
-        precision="fp16",  # fp16 needs a cuda device; a cpu connection is fp32
+        precision="fp16",  # fp16 needs a cuda device; cpu and mps connections are fp32
     )
     for one in connections.list():
         print(one.name, one.connection_type.value, one.setup_state.value)
@@ -209,24 +209,46 @@ nor the family, and neither does sending the same model reference back unchanged
 An `http` connection keeps no weights here, so a model edit resets nothing for it. It stays
 `ready`, which for that kind has always meant *there is nothing to set up on this machine*.
 
-## Running on the CPU
+## Which device runs the model
 
-A connection asking for `cuda` on a machine with no GPU falls back to the CPU, in full precision,
-with a warning in the log. It is a fallback rather than a preference - a workspace configured on a
-workstation should still open on a laptop - but it is slower by a large factor, which is why it is
-said out loud rather than silently done.
+A local connection names the device it runs on, and there are three to name.
 
-Half precision applies on CUDA only, and the kernel now says so rather than absorbing it: a `cpu`
-connection asking for `fp16` is refused at creation. On a CPU it was never the conservative choice
-it looks like - `float16` arithmetic outside CUDA's autocast is slower than the `float32` it was
-avoiding - and a setting the adapters drop is one the row would otherwise go on displaying as
-though it had an effect.
-
-**Both fields are closed vocabularies.** `device` is `cpu`, `cuda`, or `cuda:N` for the second GPU
-on a machine that has one; `precision` is `fp16` or `fp32`, and `float16`, `half`, `float32` and
-`full` are accepted as spellings of those two. Anything else is refused with a sentence naming the
-members. What this closes is a gap rather than a freedom: `gpu` used to be accepted and then
-resolved onto the CPU, so the connection described a run that never happened.
+| Device | What it is | Precision |
+| --- | --- | --- |
+| `cpu` | The processor. Every machine has one, and it is the default a new connection opens on | `fp32` |
+| `cuda` | An NVIDIA GPU. A machine with more than one addresses the rest as `cuda:1`, `cuda:2` and so on | `fp16` or `fp32` |
+| `mps` | Apple Silicon's GPU, on an M-series Mac. There is only ever one of it | `fp32` |
+
+**On Apple Silicon nothing needs configuring beyond choosing the device.** The `local-inference`
+extra is the same one everybody installs, the macOS wheels it brings carry Metal support already,
+and there is no second package index, no environment variable and no build flag. Create the
+connection with `mps` and it runs on the GPU.
+
+**A device this machine does not offer falls back to the CPU**, in full precision, with a warning
+in the log naming the connection and the device it asked for. The same rule covers all three, so
+`mps` on a machine with no Metal behaves exactly as `cuda` on a machine with no NVIDIA GPU. It is a
+fallback rather than a preference - a workspace configured on a workstation should still open on a
+laptop - but it is slower by a large factor, which is why it is said out loud rather than silently
+done.
+
+**Half precision applies on CUDA only**, and the kernel says so rather than absorbing it: a `cpu`
+or `mps` connection asking for `fp16` is refused at creation. On a CPU it was never the
+conservative choice it looks like - `float16` arithmetic outside CUDA's autocast is slower than the
+`float32` it was avoiding - and Metal has no float64 at all with a bfloat16 that varies between
+releases, so full precision is the only format that behaves the same on every Mac. A setting the
+adapters would drop is one the row would otherwise go on displaying as though it had an effect.
+
+Where Metal has no implementation for an operator a model reaches for, that one operator runs on
+the CPU and the rest of the forward pass stays on the GPU. Nothing has to be turned on for this;
+the adapters ask for it themselves.
+
+**Both fields are closed vocabularies.** `device` is `cpu`, `mps`, `cuda`, or `cuda:N` for the
+second GPU on a machine that has one; `precision` is `fp16` or `fp32`, and `float16`, `half`,
+`float32` and `full` are accepted as spellings of those two. Anything else is refused with a
+sentence naming the members. What this closes is a gap rather than a freedom: `gpu` used to be
+accepted and then resolved onto the CPU, so the connection described a run that never happened.
+A device is in the vocabulary when the adapters can honour it, which is why `mps` is in it and
+`gpu` and `auto` are not.
 
 ## What a connection can be asked for
 
@@ -467,7 +489,8 @@ form.
   each one is pinned to a revision this build was checked against. **Custom model...** is the last
   entry and reveals the free model id and revision fields: the list guides, it does not restrict,
   and any model this build has an adapter for remains typeable. Device and precision are lists too,
-  and the precision list follows the device, because half precision applies on CUDA only. Underneath
+  and the precision list follows the device, because half precision applies on CUDA only - so
+  picking `mps` leaves `fp32` as the only precision offered. Underneath
   is what fetching that revision would cost - the size described above, read while you are still
   deciding. If this machine has no `local-inference` extra the size cannot be read, and the form
   says so, in the server's own words, with the install command. **It stays usable**: creating a
@@ -516,7 +539,7 @@ downloaded, or nothing of the right kind - and each names a different thing to d
 visionset inference size some/model --revision abc123
 visionset inference create local-detector \
     --type local --model some/model --revision abc123 --device cuda --precision fp16
-# --device takes cpu, cuda or cuda:N; --precision takes fp16 or fp32, and fp16 needs a cuda device
+# --device takes cpu, mps, cuda or cuda:N; --precision takes fp16 or fp32, and fp16 needs a cuda device
 visionset inference list
 visionset inference show local-detector --json
 visionset inference update local-detector --revision def456

docs/install.md

@@ -79,7 +79,9 @@ pip install "visionset[local-inference]"
 ```
 
 That brings torch, torchvision, transformers, accelerate and huggingface_hub - roughly two
-gigabytes, most of it CUDA - which is exactly why it is not in the base install. Without it you can still create a
+gigabytes, most of it CUDA - which is exactly why it is not in the base install. It is the same
+command on every platform: the macOS wheels it installs carry Apple Silicon GPU support already,
+so a Mac needs no second index and no build flag to run a connection on `mps`. Without it you can still create a
 local connection, list it, and see what it is configured for; what you cannot do is fetch its
 weights or ask it to predict. Both refusals name the command above rather than saying
 "unavailable", the way a missing `ffmpeg` does.

frontend/ui-core/src/screens/inferenceCatalog.test.ts

@@ -67,17 +67,24 @@ it("offers half precision on CUDA and on every address of it", () => {
   // A second GPU is still a GPU. This is the kernel's `precisions_for`, and the
   // two answer the same way or the form offers what the kernel refuses.
   expect(precisionsFor("cuda:1")).toEqual(["fp16", "fp32"]);
+  // Metal has no float64 and an inconsistent bfloat16, so full precision is the
+  // only format that behaves the same on every Mac — and the kernel refuses the
+  // pairing at creation, which a form still offering it would walk straight into.
+  expect(precisionsFor("mps")).toEqual(["fp32"]);
 });
 
 it("keeps a precision that survives a device change and replaces one that does not", () => {
   expect(precisionOn("cuda", "fp32")).toBe("fp32");
   expect(precisionOn("cpu", "fp32")).toBe("fp32");
   expect(precisionOn("cpu", "fp16")).toBe("fp32");
+  // Moving a half-precision CUDA connection onto Metal cannot keep the setting.
+  expect(precisionOn("mps", "fp16")).toBe("fp32");
 });
 
-it("offers the two devices every machine can be asked about", () => {
+it("offers the devices every machine can be asked about", () => {
   // `cuda:N` is deliberately absent: how many GPUs this machine has is not
   // something a static list can know, so it is typed by the kernel's pattern and
-  // shown by the form only when a row already carries one.
-  expect([...DEVICES]).toEqual(["cpu", "cuda"]);
+  // shown by the form only when a row already carries one. `mps` needs no such
+  // escape, because a Mac has exactly one.
+  expect([...DEVICES]).toEqual(["cpu", "cuda", "mps"]);
 });

frontend/ui-core/src/screens/inferenceCatalog.ts

@@ -161,18 +161,28 @@ export function curatedEntry(modelId: string, revision: string): CuratedModel |
   return found !== undefined && found.revision === revision ? found : undefined;
 }
 
-/** The devices a form offers, in the order it offers them. */
-export const DEVICES = ["cpu", "cuda"] as const;
+/**
+ * The devices a form offers, in the order it offers them.
+ *
+ * `mps` is Apple Silicon's GPU, and it is one entry rather than a platform
+ * branch: which devices a machine actually has is answered where the model is
+ * loaded, not by a form guessing from a user agent.
+ */
+export const DEVICES = ["cpu", "cuda", "mps"] as const;
 
 /**
  * The precisions that are honoured on that device — the kernel's
  * `precisions_for`, offering-side.
  *
  * Half precision is CUDA-only in both local adapters, so `cpu` + `fp16` is not a
  * slower run but a setting with no effect that the row would go on displaying as
- * though it had one. A machine addressing a second GPU writes `cuda:1`, which is
- * not a member of {@link DEVICES} and is still a CUDA device — hence the prefix
- * test rather than an equality against `"cuda"`.
+ * though it had one. `mps` answers the same way and for its own reason: Metal has
+ * no float64 and an inconsistent bfloat16, so full precision is the only format
+ * that behaves the same on every machine offering the device. A machine
+ * addressing a second GPU writes `cuda:1`, which is not a member of
+ * {@link DEVICES} and is still a CUDA device — hence the prefix test rather than
+ * an equality against `"cuda"`, which is also what leaves every non-CUDA device
+ * on `fp32` without naming each one.
  */
 export function precisionsFor(device: string): readonly Precision[] {
   return device.startsWith("cuda") ? ["fp16", "fp32"] : ["fp32"];

src/visionset/cli/inference.py

@@ -76,11 +76,13 @@ def inference_create(
     ],
     device: Annotated[
         str | None,
-        typer.Option("--device", help="Local only. cpu, cuda, or cuda:N for a second GPU."),
+        typer.Option("--device", help="Local only. cpu, mps, cuda, or cuda:N for a second GPU."),
     ] = None,
     precision: Annotated[
         Precision | None,
-        typer.Option("--precision", help="Local only. fp16 needs a cuda device."),
+        typer.Option(
+            "--precision", help="Local only. fp16 needs a cuda device; cpu and mps run in fp32."
+        ),
     ] = None,
     endpoint_url: Annotated[
         str | None, typer.Option("--endpoint", help="HTTP only. Where to send predictions.")
@@ -134,11 +136,13 @@ def inference_update(
     model_revision: Annotated[str | None, typer.Option("--revision", help="Move the pin.")] = None,
     device: Annotated[
         str | None,
-        typer.Option("--device", help="Local only. cpu, cuda, or cuda:N for a second GPU."),
+        typer.Option("--device", help="Local only. cpu, mps, cuda, or cuda:N for a second GPU."),
     ] = None,
     precision: Annotated[
         Precision | None,
-        typer.Option("--precision", help="Local only. fp16 needs a cuda device."),
+        typer.Option(
+            "--precision", help="Local only. fp16 needs a cuda device; cpu and mps run in fp32."
+        ),
     ] = None,
     endpoint_url: Annotated[str | None, typer.Option("--endpoint", help="HTTP only.")] = None,
     json_out: JsonOption = False,

src/visionset/inference/__init__.py

@@ -31,6 +31,17 @@
 connection may hold a detector that answers words or a segmenter that answers
 places and those are not interchangeable.
 
+**One environment variable is set as this module is read**, and it is the only
+side effect importing this package has. ``PYTORCH_ENABLE_MPS_FALLBACK`` is what
+lets an operator Metal has not implemented run on the CPU instead of raising, and
+the array library reads it while it initialises rather than when such an operator
+is reached — so by the time a connection has been resolved to a device it is
+already too late to set. This module is the earliest place that is certain to be
+read before torch is imported anywhere in the package, which is what makes it the
+right place despite the setting having nothing to do with composition. It costs
+one dictionary write on every machine, does nothing at all on a machine with no
+Metal, and ``setdefault`` leaves an operator who set it to ``0`` alone.
+
 **What each surface reaches for.** ``fetch_weights`` is the download,
 ``check_integrity`` is the full re-read that tells damage from completeness,
 ``suggest`` is one click's worth of interactive segmentation, and ``provider_for``
@@ -41,6 +52,7 @@
 
 from __future__ import annotations
 
+from visionset.inference._device import MPS_FALLBACK_VARIABLE, enable_mps_fallback
 from visionset.inference._extra import EXTRA, INSTALL_COMMAND, MODULES, require
 from visionset.inference.cache import (
     DEFAULT_EMBEDDING_CAPACITY,
@@ -97,8 +109,12 @@
     with_families,
 )
 
+enable_mps_fallback()
+
 __all__ = [
     "EPSILON",
+    "MPS_FALLBACK_VARIABLE",
+    "enable_mps_fallback",
     "DEFAULT_EMBEDDING_CAPACITY",
     "DEFAULT_IOU_THRESHOLD",
     "DEFAULT_PROVIDER_CAPACITY",