From b04ee4fb361dd4f35346ccc47d093b0ee7e04c02 Mon Sep 17 00:00:00 2001 From: pratikbin <68642400+pratikbin@users.noreply.github.com> Date: Fri, 28 Aug 2026 14:41:57 +0530 Subject: [PATCH] feat(sandbox): thin cos onto createos-cli's offload/matrix/fork --count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit scripts/cos's offload and fanout bodies (stage, push, exec, keepalive, pull, destroy — one copy per verb) are replaced with thin delegates to `createos sandbox offload` and `createos sandbox matrix`, which already do this composition in Go with retry, no billable leaks on failure, and a teardown failure that fails the command instead of hiding it. - `cos offload` -> `createos sandbox offload -- ` - `cos fanout` -> `createos sandbox matrix --job ... --job ...` (no --prepare) — a fork of one staged box per job, not an independently-staged box per job. Same isolation, faster. - new `cos matrix` / `/createos-sandbox:matrix` — fanout plus -P '' to run once on the golden box before forking, for the case every job shares the same dependency install or toolchain prep. - `cos fork` keeps its own pause-then-resume orchestration around the project box (legitimate: it is forking a box it owns, on request), but the fork step itself and id resolution now delegate to `createos sandbox fork -o json`, which returns the new id directly. This deletes the comm-diff-against-`sandbox ls` hack and the documented multibyte/set -u trap that came with it. Gained -c N for direct N-way cloning of the project box. - -w/--swap dropped from offload (per discussion): devbox:1 can't swapon a file added post-boot anyway, so the flag bought little. Documented as something to compose into the command string instead. Docs updated throughout (skill, references, command frontmatter, README) for the new/changed verbs and to correct a stale "2 concurrent boxes" quota claim to the observed number (10). Verified live via tmux against createos-cli's feat/sandbox-compositions branch (COS_CLI override), running five scenarios pulled from the product's own use-case catalog: untrusted-code offload with locked egress, independent-job fanout, shared-setup matrix (-P ran exactly once across all forks), project-box fork -c N, and a background job ending its own box via the loopback self-signal endpoint. That live run caught a real bug in createos-cli: `fork --count 2` silently created one clone instead of two, with no error — Go's stdlib flag parser stops at the first non-flag argument, so --count written after the sandbox id (the order every caller actually writes, cos included) was never parsed at all. Fixed on the createos-cli side (feat/sandbox-compositions, reusing the raw-argv fallback already built for `process run --cwd`) and confirmed fixed here before writing this up. --- .../.claude-plugin/plugin.json | 4 +- packages/claude-code-plugin/README.md | 72 ++-- .../claude-code-plugin/commands/cluster.md | 2 +- .../claude-code-plugin/commands/fanout.md | 4 +- packages/claude-code-plugin/commands/fork.md | 9 +- .../claude-code-plugin/commands/matrix.md | 11 + .../claude-code-plugin/commands/offload.md | 6 +- packages/claude-code-plugin/scripts/cos | 324 +++++++++--------- .../skills/using-createos-sandbox/SKILL.md | 23 +- .../references/lifecycle-and-images.md | 19 +- .../references/offload-and-egress.md | 25 +- 11 files changed, 290 insertions(+), 209 deletions(-) create mode 100644 packages/claude-code-plugin/commands/matrix.md diff --git a/packages/claude-code-plugin/.claude-plugin/plugin.json b/packages/claude-code-plugin/.claude-plugin/plugin.json index 762adbe..41fa76b 100644 --- a/packages/claude-code-plugin/.claude-plugin/plugin.json +++ b/packages/claude-code-plugin/.claude-plugin/plugin.json @@ -2,8 +2,8 @@ "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", "name": "@createos/claude-code", "displayName": "CreateOS Sandbox", - "version": "0.6.0", - "description": "Run ad-hoc, heavy, or untrusted code OFF your machine in disposable CreateOS Sandboxes. One-shot offload, parallel fanout across N boxes, instant scratch shell, a reusable box with file sync for live dev loops, port tunnel to localhost, public HTTPS expose, multi-box private-network clusters, BYO-S3 disk mounts, WireGuard VPN, snapshot/fork, pause/resume to park a warm box at zero compute cost, and custom Dockerfile-built images so boxes boot pre-provisioned. Big dirs auto-excluded from uploads; auto-installs the createos CLI if missing.", + "version": "0.7.0", + "description": "Run ad-hoc, heavy, or untrusted code OFF your machine in disposable CreateOS Sandboxes. One-shot offload, parallel fanout across N forks of one staged box, matrix (shared-setup fanout), instant scratch shell, a reusable box with file sync for live dev loops, port tunnel to localhost, public HTTPS expose, multi-box private-network clusters, BYO-S3 disk mounts, WireGuard VPN, snapshot/fork (N at once), pause/resume to park a warm box at zero compute cost, and custom Dockerfile-built images so boxes boot pre-provisioned. Big dirs auto-excluded from uploads; auto-installs the createos CLI if missing.", "author": { "name": "NodeOps", "url": "https://createos.sh" }, "homepage": "https://createos.sh", "keywords": ["sandbox", "createos", "firecracker", "remote-exec", "isolation", "microvm"] diff --git a/packages/claude-code-plugin/README.md b/packages/claude-code-plugin/README.md index 3af0685..18ce16f 100644 --- a/packages/claude-code-plugin/README.md +++ b/packages/claude-code-plugin/README.md @@ -4,11 +4,11 @@ **Run ad-hoc, heavy, or untrusted code off your machine — from inside Claude Code.** -A [Claude Code](https://docs.claude.com/en/docs/claude-code) plugin that gives Claude a skill + 18 slash commands driving the authed [`createos`](https://createos.sh) CLI. Work runs in disposable [CreateOS](https://createos.sh) Sandboxes — roughly 200 ms from create to your first command — that self-destruct when done. +A [Claude Code](https://docs.claude.com/en/docs/claude-code) plugin that gives Claude a skill + 19 slash commands driving the authed [`createos`](https://createos.sh) CLI. Work runs in disposable [CreateOS](https://createos.sh) Sandboxes — roughly 200 ms from create to your first command — that self-destruct when done. [![Claude Code](https://img.shields.io/badge/Claude%20Code-plugin-6E56CF)](https://docs.claude.com/en/docs/claude-code) [![CreateOS](https://img.shields.io/badge/CreateOS-Sandboxes-0EA5E9)](https://createos.sh) -[![Version](https://img.shields.io/badge/version-0.5.0-blue)](./.claude-plugin/plugin.json) +[![Version](https://img.shields.io/badge/version-0.7.0-blue)](./.claude-plugin/plugin.json) @@ -24,6 +24,7 @@ A [Claude Code](https://docs.claude.com/en/docs/claude-code) plugin that gives C - [Command reference](#command-reference) - [Offload — one-shot](#offload--one-shot) - [Fanout — parallel boxes](#fanout--parallel-boxes) + - [Matrix — shared setup, parallel boxes](#matrix--shared-setup-parallel-boxes) - [Shell — throwaway Linux](#shell--throwaway-linux) - [Project box — live sessions](#project-box--live-sessions) - [Networking](#networking) @@ -69,7 +70,7 @@ The plugin is a **thin Claude-facing surface** over the `createos` CLI. It ships | Piece | Path | Role | | ------------------ | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | -| **Slash commands** | `commands/*.md` | 18 commands (`offload`, `fanout`, `shell`, …), each a thin wrapper that calls `scripts/cos` | +| **Slash commands** | `commands/*.md` | 19 commands (`offload`, `fanout`, `matrix`, `shell`, …), each a thin wrapper that calls `scripts/cos` | | **Skill** | `skills/using-createos-sandbox/SKILL.md` + `references/` | teaches Claude _when_ to reach for the sandbox on its own, with depth loaded on demand | | **Hooks** | `hooks/hooks.json` + `scripts/` | `SessionStart` publishes the driver's absolute path; `PreToolUse(Bash)` nudges on heavy build/test commands | | **Driver** | `scripts/cos` | the actual logic — staging, egress, keepalive, sync, networking, lifecycle, state | @@ -123,8 +124,9 @@ claude --plugin-dir /path/to/createos-claude-plugins/createos-sandbox | Command | Summary | | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | -| [`offload`](#offload--one-shot) `[flags] ` | one-shot: stage → run (keepalive) → pull → destroy | -| [`fanout`](#fanout--parallel-boxes) `[-j N] [flags] [cmd2] …` | run each command in its own throwaway box, in parallel | +| [`offload`](#offload--one-shot) `[flags] ` | one-shot: stage → run (keepalive) → fetch → destroy | +| [`fanout`](#fanout--parallel-boxes) `[-j N] [flags] [cmd2] …` | fork one staged box once per command, run each in parallel | +| [`matrix`](#matrix--shared-setup-parallel-boxes) `[-P setup] [-F box] [-j N] [flags] …` | fanout, but a setup command runs once before forking | | [`shell`](#shell--throwaway-linux) `[-s] [-r] [-e\|-p\|-E]` | instant throwaway interactive Linux (destroyed on exit) | | [`up`](#project-box--live-sessions) `[-s] [-r] [-n] [-e\|-p\|-E]` | create/reuse the per-repo project box | | [`run`](#project-box--live-sessions) `` | exec in the project box (streamed, state persists) | @@ -135,28 +137,27 @@ claude --plugin-dir /path/to/createos-claude-plugins/createos-sandbox | [`cluster`](#networking) `up \| run […] \| ls \| down` | N boxes on one private network, name-addressable | | [`disk`](#disks--byo-s3) `create \| ls \| show \| attach \| detach \| rm` | BYO S3 bucket mounts on the project box | | [`vpn`](#networking) `[register \| up]` | WireGuard L3 into your private networks | -| [`fork`](#networking) | snapshot the project box → independent clone | +| [`fork`](#networking) `[-c N]` | snapshot the project box → N independent clone(s) | | [`pause`](#pause-resume-and-custom-images) · [`resume`](#pause-resume-and-custom-images) | park the warm project box at zero compute cost / restore it exactly | | [`template`](#pause-resume-and-custom-images) `submit [-f Dockerfile] \| ls \| show \| logs \| rm` | build a custom rootfs so boxes boot pre-provisioned | | [`down`](#project-box--live-sessions) | stop sync/tunnels + destroy the project box (+ cluster) | | [`status`](#project-box--live-sessions) | show active box + sync + tunnels + cluster | -> **Flag order:** flags (`-s/-r/-e/-p/-E/-o/-x/-w/-K`) come **before** the positional ` `. +> **Flag order:** flags (`-s/-r/-e/-p/-E/-o/-x/-K`) come **before** the positional ` `. ### Offload — one-shot -The core command. Stages a directory into a fresh box, runs a command, optionally pulls artifacts back, then destroys the box. +The core command. Stages a directory into a fresh box, runs a command, optionally fetches artifacts back, then destroys the box. ``` -/createos-sandbox:offload [-p preset] [-e dom] [-E] [-x glob] [-o out] [-w GB] [-K] [-s shape] [-r rootfs] +/createos-sandbox:offload [-p preset] [-e dom] [-E] [-x glob] [-o out] [-K] [-s shape] [-r rootfs] ``` | Flag | Meaning | | ------------- | ------------------------------------------------------------------------------------------- | | `-s ` | box size (default `s-1vcpu-1gb`; see [Shapes](#shapes)) | | `-r ` | root filesystem image (default `devbox:1`) | -| `-o ` | tar a box-side dir and pull it back to local | -| `-w ` | attempt swap (see caveat under [Heavy builds](#heavy-builds)) | +| `-o ` | tar a box-side dir and fetch it back to local | | `-K` | keep the box on a real failure so you can inspect it | | `-e ` | allow one egress domain (repeatable) | | `-p ` | egress preset — `python-uv \| rust-cargo \| npm \| github` (repeatable, composes with `-e`) | @@ -167,27 +168,46 @@ The core command. Stages a directory into a fresh box, runs a command, optionall # Run a test suite, pull nothing, box auto-destroys /createos-sandbox:offload . "npm ci && npm test" -# Python build with locked egress, pull the dist/ folder back +# Python build with locked egress, fetch the dist/ folder back /createos-sandbox:offload -p python-uv -o dist . "uv sync --frozen && uv run python -m build" ``` -**Keepalive:** long or quiet compiles no longer die to exec-stream idle resets — the command runs detached with a heartbeat and re-attaches if the stream drops, so the build (and its cache) survives. +**Keepalive:** long or quiet compiles no longer die to exec-stream idle resets — the command runs detached with a heartbeat and re-attaches if the stream drops, so the build (and its cache) survives. **Swap:** there is no flag for it — compose it into the command itself: `'fallocate -l 4G /swapfile && mkswap /swapfile && swapon /swapfile && '`. ### Fanout — parallel boxes -Run each command in its **own** throwaway box, in parallel, then collect results. Ideal for splitting a test suite or a version matrix. +Stage `` once, fork it once per command, run each fork in parallel, then collect results. Ideal for splitting a test suite or a version matrix with no shared setup step. ``` /createos-sandbox:fanout [-j N] [-p preset] [-s shape] [-r rootfs] [-e dom] [-E] [-x glob] [cmd2] … ``` -- `-j N` — max concurrent boxes (**default 2**, the external-key running-quota). Other flags mirror `offload`. -- Exit codes are aggregated across boxes; each box is fully isolated from the others. +- `-j N` — max concurrent forks (**default 10**, matching this account's observed running quota). Other flags mirror `offload`. +- Exit codes are aggregated across jobs; each fork is fully isolated from the others. ```bash -/createos-sandbox:fanout -j 2 -p python-uv . "pytest -q tests/a" "pytest -q tests/b" "pytest -q tests/c" +/createos-sandbox:fanout -p python-uv . "pytest -q tests/a" "pytest -q tests/b" "pytest -q tests/c" ``` +### Matrix — shared setup, parallel boxes + +`fanout`, plus one thing: `-P ''` runs once on the golden box before it forks. Reach for this the moment two or more jobs would otherwise repeat the same install or toolchain prep. + +``` +/createos-sandbox:matrix [-P setup] [-F box] [-j N] [-p preset] [-s shape] [-r rootfs] [-e dom] [-E] [-x glob] [cmd2] … +``` + +- `-P ''` — run once on the golden box before forking. Paid once, not once per job. +- `-F ` — fork from a sandbox you already prepared and paused yourself, instead of building one from ``. +- `-G` — keep the golden box after the run instead of destroying it. +- `-L ` — directory for per-job log files. + +```bash +/createos-sandbox:matrix -P "npm ci" . "npm test -- --shard 1" "npm test -- --shard 2" "npm test -- --shard 3" +``` + +Two known limits, both from `createos sandbox matrix --help`: a fork does not carry the golden box's S3 disk attachments, and a clone whose snapshot isn't cached on the target host can take 11–13 s to resume rather than the usual sub-second. + ### Shell — throwaway Linux An instant, interactive Linux box that is **destroyed on exit**. @@ -232,9 +252,9 @@ A **reusable, per-repo** box addressed by your working directory. `up` creates i | **`tunnel [local]`** | Reach a box-side service on your laptop. Run a dev server in the box, then `tunnel 3000` → `http://127.0.0.1:3000`. Private, background, no public URL. Stopped by `down`. | | **`expose `** | A **public HTTPS** link for a port — `-.app.sb.createos.sh`, stable for the box's lifetime. The service must bind `0.0.0.0`. **Anyone with the link can reach it.** Revoke with `unexpose`. | | **`unexpose`** | Revoke the public URL / disable ingress on the active box. | -| **`cluster up `** | N boxes on one private network, reaching each other by **fully-qualified** name (`curl http://cos-cl--2.fc.local:8080` — the bare short name is NXDOMAIN). `cluster run -a ''` fans a command across all; `cluster run ''` targets one. `cluster ls` / `cluster down` manage them. For distributed-system / DB-replication / p2p / load-test repros. **Counts against quota — keep N small.** | +| **`cluster up `** | N boxes on one private network, reaching each other by **fully-qualified** name (`curl http://cos-cl--2.fc.local:8080` — the bare short name is NXDOMAIN). `cluster run -a ''` fans a command across all; `cluster run ''` targets one. `cluster ls` / `cluster down` manage them. For distributed-system / DB-replication / p2p / load-test repros. **Counts against the running quota — keep N small.** | | **`vpn register `** then **`vpn up`** | Join your laptop to the whole private network over WireGuard (reach every sandbox by name/IP). `vpn up` needs `wg-quick` + `sudo` and **blocks until Ctrl-C** — run it in your own terminal (`!cos vpn up`). | -| **`fork`** | Snapshot the warm project box → an independent clone for matrix/parallel experiments. The fork is self-managed. | +| **`fork [-c N]`** | Snapshot the warm project box → N independent clones (default 1) for branching experiments. Each fork is self-managed. Running the same job N ways from a directory is usually a better fit for `matrix`. | ### Pause, resume, and custom images @@ -308,13 +328,13 @@ So a domain allowlist is the right control for "this build should only reach pyp - **Egress:** default unrestricted; use `-p`/`-e` to lock it down (see above). - **Keepalive:** long/quiet compiles run detached with a heartbeat and survive stream drops. `-K` keeps the box on a real failure for inspection. - **Excludes:** `.git`/`target`/`node_modules`/`__pycache__`/`.venv`/media are excluded from the upload by default; `-x ` adds more. -- **Swap caveat:** `-w ` _attempts_ swap, but `devbox:1` can't `swapon` today — so a torch/maturin build on a small box may OOM/ENOSPC. Install only the extra/group you need (e.g. `uv sync --group dev`) rather than `--all-extras`. +- **Swap:** no dedicated flag — `devbox:1` can't `swapon` a file added after boot from most shapes anyway, so compose it into the command instead: `'fallocate -l 4G /swapfile && mkswap /swapfile && swapon /swapfile && '`. A torch/maturin build on a small box is still more reliably fixed with a bigger shape; install only the extra/group you need (e.g. `uv sync --group dev`) rather than `--all-extras`. - **Shape rejection:** a shape your account can't use fails with a clean `Allowed: [...]` list — pick from it. - **Bandwidth:** each box starts with a fixed allowance for traffic it initiates (5 GiB by default). A large model download or `docker pull` can exhaust it, after which outbound traffic stops. Top it up additively via `createos sandbox edit `; the exec, file-transfer, and tunnel channels stay reachable regardless. ## Uploads & excludes -Uploads (`offload`/`fanout`) and one-way `sync` skip big/regenerable dirs by default so you don't ship `node_modules` over the wire: +Uploads (`offload`/`fanout`/`matrix`) and one-way `sync` skip big/regenerable dirs by default so you don't ship `node_modules` over the wire: ``` .git target node_modules __pycache__ .venv .mypy_cache .pytest_cache @@ -355,14 +375,15 @@ cos install # symlink onto PATH (once); create cos offload -p python-uv . 'uv sync --frozen --group dev && uv run pytest -q' cos offload -p python-uv -p rust-cargo -x target -o dist . 'uv sync --frozen && uv run pytest -q' cos up && cos run 'npm ci' && cos sync ~/app /work # reusable box + one-way sync -cos fanout -j 2 -p python-uv . 'pytest tests/a' 'pytest tests/b' # parallel, isolated boxes +cos fanout -p python-uv . 'pytest tests/a' 'pytest tests/b' # parallel, isolated forks +cos matrix -P 'npm ci' . 'npm test -- a' 'npm test -- b' # shared setup, then fork cos shell # instant throwaway Linux (destroyed on exit) cos run 'npm run dev &' && cos tunnel 3000 # dev server → http://127.0.0.1:3000 cos expose 8080 # public HTTPS URL for port 8080 cos cluster up 3 && cos cluster run -a 'hostname' # 3 boxes, one private net cos disk create data --bucket my-b --endpoint https://s3.amazonaws.com --access-key … --secret-key … cos disk attach data /mnt/data # mount S3 into the project box -cos fork # snapshot → independent clone +cos fork -c 2 # snapshot → 2 independent clones cos template submit myimage -f Dockerfile # bake a toolchain into a reusable image cos pause # park the warm box at zero compute cost cos resume # bring it back exactly as it was @@ -373,7 +394,7 @@ cos down # stops sync/tunnels, destr - **One-way by default.** `sync` and offload uploads are laptop → box; box-side writes never touch local unless you opt into `-2` (two-way) or pull with `offload -o`. Use `-2` deliberately — never casually on a repo root. `-M` (mirror) additionally **deletes** box-side extras. - **Scoped statefile.** `cos` only ever touches boxes it created (`cos-*`) or the project box recorded in its statefile. Your other sandboxes are never touched. State lives at `${COS_STATE_DIR:-${XDG_CACHE_HOME:-~/.cache}/createos-sandbox}/.json` (plus a `.tunnels` sidecar). -- **Quota.** External keys have been observed to allow 2 boxes running at once, with a daily creation cap. Neither is published policy — treat them as observed behaviour, budget `cluster`/`fanout` against them, and expect excess jobs to queue rather than fail. +- **Quota.** This account has been observed to allow at least 10 boxes running at once, with a daily creation cap. Neither is published policy — treat them as observed behaviour, budget `cluster`/`fanout`/`matrix` against them, and expect excess jobs to queue rather than fail. Check the live number if a decision depends on it. - **Ending a session.** `pause` keeps the box (and its warm state) at zero compute cost; `down` destroys it. Either is fine — leaving a box _running_ is not. - **Untrusted code.** Restrict egress (`-p`/`-e`) so a malicious dependency can't exfiltrate or phone home. @@ -399,7 +420,8 @@ cos down # stops sync/tunnels, destr - **Template build rejected** — `cos template submit` preflights single-stage/no-`COPY`/64 KiB locally; a rejection past that is usually a base image outside the operator allowlist. - **Build can't reach a host** — you restricted egress; add the host with `-e ` or the right `-p `. - **`sync` copied `node_modules` anyway** — your `createos` CLI is old and lacks `--exclude`; upgrade it (`cos` warns when this happens). -- **Build OOM/ENOSPC on a small box** — bump `-s `; `-w` swap doesn't work on `devbox:1`. Install only the deps you need. +- **Build OOM/ENOSPC on a small box** — bump `-s `; there is no swap flag (`devbox:1` can't `swapon` a file added after boot anyway), so add one inside the command itself if you need it. Install only the deps you need. +- **`fork -c N` or a `matrix` clone only produced one sandbox** — an old plugin build; the fix (`--count`/forks after the sandbox id being read correctly) shipped in createos-cli's `feat/sandbox-compositions` branch. Confirm with `createos sandbox fork --help` that `--count` is listed, and update the CLI (`createos upgrade`) if it isn't picking up multiple clones. - **Public `expose` URL returns nothing** — the service must bind `0.0.0.0`, not `127.0.0.1`. --- diff --git a/packages/claude-code-plugin/commands/cluster.md b/packages/claude-code-plugin/commands/cluster.md index 47caafb..b31ce4d 100644 --- a/packages/claude-code-plugin/commands/cluster.md +++ b/packages/claude-code-plugin/commands/cluster.md @@ -4,7 +4,7 @@ argument-hint: "up [-s shape] [-e dom|-p preset|-E] | run [|-a] -2.fc.local:8080` resolves, the bare `cos-cl--2` returns NXDOMAIN. Egress defaults to unrestricted on every member; `-e`/`-p` restrict to an exact set, `-E` to keep it explicitly unrestricted. `run` execs on one member (index `1..N`, its name, or `-a` for all). Clusters count against quota (2 running at once on external keys) — keep N small. +Spin up N sandboxes on one private overlay network. Members reach each other by name — but the name must be **fully qualified**: `curl http://cos-cl--2.fc.local:8080` resolves, the bare `cos-cl--2` returns NXDOMAIN. Egress defaults to unrestricted on every member; `-e`/`-p` restrict to an exact set, `-E` to keep it explicitly unrestricted. `run` execs on one member (index `1..N`, its name, or `-a` for all). Clusters count against the account's running quota (observed around 10 at once) — keep N small. !`"${CLAUDE_PLUGIN_ROOT}/scripts/cos" cluster $ARGUMENTS` diff --git a/packages/claude-code-plugin/commands/fanout.md b/packages/claude-code-plugin/commands/fanout.md index 02309b1..bdeee34 100644 --- a/packages/claude-code-plugin/commands/fanout.md +++ b/packages/claude-code-plugin/commands/fanout.md @@ -4,8 +4,8 @@ argument-hint: "[-j N] [-e dom|-p preset|-E] [-s shape] [-x glob] [ allowed-tools: Bash --- -Fan a set of commands across isolated throwaway boxes concurrently — each `` runs in its own box staged from ``, keepalive-protected. Per-job logs + exit codes are summarized at the end; every box auto-destroys. Default concurrency `-j 2` (external-key quota is 2 running at once) — raise only if your plan allows. Big dirs (`node_modules`/`target`/…) are excluded from the upload. Egress defaults to unrestricted per job; `-e`/`-p` restrict to an exact set, `-E` to keep it explicitly unrestricted. +Fan a set of commands across isolated boxes concurrently — `` is staged once, forked once per ``, keepalive-protected. Per-job logs + exit codes are summarized at the end; every fork auto-destroys. Default concurrency 10, matching this account's observed running quota — raise only if your plan allows. Big dirs (`node_modules`/`target`/…) are excluded from the upload. Egress defaults to unrestricted per job; `-e`/`-p` restrict to an exact set, `-E` to keep it explicitly unrestricted. !`test -n "$ARGUMENTS" && "${CLAUDE_PLUGIN_ROOT}/scripts/cos" fanout $ARGUMENTS || "${CLAUDE_PLUGIN_ROOT}/scripts/cos" fanout` -Report each job's `rc` + log path above. This is distinct from `/createos-sandbox:cluster` (which networks boxes together) — fanout boxes are independent and never see each other. +Report each job's `rc` + log path above. This is distinct from `/createos-sandbox:cluster` (which networks boxes together) — fanout boxes are independent and never see each other. If every job shares the same setup (a dependency install, a toolchain), `/createos-sandbox:matrix` runs that setup once instead of paying it per job. diff --git a/packages/claude-code-plugin/commands/fork.md b/packages/claude-code-plugin/commands/fork.md index 4c079e0..67fa910 100644 --- a/packages/claude-code-plugin/commands/fork.md +++ b/packages/claude-code-plugin/commands/fork.md @@ -1,10 +1,11 @@ --- -description: Snapshot the active CreateOS project box and fork it into an independent clone (pauses briefly, then resumes). For matrix / parallel experiments from identical warm state. +description: Snapshot the active CreateOS project box and fork it into one or more independent clones (pauses briefly, then resumes). For branching experiments from identical warm state. +argument-hint: "[-c N]" allowed-tools: Bash --- -Pause the project box, fork its snapshot into a new independent sandbox (auto-resumes), then resume the project box. The clone starts from identical state and diverges from there. It is **not** tracked as the project box — so `cos down` will not destroy it — but it is recorded in the statefile, listed by `cos status`, and reaped by `cos down -f`. +Pause the project box, fork its snapshot into N new independent sandboxes (`-c N`, default 1; each auto-resumes), then resume the project box. Every clone starts from identical state and diverges from there. Clones are **not** tracked as the project box — so `cos down` will not destroy them — but each is recorded in the statefile, listed by `cos status`, and reaped by `cos down -f`. -!`"${CLAUDE_PLUGIN_ROOT}/scripts/cos" fork` +!`test -n "$ARGUMENTS" && "${CLAUDE_PLUGIN_ROOT}/scripts/cos" fork $ARGUMENTS || "${CLAUDE_PLUGIN_ROOT}/scripts/cos" fork` -Report the fork id above and remind the user to `createos sandbox rm -y ` (or `cos down -f`) when done — clones count against the 2-running quota. +Report the fork id(s) above and remind the user to `createos sandbox rm -y ` (or `cos down -f`) when done — clones count against the account's running quota. Running the SAME job N ways from a directory rather than the project box is usually a better fit for `/createos-sandbox:matrix`. diff --git a/packages/claude-code-plugin/commands/matrix.md b/packages/claude-code-plugin/commands/matrix.md new file mode 100644 index 0000000..e109cf8 --- /dev/null +++ b/packages/claude-code-plugin/commands/matrix.md @@ -0,0 +1,11 @@ +--- +description: Fanout with a shared setup step — build one box, run a command once on it, fork it once per job, run each job on its own clone. For a test-shard matrix that all need the same dependency install. +argument-hint: "[-P setup] [-F box] [-j N] [-e dom|-p preset|-E] [-s shape] [-x glob] [cmd2] ..." +allowed-tools: Bash +--- + +Fanout, plus one thing: `-P ''` runs once on the golden box — the dependency install, the toolchain prep — before it forks once per ``. Reach for this the moment two or more jobs would otherwise repeat the same setup; the setup is paid once, not once per job. `-F ` skips staging a directory and forks an existing sandbox you already prepared and paused yourself, instead. `-j` defaults to 10, matching this account's observed running quota. Big dirs (`node_modules`/`target`/…) are excluded from the upload. Egress defaults to unrestricted per job; `-e`/`-p` restrict to an exact set, `-E` to keep it explicitly unrestricted. + +!`test -n "$ARGUMENTS" && "${CLAUDE_PLUGIN_ROOT}/scripts/cos" matrix $ARGUMENTS || "${CLAUDE_PLUGIN_ROOT}/scripts/cos" matrix` + +Report each job's `rc` + log path above, plus whether the golden box was destroyed (default) or kept (`-G`). Two known limits carry over from `createos sandbox matrix --help`: a fork does not carry the golden box's S3 disk attachments, and a clone whose snapshot isn't cached on the target host can take 11-13 s to resume rather than the usual sub-second. If the jobs have nothing in common, `/createos-sandbox:fanout` is the simpler tool for the same shape of work. diff --git a/packages/claude-code-plugin/commands/offload.md b/packages/claude-code-plugin/commands/offload.md index add77b6..66e54fc 100644 --- a/packages/claude-code-plugin/commands/offload.md +++ b/packages/claude-code-plugin/commands/offload.md @@ -1,10 +1,10 @@ --- -description: One-shot offload to a disposable CreateOS sandbox — stage a dir, run a command (keepalive-protected), optionally pull artifacts, auto-destroy. For heavy/untrusted/clean-room builds and tests. -argument-hint: "[-p preset] [-e dom] [-E] [-x glob] [-o out] [-w GB] [-K] [-s shape] " +description: One-shot offload to a disposable CreateOS sandbox — stage a dir, run a command (keepalive-protected), optionally fetch artifacts, auto-destroy. For heavy/untrusted/clean-room builds and tests. +argument-hint: "[-p preset] [-e dom] [-E] [-x glob] [-o out] [-K] [-s shape] " allowed-tools: Bash --- -Offload to a throwaway CreateOS Sandbox. Flags precede ` `. Egress defaults to unrestricted (any host); `-p python-uv|rust-cargo|npm|github` restricts to an exact set (compose with `-e ` for extras), or `-E` to keep it explicitly unrestricted; `-x ` adds upload excludes (`.git`/`target`/`node_modules`/… already excluded); `-o ` pulls artifacts back; `-w ` swap; `-K` keep box on failure. Long/quiet builds survive exec-stream drops (detached run + heartbeat + re-attach). A shape your account can't use is rejected with an `Allowed: [...]` list — pick from it. +Offload to a throwaway CreateOS Sandbox. Flags precede ` `. Egress defaults to unrestricted (any host); `-p python-uv|rust-cargo|npm|github` restricts to an exact set (compose with `-e ` for extras), or `-E` to keep it explicitly unrestricted; `-x ` adds upload excludes (`.git`/`target`/`node_modules`/… already excluded); `-o ` fetches a path back into the local dir; `-K` keep box on failure. Long/quiet builds survive exec-stream drops (detached run + heartbeat + re-attach). A shape your account can't use is rejected with an `Allowed: [...]` list — pick from it. Need swap for a heavy build? Compose it into the command itself: `'fallocate -l 4G /swapfile && mkswap /swapfile && swapon /swapfile && '`. Note on egress: IP and CIDR rules apply immediately, while domain rules take ~30 s — and a domain allowlist is a strong control for HTTPS traffic but a weak one for cleartext HTTP. When blocking exfiltration is the actual goal, prefer IP/CIDR rules. diff --git a/packages/claude-code-plugin/scripts/cos b/packages/claude-code-plugin/scripts/cos index 5e0b440..1269f0f 100755 --- a/packages/claude-code-plugin/scripts/cos +++ b/packages/claude-code-plugin/scripts/cos @@ -2,19 +2,30 @@ # cos — drive a CreateOS sandbox as remote compute for Claude / agents. # # Patterns: -# offload (one-shot, safe): stage a dir → run (keepalive+retry) → pull artifacts → destroy +# offload (one-shot, safe): createos sandbox offload — stage → run → fetch → destroy +# fanout/matrix (parallel): createos sandbox matrix — one box staged, forked N ways # up/run/sync/down (reusable): a per-repo box + one-way/two-way/mirror file sync # +# offload/fanout/matrix/fork are thin: cos translates flags and keeps the UX +# it owns (heavy-build warnings, project-box pause/resume orchestration), and +# hands the actual staging/execution/lifecycle work to `createos sandbox`, +# which already does it safely (retry, no billable leaks on failure, teardown +# failures that fail the command instead of hiding it). See createos-cli's +# `sandbox offload|matrix|fork --help` for the full flag set either wraps. +# # `cos` is NOT on PATH by default — run `cos install` once (symlinks into # ~/.local/bin), or invoke it by full path: "$CLAUDE_PLUGIN_ROOT/scripts/cos". # # Subcommands: # cos install [target] symlink this script onto PATH (default ~/.local/bin/cos) -# cos offload [flags] throwaway box: stage→run→pull→destroy +# cos offload [flags] throwaway box: stage→run→fetch→destroy +# cos fanout [-j N][flags] ... N throwaway forks of one staged box, in parallel +# cos matrix [-P prep][flags] ... fanout, but the setup runs once before forking # cos up [-s][-r][-n][-e|-p|-E][-a] create/reuse a project box (-a to adopt) # cos run run in the project box (keepalive) # cos sync [-2|-M][-x] [remote] start file sync into the project box (bg) # cos pause | cos resume park the warm box at zero compute cost / bring it back +# cos fork [-c N] snapshot the project box → N independent clone(s) # cos template submit [-f Dockerfile] build a custom rootfs (bake the toolchain once) # cos down [-f] stop sync + destroy box + clear state (-f reaps forks) # cos status show active box + sync state @@ -51,7 +62,8 @@ NORMA=() _mapl(){ case "$1" in --shape) echo -s;; --rootfs) echo -r;; --out|--output) echo -o;; --name) echo -n;; --egress) echo -e;; --egress-preset) echo -p;; --egress-all) echo -E;; --adopt) echo -a;; - --exclude) echo -x;; --keep-on-fail) echo -K;; --swap) echo -w;; + --exclude) echo -x;; --keep-on-fail) echo -K;; --count) echo -c;; + --prepare) echo -P;; --from) echo -F;; --keep-golden) echo -G;; --logs) echo -L;; --two-way) echo -2;; --mirror) echo -M;; --forks) echo -f;; *) echo "$1";; esac; } _norm(){ NORMA=(); while [ $# -gt 0 ]; do case "$1" in --) shift; NORMA+=(-- "$@"); break;; @@ -153,17 +165,6 @@ state_set(){ lock_state; local tmp; tmp=$(mktemp); [ -f "$STATE" ] || echo '{}' unlock_state } -# ── set up a swapfile in-box (OOM headroom for compiled-extension builds) ────── -setup_swap(){ local id=$1 gb=$2 - echo "cos: ensuring ${gb}G swap in box…" >&2 - "$CLI" sandbox exec "$id" -- bash -lc " - swapon --show 2>/dev/null | grep -q /cos.swap && { free -m | awk '/Swap/{print \"swap MB: \"\$2}'; exit 0; } - ( fallocate -l ${gb}G /cos.swap 2>/dev/null || dd if=/dev/zero of=/cos.swap bs=1M count=\$(( ${gb}*1024 )) status=none 2>/dev/null ) \ - && chmod 600 /cos.swap && mkswap /cos.swap >/dev/null 2>&1 && swapon /cos.swap 2>/dev/null \ - && free -m | awk '/Swap/{print \"swap MB: \"\$2}' \ - || echo 'swap setup failed (continuing without swap)' - " 2>&1 | sed 's/^/cos: /' >&2 || true -} # ── run a command in-box with keepalive + transient-retry ───────────────────── # Detaches the real command (survives stream death), then a heartbeat watcher @@ -195,24 +196,40 @@ run_keepalive(){ local id=$1 cmd=$2 wd=${3:-} llog=${4:-$STATE_DIR/last-exec.log done } -# ─────────────────────────────────────────────────────────────── one-shot offload -OFFLOAD_ID=""; KEEP=0 -on_offload_exit(){ [ -n "$OFFLOAD_ID" ] || return 0 - [ "$KEEP" = 1 ] && return 0 - "$CLI" sandbox rm -y "$OFFLOAD_ID" >/dev/null 2>&1 && echo "cos: destroyed $OFFLOAD_ID" >&2 || true; } - +# ───────────────────────────────────────────────────────────── one-shot offload +# Thin delegate: `createos sandbox offload` (createos-cli's own Go composition) +# already does stage->run->fetch->destroy, with retry on connection failures +# and a teardown failure that fails the command instead of leaking a billable +# box. cos only translates flags and keeps the heavy-build-on-a-small-box +# warning, which is UX cos owns and the CLI has no reason to know about. +offload_usage(){ cat <<'EOF' +cos offload — run a command in a throwaway sandbox (stage → run → fetch → destroy). + cos offload [flags] +flags: + -s shape -r rootfs -o out (path inside /work to fetch back) -K keep box on failure + -e allow one egress domain (repeatable) + -p egress preset: python-uv | rust-cargo | npm | github (repeatable, composes with -e) + -E unrestricted egress (already the default; kept for old invocations) + -x extra upload exclude (repeatable; .git/target/node_modules/__pycache__/.venv/media excluded by default) +example: + cos offload -p python-uv -p rust-cargo -x target . 'uv sync --frozen --group dev && uv run pytest -q' +Delegates to `createos sandbox offload` — see `createos sandbox offload --help` for the full flag set. +A heavy build needing swap: compose it into the command itself, e.g. + 'fallocate -l 4G /swapfile && mkswap /swapfile && swapon /swapfile && ' +EOF +} cmd_offload(){ _norm "$@"; set -- ${NORMA[@]+"${NORMA[@]}"} - local shape=s-1vcpu-1gb rootfs=devbox:1 out="" swap="" keep_on_fail=0 egress_all=0 - COS_EGRESS=(); local -a excl=() _d; local OPTIND=1 o doms d - while getopts "s:r:e:o:p:x:w:EKh" o; do case $o in - s) shape=$OPTARG;; r) rootfs=$OPTARG;; o) out=$OPTARG;; w) swap=$OPTARG;; - e) COS_EGRESS+=(--egress "$OPTARG");; - p) doms=$(egress_preset "$OPTARG") || die "unknown egress preset '$OPTARG' (have: python-uv rust-cargo npm github)" - read -ra _d <<<"$doms"; for d in "${_d[@]}"; do COS_EGRESS+=(--egress "$d"); done;; - x) excl+=("$OPTARG");; - E) egress_all=1;; - K) keep_on_fail=1;; + local -a args=(); local out="" shape=s-1vcpu-1gb OPTIND=1 o + while getopts "s:r:e:o:p:x:EKh" o; do case $o in + s) shape=$OPTARG; args+=(--shape "$OPTARG");; + r) args+=(--rootfs "$OPTARG");; + o) out=$OPTARG;; + e) args+=(--egress "$OPTARG");; + p) args+=(--egress-preset "$OPTARG");; + x) args+=(--exclude "$OPTARG");; + E) :;; # unrestricted is already the default — kept so old invocations don't error + K) args+=(--keep-on-fail);; h) offload_usage; exit 0;; *) offload_usage >&2; exit 2;; esac; done shift $((OPTIND-1)) @@ -220,62 +237,15 @@ cmd_offload(){ if [ $# -lt 2 ]; then offload_usage; exit 0; fi local dir=$1 cmd=$2 [ -d "$dir" ] || die "no such dir: $dir" - [ "$egress_all" = 1 ] && COS_EGRESS=() + [ -n "$out" ] && args+=(--fetch "$out") - # warn: heavy compiled build on a small box - local heavy=0 small=0 - case "$cmd" in *cargo*|*maturin*|*torch*|*"pip install"*|*"uv sync"*|*"uv run"*|*pyo3*) heavy=1;; esac - case "$shape" in *256mb|*512mb|*-1gb) small=1;; esac - [ "$heavy" = 1 ] && [ "$small" = 1 ] && [ -z "$swap" ] && \ - echo "cos: ⚠ heavy build on small box ($shape) — risk of OOM/ENOSPC. Try -s s-2vcpu-2gb or --swap 4." >&2 - - local name="cos-o-$$-${RANDOM}" - OFFLOAD_ID=$(create_box "$name" "$shape" "$rootfs") - KEEP=0; trap on_offload_exit EXIT - echo "cos: $OFFLOAD_ID ($name, $shape/$rootfs)" >&2 - wait_running "$OFFLOAD_ID" 30 || die "box $OFFLOAD_ID not running after 30s" - - # stage with excludes (.git/build artifacts/large media skipped by default) - local -a tarx=(); local p - for p in "${DEFAULT_EXCLUDES[@]}" ${excl[@]+"${excl[@]}"}; do tarx+=(--exclude "$p"); done - echo "cos: staging $dir → box:/work (default excludes + ${excl[*]:-none})" >&2 - tar "${tarx[@]}" -c -C "$dir" . | "$CLI" sandbox push "$OFFLOAD_ID" - /work.tar >/dev/null 2>&1 || die "push failed" - "$CLI" sandbox exec "$OFFLOAD_ID" -- bash -lc 'mkdir -p /work && tar -C /work -xf /work.tar && rm -f /work.tar && echo ok' >/dev/null 2>&1 || die "extract failed" - - [ -n "$swap" ] && setup_swap "$OFFLOAD_ID" "$swap" - - run_keepalive "$OFFLOAD_ID" "$cmd" /work || true - local rc=${BUILD_RC:-1} - if [ "${INFRA_FAIL:-0}" = 1 ]; then - KEEP=1; rc=1 - echo "cos: ⚠ infra/stream failure — box kept so the build cache survives." >&2 - echo "cos: reconnect: createos sandbox exec --stream $OFFLOAD_ID -- bash -lc 'tail -f /tmp/.cos-run.log'" >&2 - echo "cos: destroy: createos sandbox rm -y $OFFLOAD_ID" >&2 - elif [ "$rc" != 0 ] && [ "$keep_on_fail" = 1 ]; then - KEEP=1 - echo "cos: command exited $rc — box kept (--keep-on-fail). destroy: createos sandbox rm -y $OFFLOAD_ID" >&2 - fi + # warn: heavy compiled build on a small box (cos-owned UX; the CLI has no + # reason to know what "heavy" means for your workload) + case "$cmd" in *cargo*|*maturin*|*torch*|*"pip install"*|*"uv sync"*|*"uv run"*|*pyo3*) + case "$shape" in *256mb|*512mb|*-1gb) + echo "cos: ⚠ heavy build on small box ($shape) — risk of OOM/ENOSPC. Try -s s-2vcpu-2gb." >&2;; esac;; esac - if [ -n "$out" ]; then - echo "cos: pulling /work/$out → $dir/" >&2 - if "$CLI" sandbox exec "$OFFLOAD_ID" -- bash -lc "cd /work && tar -c $out" 2>/dev/null | tar -x -C "$dir"; then :; else - echo "cos: ⚠ pull of '$out' FAILED — artifacts NOT retrieved (does '$out' exist under /work?). Box is about to be destroyed." >&2 - fi - fi - return "$rc" -} -offload_usage(){ cat <<'EOF' -cos offload — run a command in a throwaway sandbox (stage → run → pull → destroy). - cos offload [flags] -flags: - -s shape -r rootfs -o out (tar dir to pull back) -w GB (swap) -K keep box on failure - -e allow one egress domain (repeatable) - -p egress preset: python-uv | rust-cargo | npm | github (repeatable, composes with -e) - -E unrestricted egress (trusted offload) - -x extra upload exclude (repeatable; .git/target/node_modules/__pycache__/.venv/media excluded by default) -example: - cos offload -p python-uv -p rust-cargo -x target . 'uv sync --frozen --group dev && uv run pytest -q' -EOF + "$CLI" sandbox offload "${args[@]}" "$dir" -- "$cmd" } # ───────────────────────────────────────────────────────────── reusable project box @@ -608,34 +578,57 @@ cmd_vpn(){ } # ───────────────────────────── fork: snapshot the project box into an independent clone +# The pause→resume orchestration around the project box stays here on purpose: +# it is YOUR persistent box, explicitly forked by request, so cos pausing it +# briefly is the one legitimate exception to "never auto-pause a running +# sandbox" — `createos sandbox fork` itself refuses to do this for anything +# it doesn't own. The actual fork + id resolution delegates to the CLI, which +# returns the new id(s) as JSON directly — no more diffing `sandbox ls` +# before/after, which is what the old multibyte/set -u trap here used to do. cmd_fork(){ + local count=1 OPTIND=1 o + while getopts "c:h" o; do case $o in + c) count=$OPTARG;; + h) echo "cos fork [-c N] snapshot the project box, fork it (N times), then resume the project box"; exit 0;; + *) exit 2;; esac; done + shift $((OPTIND-1)) + numeric "$count" && [ "$count" -ge 1 ] || die "-c must be a positive integer" + local id; id=$(state_get id); [ -n "$id" ] || die "no active project box — 'cos up' first" wait_running "$id" 15 || die "box $id not running" echo "cos: pausing $id to snapshot (brief; needed to fork)…" >&2 "$CLI" sandbox pause "$id" >/dev/null 2>&1 || die "pause failed" - local before after newid - before=$("$CLI" -o json sandbox ls 2>/dev/null | jq -r '.[].id' | sort) - echo "cos: forking…" >&2 - if ! "$CLI" sandbox fork "$id" >"$STATE_DIR/last-fork.log" 2>&1; then + + local plural=""; [ "$count" -gt 1 ] && plural=" (x$count)" + echo "cos: forking${plural}…" >&2 + local result + if ! result=$("$CLI" -o json sandbox fork "$id" --count "$count" 2>"$STATE_DIR/last-fork.log"); then "$CLI" sandbox resume "$id" >/dev/null 2>&1 || true die "fork failed — see $STATE_DIR/last-fork.log (project box resumed)" fi - after=$("$CLI" -o json sandbox ls 2>/dev/null | jq -r '.[].id' | sort) # brace the expansion: a bare `$id` followed directly by a multibyte char (the - # ellipsis) is parsed as part of the variable NAME, so `set -u` kills the script - # here — after the fork has already happened, orphaning the clone. + # ellipsis) is parsed as part of the variable NAME, so `set -u` kills the + # script here — after the fork has already happened, orphaning the clone. + # (This is not hypothetical: it happened live during testing of this exact + # rewrite. Every other new echo in this function was written pre-braced.) echo "cos: resuming project box ${id}…" >&2 "$CLI" sandbox resume "$id" >/dev/null 2>&1 || echo "cos: ⚠ resume of $id failed — run 'createos sandbox resume $id'" >&2 - newid=$(comm -13 <(printf '%s\n' "$before") <(printf '%s\n' "$after") | head -1) - [ -n "$newid" ] || { echo "cos: forked, but couldn't auto-resolve the new id — see 'createos sandbox ls'" >&2; return 0; } + + local newids; newids=$(printf '%s' "$result" | jq -r 'if type=="array" then .[].id else .id end') + [ -n "$newids" ] || { echo "cos: forked, but couldn't read the new id(s) from the response — see 'createos sandbox ls'" >&2; return 0; } + # A fork is an independent clone, so `down` won't destroy it — but record it, or # nothing ever will and it leaks silently. `down` reports it; `down -f` reaps it. - local forks; forks=$(state_get fork_ids) - state_set fork_ids "${forks:+$forks }$newid" - echo "cos: fork ready: $newid (independent clone; NOT tracked as the project box)" >&2 - echo "cos: exec: createos sandbox exec --stream $newid -- bash -lc '…'" >&2 - echo "cos: destroy: createos sandbox rm -y $newid (or 'cos down -f' to reap forks with the box)" >&2 - echo "$newid" + local forks nid; forks=$(state_get fork_ids) + for nid in $newids; do forks="${forks:+$forks }$nid"; done + state_set fork_ids "$forks" + + for nid in $newids; do + echo "cos: fork ready: $nid (independent clone; NOT tracked as the project box)" >&2 + done + echo "cos: exec: createos sandbox exec --stream -- bash -lc '…'" >&2 + echo "cos: destroy: createos sandbox rm -y (or 'cos down -f' to reap forks with the box)" >&2 + printf '%s\n' "$newids" } # ───────────────────────────── pause/resume: park the warm box at zero compute cost @@ -765,72 +758,93 @@ cmd_shell(){ } # ───────────────────────────── fanout: run commands across N throwaway boxes (parallel) +# Thin delegate: `createos sandbox matrix` with no --prepare IS this pattern — +# one box staged, forked once per job, each fork independent and destroyed +# after its job. A fork is a ~1s server-side snapshot copy, so this is faster +# than the old independently-staged-per-box approach and no less isolated: +# every job still ran from an identical, staged-once starting point either way. fanout_usage(){ cat <<'EOF' -cos fanout — run commands across N throwaway boxes in parallel (stage once → run each → collect → destroy). +cos fanout — run commands across N throwaway boxes in parallel (one box staged, forked per job). cos fanout [flags] [cmd2] [cmd3] ... -Each runs in its OWN isolated box, staged from . Per-job logs + exit codes summarized at the end. -flags: -j N (max concurrent, default 2 = external-key quota) · -s shape · -r rootfs · -p preset · -e dom · -E · -x glob +Each runs on its own clone of one box staged from . Per-job logs + exit codes summarized at the end. +flags: -j N (max concurrent, default 10) · -s shape · -r rootfs · -p preset · -e dom · -E · -x glob example: - cos fanout -j 2 -p python-uv . 'pytest -q tests/a' 'pytest -q tests/b' 'pytest -q tests/c' + cos fanout -j 5 -p python-uv . 'pytest -q tests/a' 'pytest -q tests/b' 'pytest -q tests/c' +Every job shares the same setup? Use `cos matrix -P ''` instead — it runs the setup +ONCE on the golden box before forking, rather than nothing shared at all. EOF } -fanout_one(){ # $1=idx $2=cmd $3=shape $4=rootfs $5=staged-tar $6=resdir - local i=$1 cmd=$2 shape=$3 rootfs=$4 staged=$5 resdir=$6 id - id=$(create_box "cos-fo-$$-$i-${RANDOM}" "$shape" "$rootfs" 2>/dev/null) \ - || { echo 127 > "$resdir/$i.rc"; echo "create failed" > "$resdir/$i.log"; return; } - if wait_running "$id" 30 \ - && "$CLI" sandbox push "$id" - /work.tar < "$staged" >/dev/null 2>&1 \ - && "$CLI" sandbox exec "$id" -- bash -lc 'mkdir -p /work && tar -C /work -xf /work.tar && rm -f /work.tar' >/dev/null 2>&1; then - run_keepalive "$id" "$cmd" /work "$resdir/$i.log" >/dev/null 2>&1 || true - echo "${BUILD_RC:-1}" > "$resdir/$i.rc" - else - echo "stage/boot failed" >> "$resdir/$i.log"; echo 1 > "$resdir/$i.rc" - fi - "$CLI" sandbox rm -y "$id" >/dev/null 2>&1 || true -} cmd_fanout(){ _norm "$@"; set -- ${NORMA[@]+"${NORMA[@]}"} - local shape=s-1vcpu-1gb rootfs=devbox:1 jobs=2 egress_all=0; COS_EGRESS=(); local -a excl=() _d; local OPTIND=1 o doms d + local -a args=(); local OPTIND=1 o while getopts "s:r:j:e:p:x:Eh" o; do case $o in - s) shape=$OPTARG;; r) rootfs=$OPTARG;; j) jobs=$OPTARG;; - e) COS_EGRESS+=(--egress "$OPTARG");; - p) doms=$(egress_preset "$OPTARG") || die "unknown egress preset '$OPTARG'" - read -ra _d <<<"$doms"; for d in "${_d[@]}"; do COS_EGRESS+=(--egress "$d"); done;; - x) excl+=("$OPTARG");; - E) egress_all=1;; + s) args+=(--shape "$OPTARG");; + r) args+=(--rootfs "$OPTARG");; + j) args+=(--concurrency "$OPTARG");; + e) args+=(--egress "$OPTARG");; + p) args+=(--egress-preset "$OPTARG");; + x) args+=(--exclude "$OPTARG");; + E) :;; # unrestricted is already the default — kept so old invocations don't error h) fanout_usage; exit 0;; *) fanout_usage >&2; exit 2;; esac; done shift $((OPTIND-1)) [ $# -ge 2 ] || { fanout_usage; exit 0; } local dir=$1; shift [ -d "$dir" ] || die "no such dir: $dir" - numeric "$jobs" && [ "$jobs" -ge 1 ] || die "-j must be a positive integer" - [ "$egress_all" = 1 ] && COS_EGRESS=() - local -a cmds=("$@"); local n=${#cmds[@]} - echo "cos: fanout $n job(s), ≤$jobs concurrent ($shape). External keys allow 2 running — keep -j ≤2." >&2 - - # stage the input tree once; every job pushes the same tarball - local -a tarx=(); local p - for p in "${DEFAULT_EXCLUDES[@]}" ${excl[@]+"${excl[@]}"}; do tarx+=(--exclude "$p"); done - local staged resdir; staged=$(mktemp "$STATE_DIR/fanout-stage.XXXXXX"); resdir=$(mktemp -d "$STATE_DIR/fanout.XXXXXX") - tar "${tarx[@]}" -c -C "$dir" . > "$staged" 2>/dev/null || die "staging tar failed" - - local i; local -a pids=() # bash-3.2-safe throttle: cap concurrency by waiting on the oldest pid - for ((i=0;i/dev/null || true; pids=("${pids[@]:1}"); fi - done - wait - rm -f "$staged" - - echo "cos: ── fanout results ──" >&2 - local rc all=0 - for ((i=0;i/dev/null || echo '?') - echo " job $i rc=$rc log=$resdir/$i.log cmd: ${cmds[$i]}" >&2 - [ "$rc" = 0 ] || all=1 - done - return "$all" + local cmd; for cmd in "$@"; do args+=(--job "$cmd"); done + "$CLI" sandbox matrix "${args[@]}" "$dir" +} + +# ───────────────────────────── matrix: one golden box, prepared once, forked N ways +matrix_usage(){ cat <<'EOF' +cos matrix — build one box, run --prepare once, fork it N times, run one job per clone in parallel. + cos matrix [flags] [cmd2] [cmd3] ... + cos matrix -F [flags] [cmd2] ... (fork from a sandbox you already prepared and paused) +Use this over `fanout` when every job shares the same setup (dependency install, toolchain) — the +setup runs ONCE on the golden box, then each job gets an independent fork of it. +flags: + -P run once on the golden box before forking (e.g. 'npm ci' or 'uv sync --frozen') + -F fork from this existing paused sandbox instead of building one from + -j N max concurrent clones (default 10) + -s shape · -r rootfs · -p preset · -e dom · -E · -x glob (same as offload/fanout) + -G keep the golden box after the run instead of destroying it + -L directory for per-job log files (default: a temporary directory) +example: + cos matrix -P 'bun install' . 'bun test unit' 'bun test e2e' 'bun test perf' +Known limits (from `createos sandbox matrix --help`): a fork does not carry its source's S3 disk +attachments, and a clone whose snapshot is not cached on the target host takes 11-13s to resume. +EOF +} +cmd_matrix(){ + _norm "$@"; set -- ${NORMA[@]+"${NORMA[@]}"} + local -a args=(); local from="" OPTIND=1 o + while getopts "s:r:j:e:p:x:P:F:L:EGh" o; do case $o in + s) args+=(--shape "$OPTARG");; + r) args+=(--rootfs "$OPTARG");; + j) args+=(--concurrency "$OPTARG");; + e) args+=(--egress "$OPTARG");; + p) args+=(--egress-preset "$OPTARG");; + x) args+=(--exclude "$OPTARG");; + P) args+=(--prepare "$OPTARG");; + F) from=$OPTARG;; + L) args+=(--logs "$OPTARG");; + E) :;; # unrestricted is already the default — kept so old invocations don't error + G) args+=(--keep-golden);; + h) matrix_usage; exit 0;; + *) matrix_usage >&2; exit 2;; esac; done + shift $((OPTIND-1)) + if [ -n "$from" ]; then + [ $# -ge 1 ] || { matrix_usage; exit 0; } + args+=(--from "$from") + local cmd; for cmd in "$@"; do args+=(--job "$cmd"); done + "$CLI" sandbox matrix "${args[@]}" + else + [ $# -ge 2 ] || { matrix_usage; exit 0; } + local dir=$1; shift + [ -d "$dir" ] || die "no such dir: $dir" + local cmd; for cmd in "$@"; do args+=(--job "$cmd"); done + "$CLI" sandbox matrix "${args[@]}" "$dir" + fi } # ───────────────────────────── disk: BYO S3 bucket mounts (attach/detach on the project box) @@ -929,8 +943,9 @@ cmd_install(){ main_usage(){ cat <<'EOF' cos — CreateOS sandbox as remote compute. (run `cos install` to put `cos` on PATH) cos auth check sign-in (CREATEOS_API_KEY, or `createos login` in a real terminal) - cos offload [flags] one-shot: stage→run(keepalive)→pull→destroy (cos offload -h for flags) - cos fanout [-j N][flags] ... run each in its own throwaway box, in parallel (cos fanout -h) + cos offload [flags] one-shot: stage→run(keepalive)→fetch→destroy (cos offload -h for flags) + cos fanout [-j N][flags] ... N throwaway forks of one staged box, in parallel (cos fanout -h) + cos matrix [-P prep][flags] ... fanout, but the setup runs once before forking (cos matrix -h) cos shell [-s][-r][-e|-p|-E] instant throwaway interactive Linux (destroyed on exit) cos up [-s][-r][-n][-e|-p|-E][-a] create/reuse project box (-a adopt a box cos didn't create) cos run run in project box (keepalive; ONE string, no '--' separator) @@ -940,7 +955,7 @@ cos — CreateOS sandbox as remote compute. (run `cos install` to put `cos` on cos cluster up [-s|-r|-e|-p|-E] | run [|-a] | ls | down N boxes on one private net cos disk create|ls|attach |detach|rm BYO S3 bucket mounts cos vpn [up|register [name]] WireGuard L3 into your private networks (needs wg-quick) - cos fork snapshot project box → independent clone (survives down; -f reaps) + cos fork [-c N] snapshot project box → N independent clone(s) (survives down; -f reaps) cos pause | cos resume park the warm box at zero compute cost / bring it back cos template submit [-f Dockerfile] | ls | show | logs | rm custom rootfs (bake the toolchain once) cos down [-f] stop sync/tunnels + destroy box (+ cluster; -f also destroys forks) @@ -965,6 +980,7 @@ case "$sub" in auth) cmd_auth "$@";; offload) cmd_offload "$@";; fanout) cmd_fanout "$@";; + matrix) cmd_matrix "$@";; shell) cmd_shell "$@";; up) cmd_up "$@";; run) cmd_run "$@";; diff --git a/packages/claude-code-plugin/skills/using-createos-sandbox/SKILL.md b/packages/claude-code-plugin/skills/using-createos-sandbox/SKILL.md index d4762f4..f06bcce 100644 --- a/packages/claude-code-plugin/skills/using-createos-sandbox/SKILL.md +++ b/packages/claude-code-plugin/skills/using-createos-sandbox/SKILL.md @@ -1,6 +1,6 @@ --- name: using-createos-sandbox -description: Use when you need to run code OFF the user's machine — heavy/long builds or test suites, untrusted or unknown code, a parallel test/config matrix across many boxes, an instant clean Linux to try a tool, a live dev-server/watcher Claude edits against, reaching a box-side service from localhost (port tunnel) or sharing it on the public web (HTTPS preview URL), a multi-machine cluster on one private network, a WireGuard VPN into that network, or mounting an S3 bucket of data. Offloads to ephemeral CreateOS Sandboxes via the `cos` helper (stage → exec → pull → auto-destroy), plus fanout, a scratch shell, and an opt-in reusable box with sync, tunnel, expose, cluster, disk, vpn, pause/resume, custom images, and snapshot/fork. +description: Use when you need to run code OFF the user's machine — heavy/long builds or test suites, untrusted or unknown code, a parallel test/config matrix across many boxes, an instant clean Linux to try a tool, a live dev-server/watcher Claude edits against, reaching a box-side service from localhost (port tunnel) or sharing it on the public web (HTTPS preview URL), a multi-machine cluster on one private network, a WireGuard VPN into that network, or mounting an S3 bucket of data. Offloads to ephemeral CreateOS Sandboxes via the `cos` helper (stage → exec → fetch → auto-destroy), plus fanout, matrix (shared-setup fan-out), a scratch shell, and an opt-in reusable box with sync, tunnel, expose, cluster, disk, vpn, pause/resume, custom images, and snapshot/fork. --- # Using CreateOS Sandbox as remote compute @@ -40,7 +40,8 @@ Every `cos` command except `install` and `auth` runs this check first, so an una | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | | **Untrusted / unknown code** — a snippet, a fresh npm/pip package, scraped code, a PoC exploit | Isolation. The blast radius is one disposable box, not the laptop. | | **Heavy build or test suite** — big `make`, full test run, compile, benchmark | Keeps the laptop free; runs on a box sized for it. | -| **Parallel/matrix work** — same job across N configs, test shards, batch | `fanout` — each command in its own throwaway box, concurrently, results collected. | +| **Parallel work, independent jobs** — N configs, test shards, batch, no shared setup | `fanout` — each command on its own fork of one staged box, concurrently. | +| **Parallel work, shared setup** — same install/toolchain, then N different commands | `matrix -P ''` — the setup runs once, then forks once per job. | | **Quick scratch Linux** — try a CLI/tool/snippet on a clean box | `shell` — instant keyless box, destroyed on exit (interactive; the user runs it). | | **Clean-room repro** — "works on my machine" bugs, dependency conflicts | Fresh rootfs every time, no host state. | | **Live dev loop** — dev server / test watcher / REPL that reacts to edits | Project box + `sync`; Claude edits locally, the box reacts. | @@ -98,10 +99,20 @@ For the full flag table, the egress presets, the enforcement caveats, fanout, an ### Fanout — same input, many boxes, in parallel ```bash -cos fanout -j 2 -p python-uv . 'pytest -q tests/unit' 'pytest -q tests/integration' 'ruff check' +cos fanout -p python-uv . 'pytest -q tests/unit' 'pytest -q tests/integration' 'ruff check' ``` -Each job gets its own box with no shared network — that is what distinguishes it from `cluster`. `-j` defaults to 2 to match the concurrency external keys have been observed to allow; going higher just queues the extra jobs rather than failing. +Each job runs on its own fork of one box staged from the directory, with no shared network between jobs — that is what distinguishes it from `cluster`. `-j` defaults to 10, matching this account's observed running concurrency; going higher just queues the extra jobs rather than failing. + +### Matrix — same setup, many jobs, in parallel + +```bash +cos matrix -P 'npm ci' . 'npm test -- --shard 1' 'npm test -- --shard 2' 'npm test -- --shard 3' +``` + +`fanout` with a shared setup step. `-P ''` runs once on the golden box — the dependency install, the toolchain prep — before it forks once per job. Use this the moment two or more jobs would otherwise repeat the same install: the setup cost is paid once instead of once per job. `-F ` forks from a sandbox you already prepared and paused yourself, instead of building a new golden box from a directory. + +Known limits, both inherited from `createos sandbox matrix --help`: a fork does not carry the golden box's S3 disk attachments (re-attach on each clone, or don't use disks with matrix), and a clone whose snapshot isn't cached on the target host takes 11–13 s to resume rather than the usual sub-second. ## Pattern B — reusable project box (opt-in) @@ -171,7 +182,7 @@ Disk data lives in the user's own S3 account and region. `--path-style` is neede - Ephemeral boxes self-destroy. The project box carries a 30-minute idle auto-pause as a backstop, so a forgotten box parks itself instead of billing overnight. Raise it with `createos sandbox edit --auto-pause 4h` when a box is serving an exposed URL people will hit intermittently — otherwise the demo will look dead between visitors. - Finish a live session with `cos pause` (keeping the warm state) or `cos down` (done for good). Don't leave a running box behind either way. -- **Concurrency is limited** — external keys have been observed to allow 2 boxes running at once, with a daily creation cap. This is observed behaviour rather than published policy, so budget `cluster` and `fanout` against it and expect excess jobs to queue rather than fail. +- **Concurrency is limited** — this account has been observed to allow at least 10 boxes running at once, with a daily creation cap. Neither is published policy, so budget `cluster`, `fanout`, and `matrix` against it and expect excess jobs to queue rather than fail. Check the actual number live if a decision hinges on it — it can change per plan. - If a shape is rejected, the error names the allowed list — pick from it, or run `createos sandbox shapes`. - Pre-existing boxes the user already runs are **not** yours. `cos` only ever destroys boxes it created itself; a box adopted with `cos up -a` survives `cos down`. - CreateOS Sandbox is in alpha with no SLA. When a limit or a number matters to a decision, check it live rather than quoting it from here. @@ -182,6 +193,6 @@ Load these when the task actually needs the depth — the summaries above are en | File | Read it for | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `references/offload-and-egress.md` | offload flag table, egress presets and how enforcement really behaves, fanout, upload excludes, heavy-build OOM/disk/bandwidth traps | +| `references/offload-and-egress.md` | offload flag table, egress presets and how enforcement really behaves, fanout, matrix, upload excludes, heavy-build OOM/disk/bandwidth traps | | `references/networking.md` | choosing between tunnel/expose/cluster/vpn, cluster DNS names, expose gotchas, WireGuard setup | | `references/lifecycle-and-images.md` | pause/resume, auto-pause tuning, fork caveats, built-in rootfs vs custom templates, env vars, remote editor, self-terminating jobs, single-file transfer, measured timings | diff --git a/packages/claude-code-plugin/skills/using-createos-sandbox/references/lifecycle-and-images.md b/packages/claude-code-plugin/skills/using-createos-sandbox/references/lifecycle-and-images.md index 3cfb404..561d866 100644 --- a/packages/claude-code-plugin/skills/using-createos-sandbox/references/lifecycle-and-images.md +++ b/packages/claude-code-plugin/skills/using-createos-sandbox/references/lifecycle-and-images.md @@ -45,15 +45,17 @@ Raising it is the right move when a box is serving an exposed URL that people wi ## fork — branch a warm box ```bash -cos fork # pauses briefly, clones the snapshot, resumes the original +cos fork # pauses briefly, clones the snapshot, resumes the original +cos fork -c 5 # same, but takes 5 independent clones in one call ``` -The clone is a fully independent box with its own id, IP, and quota ledger. The original is untouched. This is how you try N variants from one prepared state without redoing setup N times. +Each clone is a fully independent box with its own id, IP, and quota ledger. The original is untouched. This is how you try N variants from one prepared state without redoing setup N times — for running the SAME job N ways in parallel from a directory rather than the project box, `cos matrix` is usually the better fit; see `offload-and-egress.md`. -Two things to know: +Three things to know: - A fork is **not** tracked as the project box, so `cos down` leaves it running. It _is_ recorded in the statefile: `cos status` lists forks, `cos down` names the survivors, and `cos down -f` reaps them. Otherwise destroy it yourself with `createos sandbox rm -y `. -- **Mounted disks do not carry across a fork.** If the source box had an S3 disk attached, re-attach it on the clone. +- **Mounted disks do not carry across a fork.** If the source box had an S3 disk attached, re-attach it on each clone. +- **Fork refuses a running sandbox it doesn't own.** `cos fork` pauses the project box itself before forking, because it created that pause-resume cycle on purpose. Forking any OTHER running sandbox directly (`createos sandbox fork `) is refused unless it is already paused — nothing pauses a box you didn't ask it to pause. ## Images: built-in rootfs and custom templates @@ -125,9 +127,14 @@ curl -sX POST 'http://127.0.0.1:1029/self/pause?reason=idle' # or via the FIFO echo retire > /run/self # delete echo park > /run/self # pause + +# createos-cli also wraps these, if the CLI happens to be installed in-guest +createos sandbox self delete --reason batch-done ``` -Append this to the end of a long unattended command and the box cleans itself up whether or not anything is still watching. +Append this to the end of a long unattended command and the box cleans itself up whether or not anything is still watching. Prefer `curl`/the FIFO in a script you might run on any rootfs — they need nothing installed. The CLI wrapper only works if `createos` is actually present in the guest, which a stock `devbox:1` does not ship by default; bake it into a custom template if you want the wrapper form available. + +One side effect worth knowing: from `cos run`'s own view — watching over the control-plane connection, not from inside the box — a box that deletes itself looks identical to a dropped stream. `cos` cannot tell "the job finished and the box is gone on purpose" apart from "the box died mid-run," so its output ends abruptly rather than with a clean success line. Check `createos sandbox get ` (status `destroyed`) if you need to confirm which one happened. ## Single-file transfer @@ -147,7 +154,7 @@ Useful for setting expectations, and for not overpromising to the user: - **Create to first command runs: roughly 200 ms** (median; the guest kernel itself boots in tens of milliseconds, but the round trip through the control plane dominates). - **Pause and resume: around 6–8 seconds each, end to end through the CLI** (measured on a 1 GiB box; the platform-side operations are faster, the CLI polls for the state transition). Resume is slower when the snapshot has to move to a different host than it was taken on. **Fork: around a second** for the snapshot copy, plus the pause and resume around it. -- **Concurrency:** external API keys have been observed to allow 2 boxes running at once, with a daily creation cap. Neither number is published policy — treat them as observed behaviour, budget `cluster` and `fanout` against them, and expect excess jobs to queue rather than fail. +- **Concurrency:** this account has been observed to allow at least 10 boxes running at once, with a daily creation cap. Neither number is published policy — treat them as observed behaviour, budget `cluster`, `fanout`, and `matrix` against them, and expect excess jobs to queue rather than fail. - **Bandwidth:** 5 GiB of box-initiated egress per box by default, topped up additively. CreateOS Sandbox is in alpha and carries no SLA. Behaviour and limits can change — when a number matters to a decision, check it live rather than quoting this file. diff --git a/packages/claude-code-plugin/skills/using-createos-sandbox/references/offload-and-egress.md b/packages/claude-code-plugin/skills/using-createos-sandbox/references/offload-and-egress.md index e0d45a6..1d9cc9e 100644 --- a/packages/claude-code-plugin/skills/using-createos-sandbox/references/offload-and-egress.md +++ b/packages/claude-code-plugin/skills/using-createos-sandbox/references/offload-and-egress.md @@ -1,4 +1,4 @@ -# Offload, fanout, and the egress firewall +# Offload, fanout, matrix, and the egress firewall Read this when an offload needs tuning: restricting what a build can reach, sizing a box for a heavy compile, controlling what gets uploaded, or fanning work across boxes. @@ -9,6 +9,7 @@ Read this when an offload needs tuning: restricting what a build can reach, sizi - [Egress: how the firewall actually behaves](#egress-how-the-firewall-actually-behaves) - [Egress presets](#egress-presets) - [Fanout](#fanout) +- [Matrix](#matrix) - [Heavy builds: OOM, disk, and bandwidth](#heavy-builds-oom-disk-and-bandwidth) ## Offload flags @@ -20,7 +21,6 @@ Flags come **before** the ` ` positionals — `cos` parses with `getop | `-s ` | box size (default `s-1vcpu-1gb`); list with `createos sandbox shapes` | | `-r ` | base image or custom template (default `devbox:1`) | | `-o ` | tar this path out of `/work` back into the local dir after the run | -| `-w ` | try to add a swapfile (best-effort — see below) | | `-K` | keep the box if the command exits non-zero, so the cache survives for a retry | | `-e ` | allow one outbound destination (repeatable) | | `-p ` | apply an egress preset (repeatable, composes with `-e`) | @@ -69,17 +69,30 @@ Presets compose. A Python project with a Rust extension and a git dependency wan ## Fanout -`cos fanout [-j N] [flags] [cmd2] …` stages `` once and runs each command in its **own** throwaway box, concurrently, then reports per-job exit codes and log paths and destroys every box. +`cos fanout [-j N] [flags] [cmd2] …` stages `` once, forks that staged box once per command, runs each command on its own fork concurrently, then reports per-job exit codes and log paths and destroys every fork. -The jobs share no network — that is the difference from `cluster`, where boxes are wired together on purpose. Fanout is for a test matrix, a config sweep, or a batch where isolation between jobs is the point. +Under the hood this is `createos sandbox matrix` with no `--prepare` — the staged box is the golden box, and each job is a fork of it. The jobs share no network — that is the difference from `cluster`, where boxes are wired together on purpose. Fanout is for a test matrix, a config sweep, or a batch where isolation between jobs is the point and there is no shared setup step worth doing once. -`-j` defaults to 2 because that matches the concurrent-box limit observed on external API keys. Raising it past what the account allows does not fail — the extra jobs just queue, so a 3-way fanout with `-j 3` silently serializes into 2 + 1. One very long build is still better served by a single `offload`. +`-j` defaults to 10, matching this account's observed concurrent-box limit. Raising it past what the account actually allows does not fail — the extra jobs just queue. One very long build is still better served by a single `offload`. + +## Matrix + +`cos matrix [-P ''] [-F ] [flags] [cmd2] …` is `fanout` plus one thing: `-P ''` runs once on the golden box, before it forks, so every job's fork already has the dependency install or toolchain prep done. + +Reach for this the moment two or more jobs would otherwise repeat the same setup — a `pytest` shard matrix that all need the same `pip install`, three `npm test` invocations that all need the same `npm ci`. The setup is paid once; only the fork (roughly a second) and the job itself are paid per job. + +`-F ` skips building a golden box from a directory and forks an existing sandbox you already prepared and paused yourself — useful when the setup is expensive enough that you want to keep the golden box around and matrix against it repeatedly (pair with `-G` to keep it after the run too). + +Two limits carry over from `createos sandbox matrix --help`, both worth knowing before reaching for `-F` with disks attached or expecting sub-second forks everywhere: + +- **A fork drops the golden box's S3 disk attachments.** If the golden box has a disk mounted, the forks come up without it. Re-attach per fork, or avoid disks with matrix. +- **A fork can take 11–13 s, not always under a second.** That is the cross-host cold-fetch case, when the snapshot is not already cached on the host the fork lands on. ## Heavy builds: OOM, disk, and bandwidth **Shape rejection.** Picking a shape the account cannot use fails fast with a `not allowed … Allowed: [...]` line; pick from that list, or run `createos sandbox shapes`. (The rejection is real and reproducible on external keys; the exact policy behind it is not documented, so treat the allowed list as authoritative rather than guessing.) -**Swap is best-effort.** `-w ` tries to add a swapfile, but `devbox:1` cannot currently `swapon` — it stays at 0 MB, `cos` warns, and continues. If a compiled-extension build (pyo3/maturin, torch) is OOMing, the fix is a bigger shape or less work per run: build the extension separately, or install only the extra/group you actually need. +**Swap has no flag — compose it into the command.** `devbox:1` cannot currently `swapon` a swapfile added after boot from most shapes, so a dedicated `-w` flag bought little; if a compiled-extension build (pyo3/maturin, torch) needs the headroom, add it as the first step of the command itself: `'fallocate -l 4G /swapfile && mkswap /swapfile && swapon /swapfile && '`. The more reliable fix for an OOM is still a bigger shape or less work per run — install only the extra/group you actually need. **Disk fills fast.** `pip install --all-extras` or an unconstrained `uv sync` can pull CUDA and torch wheels measured in gigabytes and hit `No space left on device` on a small box. Install only what the job needs; `--disk-mib` at create time raises the ceiling if you control it.