feat(lab): backend-agnostic ephemeral env API (Docker driver, TTL reaper, hot/cold tiers, audit) - #1
Conversation
Introduce /api/v1/lab/envs (backend-agnostic), a Docker reference driver, hot/cold storage tiers, a TTL reaper, and concave lab CLI subcommands. The design is driver-pluggable (docker today, slurm and proxmox in follow-ups) under one API. Archives stamp a peer_id so envs archived on node A can be restored on node B. Every launch/extend/archive is written to a JSONL audit log at ~/gradient/config/lab-audit.jsonl. - internal/lab: Driver interface, Registry, Manager, Store, StorageConfig, DockerDriver, FileAuditWriter, TTL reaper goroutine. - internal/api/handlers_lab.go: CRUD + extend + archive + storage + driver endpoints, role-gated (viewer lists, developer launches/extends, operator archives, admin reconfigures storage/driver). - cmd/lab_envs.go: concave lab envs/storage/driver subcommands. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| return err | ||
| } | ||
| out, _ := json.MarshalIndent(storage, "", " ") | ||
| fmt.Println(string(out)) |
There was a problem hiding this comment.
🔴 CONTRIBUTING.md violation: fmt.Println used in cmd/ (runLabStorageShow)
fmt.Println(string(out)) at cmd/lab_envs.go:196 directly writes to stdout, violating the mandatory CONTRIBUTING.md rule: "All terminal output in cmd/ must go through internal/ui/printer.go. Do not use fmt.Println or log.Printf in cmd/." No other cmd/ file uses fmt.Println — the convention is consistently followed elsewhere.
| fmt.Println(string(out)) | |
| ui.Line(string(out)) |
Was this helpful? React with 👍 or 👎 to provide feedback.
| if err != nil { | ||
| return err | ||
| } | ||
| fmt.Println("active:", mgr.Registry().Active()) |
There was a problem hiding this comment.
🔴 CONTRIBUTING.md violation: fmt.Println used in cmd/ (runLabDriverShow, line 226)
fmt.Println("active:", ...) at cmd/lab_envs.go:226 directly writes to stdout, violating the mandatory rule: "Do not use fmt.Println or log.Printf in cmd/."
| fmt.Println("active:", mgr.Registry().Active()) | |
| ui.Info("active", mgr.Registry().Active()) |
Was this helpful? React with 👍 or 👎 to provide feedback.
| return err | ||
| } | ||
| fmt.Println("active:", mgr.Registry().Active()) | ||
| fmt.Println("drivers:", strings.Join(mgr.Registry().Names(), ", ")) |
There was a problem hiding this comment.
🔴 CONTRIBUTING.md violation: fmt.Println used in cmd/ (runLabDriverShow, line 227)
fmt.Println("drivers:", ...) at cmd/lab_envs.go:227 directly writes to stdout, violating the mandatory rule: "Do not use fmt.Println or log.Printf in cmd/."
| fmt.Println("drivers:", strings.Join(mgr.Registry().Names(), ", ")) | |
| ui.Info("drivers", strings.Join(mgr.Registry().Names(), ", ")) |
Was this helpful? React with 👍 or 👎 to provide feedback.
| if err := driver.Destroy(ctx, env); err != nil { | ||
| env.LastError = "destroy: " + err.Error() | ||
| _ = m.store.Update(env) | ||
| return env, err |
There was a problem hiding this comment.
🔴 Missing StatusFailed on destroy failure causes infinite reaper retry loop with repeated archiving
In ArchiveAndDestroy, when driver.Destroy fails after a successful archive, env.Status remains StatusArchiving (set at line 209) but is never updated to StatusFailed. Since Active() (internal/lab/types.go:90-96) returns true for StatusArchiving, the reaper (internal/lab/manager.go:264-284) picks this env up again on the next tick (every 30s). Each retry re-runs ArchiveAndDestroy, which re-creates the tarball and retries the destroy. This creates an infinite loop of wasteful I/O on the cold tier (typically an HDD). Note the inconsistency: archive failure at line 213-217 correctly sets StatusFailed, but the analogous destroy failure path at line 219-222 does not.
| if err := driver.Destroy(ctx, env); err != nil { | |
| env.LastError = "destroy: " + err.Error() | |
| _ = m.store.Update(env) | |
| return env, err | |
| if err := driver.Destroy(ctx, env); err != nil { | |
| env.Status = StatusFailed | |
| env.LastError = "destroy: " + err.Error() | |
| _ = m.store.Update(env) | |
| return env, err | |
| } |
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Summary
Introduces an ephemeral JupyterLab environment system with a backend-agnostic API (
/api/v1/lab/envs) so Docker, Slurm, and Proxmox VE can all plug in under one surface. This PR ships the Docker reference driver and the full lifecycle; Slurm and Proxmox drivers follow in separate PRs.What's in this PR
internal/labpackageDriverinterface (Name,OwnsTTL,Launch,Inspect,ExtendTTL,Archive,Destroy) +Registry.DockerDriverreference implementation: random-port Jupyter containers, hot-tier bind mount, optional--gpus/--cpus/--memory, tar.gz archive with sidecar manifest JSON.Manager: launches via the active driver, persists envs to~/gradient/config/lab-envs.json(atomic JSON), runs a TTL reaper goroutine that archives+destroys expired envs on drivers that don't own their own TTL.StorageConfig(HotTier/ColdTier) persisted at~/gradient/config/lab.json. Defaults to~/gradient/envs(hot) and~/gradient/envs-archive(cold); sysadmin overrides via CLI.AuditWriterwith a JSONLFileAuditWriterthat captures every launch/extend/archive — this is the first building block of the unified audit log called out in the roadmap.peer_idon everyArchiveRef(machine-id or hostname) so mesh-aware restore is feasible in PR #3.x (concave-web) and PR #4 (Slurm).HTTP surface (
internal/api/handlers_lab.go)GET/POST /api/v1/lab/envs— list and launch (viewer / developer-gated).GET/DELETE /api/v1/lab/envs/{id}— inspect and archive-then-destroy (viewer / operator-gated).POST /api/v1/lab/envs/{id}/extend— extend TTL (developer-gated).POST /api/v1/lab/envs/{id}/archive— archive now (operator-gated).GET/PUT /api/v1/lab/storage— inspect / update tier paths (admin-gated).GET/PUT /api/v1/lab/drivers— list / select active driver (admin-gated).App.ListenAndServe.CLI (
cmd/lab_envs.go)concave lab envs launch|list|extend|archiveconcave lab storage show|set --hot <path> --cold <path>concave lab driver show|set <driver>Tests (
internal/lab/lab_test.go)Env.Remaining/Env.Expiredhelpers.Gradient-specific touches (not "drag-and-drop" upstream)
.tar.gzgets a sidecar.tar.gz.jsoncarrying the originalEnvSpec,peer_id, and size so restore is deterministic.peer_idfield lets a future handler restore an env archived on node A onto node B.~/gradient/config/lab-audit.jsonl, the seed of the unified audit log.Review & Testing Checklist for Human
internal/lab/driver.goand confirm theDriverinterface covers the Slurm and Proxmox use cases you have in mind.internal/api/handlers_lab.go(viewer, developer, operator, admin) against your policy expectations.concave lab envs launch --image jupyter/minimal-notebook:latest --ttl 10mon a box with Docker, thenconcave lab envs list,concave lab envs extend <id> --by 5m,concave lab envs archive <id>and confirm:.jsonmanifest appear in the configured cold tier afterarchive.~/gradient/config/lab-audit.jsonlhas three JSONL entries.concave lab storage set --cold /mnt/bulk(or any absolute path you control) and confirm~/gradient/config/lab.jsonis updated and rejected on relative paths.--ttl 30sand watching the env transition toarchivedwithout a manual archive call.Notes
TestDriverWizardSetupAndSelfUpdatein./cmdfails in this environment because it requires a real Docker daemon to bring up the boosting suite — confirmed the same failure onmainin a fresh worktree. Not caused by this PR.gradient-oss-mirrors+gradient-ansiblerepos are follow-up PRs on the roadmap.Link to Devin session: https://app.devin.ai/sessions/5d19efa113054ca4953d9ed9309ce705
Requested by: @ElFariss