From 34d6224fc8bffea545c1e898fb949f37c072cc29 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 03:09:08 +0000 Subject: [PATCH 01/15] docs(adr): propose openab-pty optional sidecar runtime (supersedes in-process PTY mode) --- docs/adr/openab-pty-sidecar.md | 260 +++++++++++++++++++++++++++++++++ 1 file changed, 260 insertions(+) create mode 100644 docs/adr/openab-pty-sidecar.md diff --git a/docs/adr/openab-pty-sidecar.md b/docs/adr/openab-pty-sidecar.md new file mode 100644 index 000000000..b6b5f0be0 --- /dev/null +++ b/docs/adr/openab-pty-sidecar.md @@ -0,0 +1,260 @@ +# ADR: openab-pty — Optional Sidecar Runtime for Remote Sandboxed Terminals + +- **Status:** Proposed +- **Date:** 2026-08-15 +- **Author:** @pahud +- **Related:** [ADR: ACP Server with WebSocket Transport (base, as-built)](./acp-server-websocket-base.md), [ADR: Separate Binaries with Opt-In Unified Build](./unified-binary.md), [ADR: Secrets Management](./secrets-management.md), [ADR: Identity Trust None](./identity-trust-none.md) +- **Supersedes:** the in-process "PTY Mode" proposal (PR #1477, closed) — group review verdict and rationale are preserved in that PR's consolidated review +- **Implementation:** TBD + +--- + +## 1. Context & Problem + +A distinct user need exists that OAB's ACP model does not serve: + +> "I don't need multi-agent orchestration for this task. I have one or more coding CLIs (Claude Code, Codex, Kiro, or plain bash) and I want to drive them **directly** — full terminal, keyboard input, real-time output — from any device, with the session surviving my laptop." + +Adjacent tools each carry a trade-off: Herdr is laptop-local (laptop dies, session dies), OpenDray is host-resident (shell shares host credentials). A **remote + sandboxed + raw-terminal** offering does not exist. + +A previous proposal (PR #1477) made this a second in-process backend inside the OAB unified binary. Group review rejected that *form* — not the need — on five grounds: + +1. **Positioning**: a terminal server inside the broker contradicts DESIGN.md pillar #1 ("thin bridge" as a deliberate non-decision) +2. **Blast radius**: a PTY shell co-resident with the broker shares its PID/cgroup/network namespaces and mounted credential plane; "sandbox posture unchanged" did not hold +3. **Auth**: a static shared key cannot carry pod-shell-equivalent trust +4. **Lifecycle**: the ACP session pool is turn-based and ACP-specific; PTY byte-stream liveness is incompatible, so "reuse the pool" was not an available boundary +5. **Reversibility**: absorbing a second product persona into one binary is hard to undo + +This ADR proposes the same capability in a shape that answers all five. + +--- + +## 2. Decision + +Ship **`openab-pty`**: a separate binary running as an **optional sidecar container** in the OAB pod. Not deployed by default. OAB remains a pure ACP broker; the sidecar owns everything terminal. + +``` +K8s Pod ++--------------------------------------------------------------------+ +| | +| Container: openab (broker) Container: openab-pty | +| +---------------------------+ +---------------------------+ | +| | ACP session pool | | PTY session manager (own) | | +| | Platform adapters | | portable-pty spawner | | +| | Discord/Slack/... WS | | scrollback ring buffer | | +| | | | GET /pty/{session} (WSS) | | +| | [own config view: | | | | +| | platform tokens, agents] | | [own config view: | | +| +------------+--------------+ | [pty] section only] | | +| | +-------------+-------------+ | +| | (optional, Phase 4) | | +| +<--- notification webhook ----------+ | +| | +| Shared: workspace volume (PVC) | NOT shared: credentials, | +| | PID/cgroup, listeners | ++--------------------------------------------------------------------+ + | | + Discord / Slack Web terminal (xterm.js) + (turn-based) Mobile/desktop terminal client +``` + +### Why sidecar (and what it fixes) + +| Review blocker (PR #1477) | How the sidecar form resolves it | +|---|---| +| Positioning vs Thin Bridge | OAB binary is untouched; the broker stays a pure transport. `openab-pty` is an adjacent tool that shares deployment infrastructure only — no dual persona | +| Same-pod blast radius | Separate container = separate PID namespace, cgroup, filesystem, and mounts. The shell user cannot signal the broker, exhaust its cgroup, or read its credential files. Broker platform tokens are **never mounted** into the sidecar | +| Auth below capability | The sidecar designs its token model from scratch for shell-equivalent trust (see Security model) with no ACP-key coupling | +| Pool incompatibility | The sidecar has its **own session manager** built for byte-stream lifecycle. No refactor of the shipped ACP pool; zero regression risk to the broker | +| Reversibility | Default-off sidecar with its own image/release. If demand does not materialize, deprecate the image; nothing in the broker to unwind. If demand proves out, later extraction of a shared lifecycle crate — or even single-process merge — remains open | + +### Coexistence with ACP + +ACP and PTY coexist per deployment, not per process: + +- **Same pod, two containers** — one Helm toggle (`pty.enabled=true`) adds the sidecar; the broker container is byte-identical with or without it +- **Shared workspace volume** — the PTY shell and ACP agents can see the same working tree (same PVC mount), which is the practical point of coexistence: drive a CLI by hand in the terminal, then let ACP agents continue in the same workspace from Discord +- **Nothing else shared** — listeners, tokens, session state, and failure domains are independent; a crashed or compromised sidecar does not take the broker down + +### Configuration: one source, two views + +Operators keep a **single logical `config.toml`** (the existing `configUrl` flow); the two containers consume different projections of it: + +- The broker reads its existing sections; it ignores `[pty]` +- The sidecar reads **only** `[pty]`; it must never receive platform tokens, because any secret mounted into the sidecar is readable by the human at the terminal (the shell runs in that container) +- Delivery of the split follows the configUrl ADR: the chart (or operator) passes each container its own config source. For MVP this can be two URLs/objects derived from one source of truth; a `openab-pty run -c --section pty` style filter is an acceptable alternative +- `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY auth material is sourced via the secrets resolver per `secrets-management.md`, not a raw env var + +```toml +# one config.toml — two consumers +[discord] # broker only — never mounted into the sidecar +bot_token = "${DISCORD_BOT_TOKEN}" + +[agent] # broker only +# ... + +[pty] # sidecar only +enabled = true +listen = "0.0.0.0:8090" # separate port -> separate NetworkPolicy +command = "/bin/bash" # operator-configured; never client-specified +max_sessions = 4 +absolute_session_ttl = "12h" # applies even while attached +scrollback_kib = 1024 # in-memory only; cleared on teardown +scrollback_replay = false # off by default (secrets-safe) +auth_secret_ref = "aws-sm://openab/pty-signing-key" +``` + +### Security model + +- **Transport**: WSS required; plain WS permitted only on loopback for local dev. Fail-closed: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) +- **Browser credential transport**: reuse the validated `/acp` scheme — `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); origin policy and constant-time comparison carry over +- **Token model**: short-lived per-session tokens minted from the configured signing secret (attach = present a token scoped to one session name with an expiry), replacing the static-shared-key model the review rejected. Revocation = rotate the signing secret. An identity layer is explicitly out of MVP scope; the ADR states this per `identity-trust-none.md` rather than implying otherwise +- **Command authority**: the spawned command is operator configuration only; clients can never specify it. Session names are allowlist-validated (`[a-z0-9-]{1,32}`) +- **Isolation**: the sidecar container mounts the workspace volume and its own config view; no service-account token, no broker config, no platform secrets. NetworkPolicy can (and the chart docs will recommend) restrict sidecar egress independently of the broker +- **Audit in MVP**: attach/detach, session create/kill, and auth failures are logged from Phase 1; a leaked token must be observable +- **Env**: the PTY child gets an explicit allowlist (TERM, LANG/LC_*, PATH, HOME, USER, SHELL) and nothing else; `OPENAB_*` and cloud-credential variables are never inherited + +### Session lifecycle (owned by the sidecar, designed for byte streams) + +- **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong; a half-open socket counts as detached after the ping timeout) +- **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached — capacity cannot be pinned forever by an open browser tab +- **Attach semantics (MVP)**: single-attach exclusive; a second attach with a valid token detaches the first (documented; multi-viewer is Phase 3) +- **Reconnect**: monotonic byte cursor from day one — the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes, with an explicit gap signal on overflow (fresh attach = terminal reset + full replay only when `scrollback_replay=true`) +- **Teardown**: setpgid on spawn; SIGTERM-grace-SIGKILL escalation on the process group; evict-while-attached order = notify client, close socket, kill group, close master fd, release slot; buffers cleared on teardown; scrollback never touches disk +- **Recovery taxonomy** (stated, not implied): detach/reattach survives (process alive); pod restart does not (process dead) — reattach-to-dead returns a distinct error and offers restart-in-place. Pod-lifetime durability is out of scope and documented as such + +--- + +## 3. Consequences + +### Positive + +- OAB keeps its thin-broker identity untouched — zero changes to the shipped binary, pool, or ACP path +- Fills the remote + sandboxed + raw-terminal quadrant with a real container boundary instead of a claimed one +- Highest reversibility: default-off, separately versioned, separately deprecable +- Coexistence where it matters (shared workspace) without shared failure or credential domains +- The Phase 4 notification bridge (sidecar webhook -> broker -> Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes + +### Negative + +- A second binary and image to build, test, and release (mitigated by the existing multi-binary workspace and release pipeline) +- Cross-container coordination (notification bridge, future shared-crate extraction) is more ceremony than in-process calls +- Some duplication with the ACP pool (capacity accounting, pgid kill) until a shared lifecycle crate is justified by real usage + +### Neutral + +- Deployment surface grows only for operators who opt in; everyone else sees no change +- Whether this graduates to a shared crate or a merged process is deliberately deferred until product demand is proven + +--- + +## 4. Alternatives Considered + +### A. In-process dual-persona backend (rejected — the PR #1477 proposal) + +Rejected by unanimous group review: positioning conflict with the Thin Bridge pillar, same-pod blast radius, auth/lifecycle mismatch, low reversibility. See the consolidated review on PR #1477. + +### B. Extend ACP with observability events (deferred, complementary) + +`shellOutput`/`commandLog` ACP events would improve in-bridge visibility for every client, but deliver no keyboard-level control. Worth pursuing independently; the JSONL-transcript idea from the prior-art survey belongs to that track, not this one. + +### C. Integrate OpenDray / front a commodity tool (ttyd, gotty) (rejected for MVP) + +Fronting ttyd/gotty against an OAB-managed pod delivers raw PTY-over-WS cheaply, but: no session-token minting, no scrollback-cursor reconnect contract, no lifecycle TTLs, no audit — the hardening this ADR requires would have to be built around the commodity core anyway, in a codebase we do not control. OpenDray integration inherits its host-resident model. Revisit if MVP scope proves too costly. + +### D. `kubectl exec` + tmux runbook (rejected as the product answer) + +Zero code and genuinely useful for cluster admins — but it requires kubectl credentials and cluster access, which is precisely what the target user (a developer on a phone or borrowed laptop) does not have. Documented as an operator escape hatch, not the product. + +### E. Do nothing / remain ACP-only (rejected) + +Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's host blast radius. The sidecar form lets OAB serve it without betting the broker's identity. + +--- + +## 5. Implementation Plan + +### Phase 1: `openab-pty` MVP (new crate, new binary) + +- Own session manager: named sessions, operator-configured command, allowlist-validated names +- portable-pty spawner with setpgid, escalating kill, and the teardown order above +- `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`) with a defined close-code table +- Auth: per-session tokens from the signing secret; fail-closed off-loopback; `/acp`-style browser subprotocol transport +- Monotonic cursor reconnect with gap signaling; scrollback in-memory, off-by-default replay, cleared on teardown +- Detached-idle TTL + absolute lifetime cap; single-attach exclusive +- Audit log (attach/detach/create/kill/auth-failure) and basic metrics +- Resize propagation (TIOCSWINSZ) including attach-time initial size +- Terminal-capability response filtering at the PTY boundary (known Ink-CLI startup breakage) + +### Phase 2: Deployment + web client + +- Helm: `pty.enabled` toggle adds the sidecar container, its port, and a NetworkPolicy example; config split documented per the configUrl pattern +- Minimal xterm.js page served by the sidecar; session list/create/kill endpoints (same auth bar as attach) +- Rollback procedure: disabling the toggle drains (notify + grace) then kills sessions; broker unaffected + +### Phase 3: Lifecycle hardening + +- Multi-viewer (one writer, N readers) with writer-lease semantics and read-only token scope +- Reconnect backoff, richer capacity controls (per-token limits) + +### Phase 4: Messaging bridge (optional) + +- Sidecar posts a webhook to the broker when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); broker relays to the platform thread. Bridge is one-way and feature-gated + +### Later (demand-gated, explicitly deferred) + +- Shared lifecycle crate extraction (if the ACP pool and PTY manager converge naturally) +- Single-process merge (only if operations prove the sidecar split is more cost than benefit) +- Identity layer for PTY tokens; semantic agent-state detection; JSONL transcript channel (see Alternative B) + +--- + +## 6. Prior Art Learnings + +The full survey from the superseded proposal carries over unchanged in substance; the adopt-in targets below are normalized against Section 5 of this ADR. + +### OpenDray (`internal/session/`, Go) + +| Technique | What it does | Adopt in | +|---|---|---| +| Ring buffer with monotonic cursor (`ringbuf.go`) | Monotonic `written` byte counter; clients pass `since` on reconnect and receive only missed bytes; lag past capacity is reported explicitly as a gap | Phase 1 | +| Terminal-capability response filtering (`terminal_capabilities.go`) | Strips xterm.js auto-answers (DA/CPR/Status) from stdin at the PTY boundary; one chokepoint protects every client emulator from Ink-CLI startup breakage | Phase 1 | +| Pure lifecycle state machine (`transitions.go`) | Side-effect-free `(State, Event)` table; termination split into user-stop / self-exit / runtime-shutdown so restart reconciliation targets only the interrupted class | Phase 3 | +| Server-side virtual terminal (`pump.go` + vt10x) | PTY output feeds a headless VT emulator so notifications can snapshot the post-ANSI screen (Rust: `avt`, `vt100`) | Phase 4 | +| Idle detection -> notification pipeline (`pump.go`) | Output marks activity; a watcher fires an idle event with the last N lines as snippet | Phase 4 | +| TUI chrome filtering (`claude_chrome.go`, `term.go`) | Conservative regexes strip spinner/model-bar noise from notification snapshots | Phase 4 | +| JSONL transcript as a second channel (`claude_jsonl.go`) | Reads the agent's own transcript files as a structured side channel | Alternative B track | + +### Herdr (Rust) + +| Technique | What it does | Adopt in | +|---|---|---| +| Semantic agent state detection | Per-agent detection manifests classify panes as working/blocked/idle/done, with an explain API for rule provenance | Later (demand-gated) | +| Race-safe waits | Server-owned event-driven waits pinned to the pane occupant; atomic prompt+wait | Later (demand-gated) | +| Layered restore taxonomy | Live persistence / live handoff / native session restore / history replay (off by default: secrets) / layout-only snapshot | Phase 1 adopts the secrets-safe default and the recovery taxonomy | +| Multiple read projections | `visible` / `recent` / `recent-unwrapped` / `detection` views of one PTY | Phase 2/3 | +| Callback env injection | Spawned processes receive the runtime's socket path so in-pane agents can drive it | Later (demand-gated; A2A needs its own ADR) | + +### Claude Code cross-session messaging (v2.1.224+) + +| Technique | What it does | Adopt in | +|---|---|---| +| Per-session UDS inbox + filesystem discovery | Reachability boundary = filesystem visibility; container isolation falls out for free | Future A2A ADR | +| Deliberately small message contract | Plain-text summaries only, never history or files | Future A2A ADR | +| Permission-class trust model | Inbound messages cannot approve, reconfigure, or execute; deliver/hold derived from both sides' permission classes | Future A2A ADR | +| Own-child verification, dual-track | Process evidence where available, per-session token as first-line auth frame where not | Informs Phase 1 token design | +| Message-storm prevention | Read between turns, per-sender rate limits, dedupe, queue caps | Future A2A ADR | + +--- + +## 7. References + +- [PR #1477](https://github.com/openabdev/openab/pull/1477) — superseded in-process proposal; consolidated group-review rationale +- [portable-pty crate](https://crates.io/crates/portable-pty) — cross-platform PTY handling (wezterm project) +- [xterm.js](https://xtermjs.org/) — browser terminal renderer +- [OpenDray](https://opendray.dev/) — host-resident PTY session persistence (prior art, different security model) +- [Herdr](https://herdr.dev/) — agent multiplexer with semantic state detection (prior art, laptop-local) +- [Claude Code cross-session messaging](https://code.claude.com/docs/en/cross-session-messaging) — UDS inbox, trust model, loop throttling +- [ADR: ACP Server WebSocket (base)](./acp-server-websocket-base.md) — validated browser bearer-subprotocol auth and fail-closed listener guard reused here +- [ADR: configUrl over Helm rendering](./configurl-over-helm-rendering.md) — the config delivery pattern the two-view split builds on +- `docs/agentcore.md` — AgentCore's uVM PTY path; **non-goal boundary**: AgentCore runs *agents* in remote PTYs under its own runtime; `openab-pty` gives a *human* a terminal in the OAB workspace pod. Use AgentCore when you want managed agent execution; use `openab-pty` when you want hands-on control beside your ACP agents From b562c383d5d4ddb37726ee1a09e0f8a43cd9cb61 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 03:23:08 +0000 Subject: [PATCH 02/15] docs(adr): reframe openab-pty as composable runtime with three deployment profiles --- docs/adr/openab-pty-sidecar.md | 46 +++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 14 deletions(-) diff --git a/docs/adr/openab-pty-sidecar.md b/docs/adr/openab-pty-sidecar.md index b6b5f0be0..4802f72e8 100644 --- a/docs/adr/openab-pty-sidecar.md +++ b/docs/adr/openab-pty-sidecar.md @@ -1,4 +1,4 @@ -# ADR: openab-pty — Optional Sidecar Runtime for Remote Sandboxed Terminals +# ADR: openab-pty — Composable Runtime for Remote Sandboxed Terminals - **Status:** Proposed - **Date:** 2026-08-15 @@ -31,10 +31,24 @@ This ADR proposes the same capability in a shape that answers all five. ## 2. Decision -Ship **`openab-pty`**: a separate binary running as an **optional sidecar container** in the OAB pod. Not deployed by default. OAB remains a pure ACP broker; the sidecar owns everything terminal. +Ship **`openab-pty`**: a separate binary that is an **independently runnable runtime** — deployable standalone or colocated with the OAB broker. Not deployed by default. OAB remains a pure ACP broker; `openab-pty` owns everything terminal. + +**One codebase, two composable runtimes, three deployment modes:** + +| Profile | Processes | Use case | +|---|---|---| +| 1. ACP only (current default) | `openab` | Message-broker deployments; no change from today | +| 2. PTY only | `openab-pty` | Standalone remote terminal service: workspace PVC + `[pty]` config + PTY auth secret; no Discord/Slack tokens, no platform adapters, no ACP protocol | +| 3. ACP + PTY (colocated) | `openab` + `openab-pty` sidecar | Both in one pod sharing the workspace volume: drive a CLI by hand, let ACP agents continue in the same working tree from Discord | + +Deployment mechanics: + +- **Own image**: `ghcr.io/openabdev/openab-pty` — smaller than the broker image (no platform adapter dependencies) +- **Own Service/Ingress**: `/pty/*` routes to the `openab-pty` port in both profile 2 and 3; the broker listener never serves terminal traffic +- **Helm UX**: independent toggles (`openab.enabled` / `pty.enabled`) or a convenience `--set profile=acp|pty|full` ``` -K8s Pod +Profile 3 (colocated) — K8s Pod +--------------------------------------------------------------------+ | | | Container: openab (broker) Container: openab-pty | @@ -47,18 +61,21 @@ K8s Pod | | platform tokens, agents] | | [own config view: | | | +------------+--------------+ | [pty] section only] | | | | +-------------+-------------+ | -| | (optional, Phase 4) | | +| | (colocated profile only, Phase 4)| | | +<--- notification webhook ----------+ | | | | Shared: workspace volume (PVC) | NOT shared: credentials, | | | PID/cgroup, listeners | +--------------------------------------------------------------------+ - | | - Discord / Slack Web terminal (xterm.js) - (turn-based) Mobile/desktop terminal client + +Profile 2 (standalone) is the right half alone: openab-pty + workspace PVC. ``` -### Why sidecar (and what it fixes) +### Positioning statement for the standalone profile + +Profile 2 makes `openab-pty` a small standalone product in OpenDray's category (self-hosted persistent terminal sessions), differentiated by the K8s pod sandbox and the short-lived per-session token model. This is deliberate and bounded: `openab-pty` never grows platform adapters, agent orchestration, or memory features — users who need those deploy profile 3 and get them from the broker. This boundary is what keeps the OAB broker's thin-bridge identity untouched in every profile. + +### Why a separate runtime (and what it fixes) | Review blocker (PR #1477) | How the sidecar form resolves it | |---|---| @@ -66,19 +83,19 @@ K8s Pod | Same-pod blast radius | Separate container = separate PID namespace, cgroup, filesystem, and mounts. The shell user cannot signal the broker, exhaust its cgroup, or read its credential files. Broker platform tokens are **never mounted** into the sidecar | | Auth below capability | The sidecar designs its token model from scratch for shell-equivalent trust (see Security model) with no ACP-key coupling | | Pool incompatibility | The sidecar has its **own session manager** built for byte-stream lifecycle. No refactor of the shipped ACP pool; zero regression risk to the broker | -| Reversibility | Default-off sidecar with its own image/release. If demand does not materialize, deprecate the image; nothing in the broker to unwind. If demand proves out, later extraction of a shared lifecycle crate — or even single-process merge — remains open | +| Reversibility | Default-off runtime with its own image/release. If demand does not materialize, deprecate the image; nothing in the broker to unwind. If demand proves out, later extraction of a shared lifecycle crate — or even single-process merge — remains open | ### Coexistence with ACP ACP and PTY coexist per deployment, not per process: -- **Same pod, two containers** — one Helm toggle (`pty.enabled=true`) adds the sidecar; the broker container is byte-identical with or without it +- **Same pod, two containers (profile 3)** — one Helm toggle (`pty.enabled=true`) adds the sidecar; the broker container is byte-identical across all three profiles - **Shared workspace volume** — the PTY shell and ACP agents can see the same working tree (same PVC mount), which is the practical point of coexistence: drive a CLI by hand in the terminal, then let ACP agents continue in the same workspace from Discord - **Nothing else shared** — listeners, tokens, session state, and failure domains are independent; a crashed or compromised sidecar does not take the broker down ### Configuration: one source, two views -Operators keep a **single logical `config.toml`** (the existing `configUrl` flow); the two containers consume different projections of it: +Operators keep a **single logical `config.toml`** (the existing `configUrl` flow); the two runtimes consume different projections of it. In the standalone profile (PTY only), the same file format applies — `openab-pty` reads `[pty]` plus shared basics (workspace path, log level) and ignores everything else; no Discord/Slack tokens or ACP agent config are required or accepted. - The broker reads its existing sections; it ignores `[pty]` - The sidecar reads **only** `[pty]`; it must never receive platform tokens, because any secret mounted into the sidecar is readable by the human at the terminal (the shell runs in that container) @@ -188,7 +205,7 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Phase 2: Deployment + web client -- Helm: `pty.enabled` toggle adds the sidecar container, its port, and a NetworkPolicy example; config split documented per the configUrl pattern +- Helm: independent `openab.enabled` / `pty.enabled` toggles (or `--set profile=acp|pty|full`); standalone profile gets its own Service/Ingress (`/pty/*`) and NetworkPolicy example; config split documented per the configUrl pattern; `ghcr.io/openabdev/openab-pty` image published from the existing release pipeline - Minimal xterm.js page served by the sidecar; session list/create/kill endpoints (same auth bar as attach) - Rollback procedure: disabling the toggle drains (notify + grace) then kills sessions; broker unaffected @@ -197,9 +214,10 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Multi-viewer (one writer, N readers) with writer-lease semantics and read-only token scope - Reconnect backoff, richer capacity controls (per-token limits) -### Phase 4: Messaging bridge (optional) +### Phase 4: Messaging bridge (optional, colocated profile only) -- Sidecar posts a webhook to the broker when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); broker relays to the platform thread. Bridge is one-way and feature-gated +- `openab-pty` posts a webhook to the broker when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); broker relays to the platform thread. Bridge is one-way and feature-gated +- **Not available in the PTY-only profile** — there is no broker to relay through, and `openab-pty` will not grow its own notifier (that would recreate the scope creep this ADR exists to avoid). Users who want notifications deploy profile 3 ### Later (demand-gated, explicitly deferred) From b55d5fc690c2fbd508c435b9a59fd27ec26a1bc0 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:15:54 +0000 Subject: [PATCH 03/15] docs(adr): address round-1 review - isolation tiers, credential delivery contract, token control plane --- ...b-pty-sidecar.md => openab-pty-runtime.md} | 97 ++++++++++++------- 1 file changed, 62 insertions(+), 35 deletions(-) rename docs/adr/{openab-pty-sidecar.md => openab-pty-runtime.md} (60%) diff --git a/docs/adr/openab-pty-sidecar.md b/docs/adr/openab-pty-runtime.md similarity index 60% rename from docs/adr/openab-pty-sidecar.md rename to docs/adr/openab-pty-runtime.md index 4802f72e8..13aae4f4e 100644 --- a/docs/adr/openab-pty-sidecar.md +++ b/docs/adr/openab-pty-runtime.md @@ -15,7 +15,7 @@ A distinct user need exists that OAB's ACP model does not serve: > "I don't need multi-agent orchestration for this task. I have one or more coding CLIs (Claude Code, Codex, Kiro, or plain bash) and I want to drive them **directly** — full terminal, keyboard input, real-time output — from any device, with the session surviving my laptop." -Adjacent tools each carry a trade-off: Herdr is laptop-local (laptop dies, session dies), OpenDray is host-resident (shell shares host credentials). A **remote + sandboxed + raw-terminal** offering does not exist. +Adjacent tools each carry a trade-off: Herdr is laptop-local (laptop dies, session dies), OpenDray is host-resident (shell shares host credentials). Cloud IDEs and managed web terminals (Codespaces, Cloud Shell) serve related needs but are vendor-hosted. A **self-hostable, K8s-pod-sandboxed** raw-terminal offering -- one that can live beside your ACP agents' workspace -- does not exist. A previous proposal (PR #1477) made this a second in-process backend inside the OAB unified binary. Group review rejected that *form* — not the need — on five grounds: @@ -64,8 +64,8 @@ Profile 3 (colocated) — K8s Pod | | (colocated profile only, Phase 4)| | | +<--- notification webhook ----------+ | | | -| Shared: workspace volume (PVC) | NOT shared: credentials, | -| | PID/cgroup, listeners | +| Shared: workspace volume (PVC), | NOT shared: credentials, | +| pod network namespace, pod fate | PID/cgroup, filesystem mounts| +--------------------------------------------------------------------+ Profile 2 (standalone) is the right half alone: openab-pty + workspace PVC. @@ -80,7 +80,7 @@ Profile 2 makes `openab-pty` a small standalone product in OpenDray's category ( | Review blocker (PR #1477) | How the sidecar form resolves it | |---|---| | Positioning vs Thin Bridge | OAB binary is untouched; the broker stays a pure transport. `openab-pty` is an adjacent tool that shares deployment infrastructure only — no dual persona | -| Same-pod blast radius | Separate container = separate PID namespace, cgroup, filesystem, and mounts. The shell user cannot signal the broker, exhaust its cgroup, or read its credential files. Broker platform tokens are **never mounted** into the sidecar | +| Same-pod blast radius | Separate container = separate PID namespace, cgroup, filesystem, and mounts: the shell user cannot signal the broker, exhaust its container cgroup, or read its credential files. Broker platform tokens are **never mounted** into the PTY container. Residual sharing in the colocated profile (pod network namespace, pod fate) is graded honestly in Isolation tiers below; full isolation = profiles 1+2 as separate pods | | Auth below capability | The sidecar designs its token model from scratch for shell-equivalent trust (see Security model) with no ACP-key coupling | | Pool incompatibility | The sidecar has its **own session manager** built for byte-stream lifecycle. No refactor of the shipped ACP pool; zero regression risk to the broker | | Reversibility | Default-off runtime with its own image/release. If demand does not materialize, deprecate the image; nothing in the broker to unwind. If demand proves out, later extraction of a shared lifecycle crate — or even single-process merge — remains open | @@ -90,55 +90,79 @@ Profile 2 makes `openab-pty` a small standalone product in OpenDray's category ( ACP and PTY coexist per deployment, not per process: - **Same pod, two containers (profile 3)** — one Helm toggle (`pty.enabled=true`) adds the sidecar; the broker container is byte-identical across all three profiles -- **Shared workspace volume** — the PTY shell and ACP agents can see the same working tree (same PVC mount), which is the practical point of coexistence: drive a CLI by hand in the terminal, then let ACP agents continue in the same workspace from Discord -- **Nothing else shared** — listeners, tokens, session state, and failure domains are independent; a crashed or compromised sidecar does not take the broker down +- **Shared workspace volume (opt-in)** — the PTY shell and ACP agents can see the same working tree (same PVC mount), which is the practical point of coexistence: drive a CLI by hand in the terminal, then let ACP agents continue in the same workspace from Discord. This sharing is an **explicit cross-runtime trust and concurrency bridge**, stated rather than implied: + - *Trust*: the workspace is a single trust zone. Workspace-resident credentials (`.git/credentials`, `.env` files, agent OAuth stores) are readable by the shell regardless of mount hygiene, and either side can plant content (hooks, PATH-shadowing binaries) the other later executes. Treat ACP and PTY principals as sharing workspace authority when sharing is enabled + - *Concurrency*: concurrent writes are best-effort and uncoordinated (a runtime non-goal). Recommended convention: separate git worktrees or session directories per principal; document RWO/RWX PVC implications in the chart + - PTY cannot attach to a running ACP agent subprocess — the runtimes share files, never processes +- **Separated in every profile**: credential mounts, PID namespaces, cgroups, filesystems, tokens, and session state +- **Shared in profile 3 (accepted residual risk)**: the pod network namespace (containers reach each other on localhost; Kubernetes NetworkPolicy selects pods, not containers, and cannot block intra-pod traffic), pod scheduling/restart fate, and pod-level resource pressure. "Independent failure domains" is therefore NOT a property of profile 3 -- it is a property of profiles 1+2 -### Configuration: one source, two views +**Isolation tiers** (operators choose deliberately): -Operators keep a **single logical `config.toml`** (the existing `configUrl` flow); the two runtimes consume different projections of it. In the standalone profile (PTY only), the same file format applies — `openab-pty` reads `[pty]` plus shared basics (workspace path, log level) and ignores everything else; no Discord/Slack tokens or ACP agent config are required or accepted. +| Profile | Isolation tier | +|---|---| +| 1 + 2 as separate pods | Full: independent network identity, NetworkPolicy, failure domains -- **recommended for production when strong isolation is required** | +| 3 (colocated sidecar) | Partial: process/filesystem/credential-mount separation only; shared pod network and fate; the per-session token requirement and auth on every broker listener are the remaining intra-pod barriers. A convenience tier for teams that accept this trade for same-workspace ergonomics | + +### Configuration: one source, two projected views + +Operators keep a **single logical `config.toml`** (the existing `configUrl` flow); each runtime consumes a **projection materialized outside its trust boundary**. In the standalone profile (PTY only), the same file format applies -- `openab-pty` reads `[pty]` plus shared basics (workspace path, log level); no Discord/Slack tokens or ACP agent config are required or accepted. - The broker reads its existing sections; it ignores `[pty]` -- The sidecar reads **only** `[pty]`; it must never receive platform tokens, because any secret mounted into the sidecar is readable by the human at the terminal (the shell runs in that container) -- Delivery of the split follows the configUrl ADR: the chart (or operator) passes each container its own config source. For MVP this can be two URLs/objects derived from one source of truth; a `openab-pty run -c --section pty` style filter is an acceptable alternative -- `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY auth material is sourced via the secrets resolver per `secrets-management.md`, not a raw env var +- The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only +- **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (`automountServiceAccountToken: false`; note IRSA identity is pod-level, so the PTY signing secret is fetched via a narrowly-scoped mechanism, not the broker's role), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) +- `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY-only projection's `[secrets.refs]` contains exactly one entry -- the signing key ```toml -# one config.toml — two consumers -[discord] # broker only — never mounted into the sidecar +# one logical config.toml -- projected into two filtered views; +# neither container ever mounts the other's sections + +[secrets.refs] # PTY projection carries ONLY this entry +pty_signing_key = "aws-sm://openab/pty-signing-key#value" + +[discord] # broker view only -- never delivered to the PTY runtime bot_token = "${DISCORD_BOT_TOKEN}" -[agent] # broker only +[agent] # broker view only # ... -[pty] # sidecar only +[pty] # PTY view only enabled = true -listen = "0.0.0.0:8090" # separate port -> separate NetworkPolicy -command = "/bin/bash" # operator-configured; never client-specified +listen = "0.0.0.0:8090" # own port; TLS contract below +command = "/bin/bash" # operator-configured; never client-specified max_sessions = 4 -absolute_session_ttl = "12h" # applies even while attached -scrollback_kib = 1024 # in-memory only; cleared on teardown -scrollback_replay = false # off by default (secrets-safe) -auth_secret_ref = "aws-sm://openab/pty-signing-key" +absolute_session_ttl = "12h" # applies even while attached +scrollback_kib = 1024 # in-memory only; cleared on teardown +scrollback_replay = false # governs fresh-attach full-history dump only (see lifecycle) +auth_secret = "${secrets.pty_signing_key}" ``` ### Security model -- **Transport**: WSS required; plain WS permitted only on loopback for local dev. Fail-closed: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) -- **Browser credential transport**: reuse the validated `/acp` scheme — `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); origin policy and constant-time comparison carry over -- **Token model**: short-lived per-session tokens minted from the configured signing secret (attach = present a token scoped to one session name with an expiry), replacing the static-shared-key model the review rejected. Revocation = rotate the signing secret. An identity layer is explicitly out of MVP scope; the ADR states this per `identity-trust-none.md` rather than implying otherwise +- **Transport / TLS contract**: WSS is mandatory for external clients. Two supported terminations, chosen explicitly per deployment: (a) `openab-pty` terminates TLS itself (cert mounted into the container), or (b) a trusted Ingress terminates TLS and forwards plain WS internally -- in which case the internal listener accepts non-loopback plain WS only when the deployment declares `tls_terminated_upstream = true`, and the residual internal-hop exposure is documented. Fail-closed in all cases: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) +- **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); origin policy and constant-time comparison carry over +- **Token control plane** (MVP model; an identity layer remains explicitly out of scope per `identity-trust-none.md`): + - **Create requires authentication**: sessions are created by the operator via a loopback/Unix-socket CLI (`openab-pty session create `) or an admin bootstrap credential. Unauthenticated remote create/list/kill is NOT provided in MVP + - **One-time issuance at creation**: creating a session mints an immutable `generation`, signs one scoped attach token (claims: session ID + generation, audience, action scope, expiry), and returns it exactly once + - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface + - **Per-session revocation**: kill/recreate bumps the generation, immediately invalidating outstanding tokens for that session; signing-key rotation remains the global escape hatch and supports an old/new overlap window for zero-downtime rotation + - **Format direction**: HMAC-SHA256 opaque token; token expiry defaults shorter than the session TTL (re-issue via the authenticated create/renew path) - **Command authority**: the spawned command is operator configuration only; clients can never specify it. Session names are allowlist-validated (`[a-z0-9-]{1,32}`) -- **Isolation**: the sidecar container mounts the workspace volume and its own config view; no service-account token, no broker config, no platform secrets. NetworkPolicy can (and the chart docs will recommend) restrict sidecar egress independently of the broker +- **Isolation**: the PTY container mounts only the workspace volume (workspace-scoped, never the broker HOME PVC) and its own config projection; no service-account token, no broker config, no platform secrets (see the deployment contract above). NetworkPolicy applies at pod scope: it can restrict the standalone profile's pod independently; in the colocated profile it cannot separate the two containers (see Isolation tiers) +- **Container defaults**: the `openab-pty` image runs as a non-root user (UID 1000), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable mount +- **Rate limiting in MVP**: per-IP WS-upgrade failure limits (e.g. 5 failures/min then a short ban) ship in Phase 1 -- audit is detection, rate limiting is prevention - **Audit in MVP**: attach/detach, session create/kill, and auth failures are logged from Phase 1; a leaked token must be observable - **Env**: the PTY child gets an explicit allowlist (TERM, LANG/LC_*, PATH, HOME, USER, SHELL) and nothing else; `OPENAB_*` and cloud-credential variables are never inherited -### Session lifecycle (owned by the sidecar, designed for byte streams) +### Session lifecycle (owned by `openab-pty`, designed for byte streams) -- **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong; a half-open socket counts as detached after the ping timeout) -- **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached — capacity cannot be pinned forever by an open browser tab +- **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong at a 15-30s interval; a half-open socket counts as detached after 2-3 missed pings -- exact values are Phase 1 config with these recommended defaults, balancing flaky mobile networks against dead-client slot pinning) +- **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached -- capacity cannot be pinned forever by an open browser tab. Expiry is client-visible: a warning control frame precedes forced teardown, and the WebSocket closes with a distinct close code so clients surface "session expired" instead of retrying a network error - **Attach semantics (MVP)**: single-attach exclusive; a second attach with a valid token detaches the first (documented; multi-viewer is Phase 3) -- **Reconnect**: monotonic byte cursor from day one — the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes, with an explicit gap signal on overflow (fresh attach = terminal reset + full replay only when `scrollback_replay=true`) +- **Reconnect**: monotonic byte cursor from day one -- the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes. The replay-to-live handoff is **atomic**: the subscriber registers under the buffer lock, captures the end offset, replays through it, then drains queued live bytes -- with connection-generation fencing so teardown of a replaced connection cannot affect its successor. On overflow the server sends an explicit `gap` control frame (bytes-dropped count) so the client can trigger a full clear/redraw instead of rendering a sliced ANSI stream +- **`scrollback_replay` vs cursor semantics** (distinct controls): incremental `since` replay is always available within the ring buffer's retention; `scrollback_replay` governs only the cursor-less full-history dump on a fresh attach (default off -- secrets-safe); setting `scrollback_kib = 0` disables retention entirely, which also disables `since` replay (every reconnect starts with a `gap` + reset) - **Teardown**: setpgid on spawn; SIGTERM-grace-SIGKILL escalation on the process group; evict-while-attached order = notify client, close socket, kill group, close master fd, release slot; buffers cleared on teardown; scrollback never touches disk -- **Recovery taxonomy** (stated, not implied): detach/reattach survives (process alive); pod restart does not (process dead) — reattach-to-dead returns a distinct error and offers restart-in-place. Pod-lifetime durability is out of scope and documented as such +- **Recovery taxonomy** (stated, not implied): detach/reattach survives (process alive); pod restart does not (process dead) -- reattach-to-dead returns a distinct error and offers **restart-in-place**: same session name, a fresh process and a new generation (old tokens invalid, empty scrollback). Pod-lifetime durability is out of scope and documented as such --- @@ -194,11 +218,12 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Phase 1: `openab-pty` MVP (new crate, new binary) - Own session manager: named sessions, operator-configured command, allowlist-validated names +- **Session bootstrap**: sessions are created via the authenticated loopback/UDS operator CLI (`openab-pty session create `), which spawns the PTY and returns the one-time attach token; `GET /pty/{session}` is attach-only. No remote create/list/kill in Phase 1 (Phase 2 adds them behind admin auth) - portable-pty spawner with setpgid, escalating kill, and the teardown order above -- `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`) with a defined close-code table -- Auth: per-session tokens from the signing secret; fail-closed off-loopback; `/acp`-style browser subprotocol transport -- Monotonic cursor reconnect with gap signaling; scrollback in-memory, off-by-default replay, cleared on teardown -- Detached-idle TTL + absolute lifetime cap; single-attach exclusive +- `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`, `gap`, `ttl-warning`) with a defined close-code table +- Auth: the token control plane above (authenticated create, one-time issuance bound to session generation, attach-only verification); fail-closed off-loopback; `/acp`-style browser subprotocol transport; per-IP upgrade-failure rate limiting +- Monotonic cursor reconnect with atomic replay/live handoff and gap signaling; scrollback in-memory, off-by-default fresh-attach replay, cleared on teardown +- Detached-idle TTL + absolute lifetime cap (with client-visible expiry warning + close code); single-attach exclusive - Audit log (attach/detach/create/kill/auth-failure) and basic metrics - Resize propagation (TIOCSWINSZ) including attach-time initial size - Terminal-capability response filtering at the PTY boundary (known Ink-CLI startup breakage) @@ -217,11 +242,13 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Phase 4: Messaging bridge (optional, colocated profile only) - `openab-pty` posts a webhook to the broker when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); broker relays to the platform thread. Bridge is one-way and feature-gated +- **Bridge authentication is required by design, stated now**: the webhook carries an HMAC signature from a dedicated bridge secret delivered at deploy time (or travels over a loopback/UDS-only endpoint) -- separate from platform tokens and from the PTY signing key, and never usable as a PTY identity plane. A compromised PTY runtime must not be able to inject arbitrary notifications - **Not available in the PTY-only profile** — there is no broker to relay through, and `openab-pty` will not grow its own notifier (that would recreate the scope creep this ADR exists to avoid). Users who want notifications deploy profile 3 ### Later (demand-gated, explicitly deferred) -- Shared lifecycle crate extraction (if the ACP pool and PTY manager converge naturally) +- Shared lifecycle crate extraction (if the ACP pool and PTY manager converge naturally). Candidate shared surface: spawn mechanics, env-allowlist construction, pgid kill/escalation; deliberately NOT shared: liveness definitions, TTL/eviction policy, persistence +- **Adoption review point**: 12 months after the standalone profile ships, review its usage; below a threshold the maintainers set then, consider deprecating the standalone image or folding PTY back to colocate-only - Single-process merge (only if operations prove the sidecar split is more cost than benefit) - Identity layer for PTY tokens; semantic agent-state detection; JSONL transcript channel (see Alternative B) From 50085af267e4db75d83ac38ebc0d529db1e7682d Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:22:30 +0000 Subject: [PATCH 04/15] docs(adr): round-2 fixes - admin credential plane, key delivery MUST, attach CAS, frame validation --- docs/adr/openab-pty-runtime.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index 13aae4f4e..f91d4a08b 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -110,7 +110,8 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - The broker reads its existing sections; it ignores `[pty]` - The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only -- **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (`automountServiceAccountToken: false`; note IRSA identity is pod-level, so the PTY signing secret is fetched via a narrowly-scoped mechanism, not the broker's role), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) +- **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (set `automountServiceAccountToken: false` at **pod** level -- it is not per-container -- and, when the broker needs IRSA/configUrl credentials in the colocated profile, project an audience-scoped token volume **into the broker container only**), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) +- **Signing-key delivery (MUST)**: the PTY signing key is materialized at deploy time (operator/Helm/CI resolves the `aws-sm://` reference) and mounted read-only (0400) at a path readable by the runtime process but not by the PTY child; the PTY container never performs runtime cloud-secret fetches and holds no cloud identity. The key never enters the child's environment, logs, or core dumps (dumpable disabled) - `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY-only projection's `[secrets.refs]` contains exactly one entry -- the signing key ```toml @@ -129,6 +130,7 @@ bot_token = "${DISCORD_BOT_TOKEN}" [pty] # PTY view only enabled = true listen = "0.0.0.0:8090" # own port; TLS contract below +tls_terminated_upstream = false # true only behind a trusted TLS-terminating Ingress command = "/bin/bash" # operator-configured; never client-specified max_sessions = 4 absolute_session_ttl = "12h" # applies even while attached @@ -142,8 +144,8 @@ auth_secret = "${secrets.pty_signing_key}" - **Transport / TLS contract**: WSS is mandatory for external clients. Two supported terminations, chosen explicitly per deployment: (a) `openab-pty` terminates TLS itself (cert mounted into the container), or (b) a trusted Ingress terminates TLS and forwards plain WS internally -- in which case the internal listener accepts non-loopback plain WS only when the deployment declares `tls_terminated_upstream = true`, and the residual internal-hop exposure is documented. Fail-closed in all cases: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) - **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); origin policy and constant-time comparison carry over - **Token control plane** (MVP model; an identity layer remains explicitly out of scope per `identity-trust-none.md`): - - **Create requires authentication**: sessions are created by the operator via a loopback/Unix-socket CLI (`openab-pty session create `) or an admin bootstrap credential. Unauthenticated remote create/list/kill is NOT provided in MVP - - **One-time issuance at creation**: creating a session mints an immutable `generation`, signs one scoped attach token (claims: session ID + generation, audience, action scope, expiry), and returns it exactly once + - **Create requires authentication -- and locality is not authentication**: the shell child may share the container (and potentially the UID) with the runtime, so a loopback/UDS endpoint alone is not an auth barrier. MVP mechanism: session creation (`openab-pty session create `) requires an **admin bootstrap credential** delivered to the operator at deploy time and never present in the PTY child's environment or filesystem view. Hardening alternative (non-MVP): a distinct privileged UID/group for the runtime with UDS peer-credential verification. Unauthenticated remote create/list/kill is NOT provided in MVP; the admin credential and the attach token are distinct planes -- an attach token can never create, list, or kill + - **One-time issuance at creation**: creating a session mints an immutable `generation`, signs one scoped attach token (claims: session ID + generation, audience, action scope = attach only, expiry), and returns it exactly once. **Issued once, not single-use**: the bearer token remains valid for reattach until its expiry or a generation bump -- reconnecting clients are not locked out; theft exposure is bounded by a short default TTL (well below the session TTL) and renewal happens only through the authenticated create/renew path - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface - **Per-session revocation**: kill/recreate bumps the generation, immediately invalidating outstanding tokens for that session; signing-key rotation remains the global escape hatch and supports an old/new overlap window for zero-downtime rotation - **Format direction**: HMAC-SHA256 opaque token; token expiry defaults shorter than the session TTL (re-issue via the authenticated create/renew path) @@ -158,7 +160,7 @@ auth_secret = "${secrets.pty_signing_key}" - **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong at a 15-30s interval; a half-open socket counts as detached after 2-3 missed pings -- exact values are Phase 1 config with these recommended defaults, balancing flaky mobile networks against dead-client slot pinning) - **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached -- capacity cannot be pinned forever by an open browser tab. Expiry is client-visible: a warning control frame precedes forced teardown, and the WebSocket closes with a distinct close code so clients surface "session expired" instead of retrying a network error -- **Attach semantics (MVP)**: single-attach exclusive; a second attach with a valid token detaches the first (documented; multi-viewer is Phase 3) +- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3) - **Reconnect**: monotonic byte cursor from day one -- the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes. The replay-to-live handoff is **atomic**: the subscriber registers under the buffer lock, captures the end offset, replays through it, then drains queued live bytes -- with connection-generation fencing so teardown of a replaced connection cannot affect its successor. On overflow the server sends an explicit `gap` control frame (bytes-dropped count) so the client can trigger a full clear/redraw instead of rendering a sliced ANSI stream - **`scrollback_replay` vs cursor semantics** (distinct controls): incremental `since` replay is always available within the ring buffer's retention; `scrollback_replay` governs only the cursor-less full-history dump on a fresh attach (default off -- secrets-safe); setting `scrollback_kib = 0` disables retention entirely, which also disables `since` replay (every reconnect starts with a `gap` + reset) - **Teardown**: setpgid on spawn; SIGTERM-grace-SIGKILL escalation on the process group; evict-while-attached order = notify client, close socket, kill group, close master fd, release slot; buffers cleared on teardown; scrollback never touches disk @@ -173,7 +175,7 @@ auth_secret = "${secrets.pty_signing_key}" - OAB keeps its thin-broker identity untouched — zero changes to the shipped binary, pool, or ACP path - Fills the remote + sandboxed + raw-terminal quadrant with a real container boundary instead of a claimed one - Highest reversibility: default-off, separately versioned, separately deprecable -- Coexistence where it matters (shared workspace) without shared failure or credential domains +- Coexistence where it matters (shared workspace) without shared process or credential-mount domains; network namespace and pod fate are shared only in the colocated profile (see Isolation tiers) - The Phase 4 notification bridge (sidecar webhook -> broker -> Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes ### Negative @@ -220,7 +222,8 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Own session manager: named sessions, operator-configured command, allowlist-validated names - **Session bootstrap**: sessions are created via the authenticated loopback/UDS operator CLI (`openab-pty session create `), which spawns the PTY and returns the one-time attach token; `GET /pty/{session}` is attach-only. No remote create/list/kill in Phase 1 (Phase 2 adds them behind admin auth) - portable-pty spawner with setpgid, escalating kill, and the teardown order above -- `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`, `gap`, `ttl-warning`) with a defined close-code table +- `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`, `gap`, `ttl-warning`) with a defined close-code table. Frame validation is strict allowlist: bounded max frame size, unknown control types rejected, resize values bounds-checked; malformed frames count toward an abuse metric and can disconnect +- Input backpressure: per-connection write watermark toward the PTY master; a client exceeding it is disconnected (fail closed) rather than growing unbounded queues or stalling the reader - Auth: the token control plane above (authenticated create, one-time issuance bound to session generation, attach-only verification); fail-closed off-loopback; `/acp`-style browser subprotocol transport; per-IP upgrade-failure rate limiting - Monotonic cursor reconnect with atomic replay/live handoff and gap signaling; scrollback in-memory, off-by-default fresh-attach replay, cleared on teardown - Detached-idle TTL + absolute lifetime cap (with client-visible expiry warning + close code); single-attach exclusive @@ -231,7 +234,7 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Phase 2: Deployment + web client - Helm: independent `openab.enabled` / `pty.enabled` toggles (or `--set profile=acp|pty|full`); standalone profile gets its own Service/Ingress (`/pty/*`) and NetworkPolicy example; config split documented per the configUrl pattern; `ghcr.io/openabdev/openab-pty` image published from the existing release pipeline -- Minimal xterm.js page served by the sidecar; session list/create/kill endpoints (same auth bar as attach) +- Minimal xterm.js page served by `openab-pty`; remote session list/create/kill endpoints gated by the **admin bootstrap credential** (the same control-plane contract as the Phase 1 CLI -- never attach tokens; attach-token scope remains attach-only) - Rollback procedure: disabling the toggle drains (notify + grace) then kills sessions; broker unaffected ### Phase 3: Lifecycle hardening From 7ad09ccd9535ca643968548ac67d7fb1ab3eeac4 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:24:27 +0000 Subject: [PATCH 05/15] docs(adr): mandate runtime/child UID privilege separation for key and credential protection --- docs/adr/openab-pty-runtime.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index f91d4a08b..574c292ac 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -112,6 +112,7 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only - **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (set `automountServiceAccountToken: false` at **pod** level -- it is not per-container -- and, when the broker needs IRSA/configUrl credentials in the colocated profile, project an audience-scoped token volume **into the broker container only**), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) - **Signing-key delivery (MUST)**: the PTY signing key is materialized at deploy time (operator/Helm/CI resolves the `aws-sm://` reference) and mounted read-only (0400) at a path readable by the runtime process but not by the PTY child; the PTY container never performs runtime cloud-secret fetches and holds no cloud identity. The key never enters the child's environment, logs, or core dumps (dumpable disabled) +- **Runtime/child privilege separation (MUST)**: file permissions alone cannot protect the key or the admin credential if the runtime and the shell child share a UID. The runtime process runs as its own UID (e.g. 1000) and **spawns every PTY child under a distinct unprivileged UID** (e.g. 1001); the signing key, admin-credential material, and control socket are owned by the runtime UID with mode 0400/0700, and the runtime sets `PR_SET_DUMPABLE=0` so the child cannot read them via the filesystem or `/proc`. Deployments that cannot provide two UIDs in the container must instead move signing/minting to an external control-plane service -- same-UID with path obscurity is NOT an accepted configuration - `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY-only projection's `[secrets.refs]` contains exactly one entry -- the signing key ```toml @@ -151,7 +152,7 @@ auth_secret = "${secrets.pty_signing_key}" - **Format direction**: HMAC-SHA256 opaque token; token expiry defaults shorter than the session TTL (re-issue via the authenticated create/renew path) - **Command authority**: the spawned command is operator configuration only; clients can never specify it. Session names are allowlist-validated (`[a-z0-9-]{1,32}`) - **Isolation**: the PTY container mounts only the workspace volume (workspace-scoped, never the broker HOME PVC) and its own config projection; no service-account token, no broker config, no platform secrets (see the deployment contract above). NetworkPolicy applies at pod scope: it can restrict the standalone profile's pod independently; in the colocated profile it cannot separate the two containers (see Isolation tiers) -- **Container defaults**: the `openab-pty` image runs as a non-root user (UID 1000), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable mount +- **Container defaults**: the `openab-pty` image runs non-root (runtime UID 1000, PTY children under the distinct unprivileged UID per the separation MUST above), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable mount - **Rate limiting in MVP**: per-IP WS-upgrade failure limits (e.g. 5 failures/min then a short ban) ship in Phase 1 -- audit is detection, rate limiting is prevention - **Audit in MVP**: attach/detach, session create/kill, and auth failures are logged from Phase 1; a leaked token must be observable - **Env**: the PTY child gets an explicit allowlist (TERM, LANG/LC_*, PATH, HOME, USER, SHELL) and nothing else; `OPENAB_*` and cloud-credential variables are never inherited From d4df522a458ad728d4896124d978870dd27ea1a4 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:27:37 +0000 Subject: [PATCH 06/15] docs(adr): keyless MVP token model - CSPRNG opaque tokens with in-memory hashes, no signing key --- docs/adr/openab-pty-runtime.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index 574c292ac..a22c970eb 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -111,16 +111,19 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - The broker reads its existing sections; it ignores `[pty]` - The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only - **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (set `automountServiceAccountToken: false` at **pod** level -- it is not per-container -- and, when the broker needs IRSA/configUrl credentials in the colocated profile, project an audience-scoped token volume **into the broker container only**), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) -- **Signing-key delivery (MUST)**: the PTY signing key is materialized at deploy time (operator/Helm/CI resolves the `aws-sm://` reference) and mounted read-only (0400) at a path readable by the runtime process but not by the PTY child; the PTY container never performs runtime cloud-secret fetches and holds no cloud identity. The key never enters the child's environment, logs, or core dumps (dumpable disabled) -- **Runtime/child privilege separation (MUST)**: file permissions alone cannot protect the key or the admin credential if the runtime and the shell child share a UID. The runtime process runs as its own UID (e.g. 1000) and **spawns every PTY child under a distinct unprivileged UID** (e.g. 1001); the signing key, admin-credential material, and control socket are owned by the runtime UID with mode 0400/0700, and the runtime sets `PR_SET_DUMPABLE=0` so the child cannot read them via the filesystem or `/proc`. Deployments that cannot provide two UIDs in the container must instead move signing/minting to an external control-plane service -- same-UID with path obscurity is NOT an accepted configuration +- **Credential-material delivery (MUST)**: MVP has no signing key to deliver (see Token format). The only secret the PTY runtime holds is the **hash of the admin bootstrap credential**, and its delivery follows this rule: **external tooling (operator/Helm/CI) resolves any logical `aws-sm://` reference at deploy time and materializes the literal value into the delivered projection -- `openab-pty` itself never resolves cloud references at runtime and holds no cloud identity.** Delivered material is owned by the runtime UID, mode 0400, never enters the child's environment, logs, or core dumps (dumpable disabled) +- **Runtime/child privilege separation (MUST)**: file permissions alone cannot protect credential material if the runtime and the shell child share a UID. The runtime process runs as its own UID (e.g. 1000) and **spawns every PTY child under a distinct unprivileged UID** (e.g. 1001); the admin-credential material, in-memory token state, and control socket are owned by the runtime UID with mode 0400/0700, and the runtime sets `PR_SET_DUMPABLE=0` so the child cannot read them via the filesystem or `/proc`. Deployments that cannot provide two UIDs in the container must move session creation to an external control-plane channel -- same-UID with path obscurity is NOT an accepted configuration - `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY-only projection's `[secrets.refs]` contains exactly one entry -- the signing key ```toml # one logical config.toml -- projected into two filtered views; # neither container ever mounts the other's sections -[secrets.refs] # PTY projection carries ONLY this entry -pty_signing_key = "aws-sm://openab/pty-signing-key#value" +[secrets.refs] # LOGICAL operator source only. Deploy tooling + # materializes the literal value into the delivered + # PTY projection -- openab-pty never resolves + # aws-sm:// itself and holds no cloud identity +pty_admin_hash = "aws-sm://openab/pty-admin#hash" [discord] # broker view only -- never delivered to the PTY runtime bot_token = "${DISCORD_BOT_TOKEN}" @@ -137,7 +140,7 @@ max_sessions = 4 absolute_session_ttl = "12h" # applies even while attached scrollback_kib = 1024 # in-memory only; cleared on teardown scrollback_replay = false # governs fresh-attach full-history dump only (see lifecycle) -auth_secret = "${secrets.pty_signing_key}" +admin_credential_hash = "${secrets.pty_admin_hash}" # verifies the admin bootstrap credential; attach tokens are in-memory only (no key) ``` ### Security model @@ -146,10 +149,10 @@ auth_secret = "${secrets.pty_signing_key}" - **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); origin policy and constant-time comparison carry over - **Token control plane** (MVP model; an identity layer remains explicitly out of scope per `identity-trust-none.md`): - **Create requires authentication -- and locality is not authentication**: the shell child may share the container (and potentially the UID) with the runtime, so a loopback/UDS endpoint alone is not an auth barrier. MVP mechanism: session creation (`openab-pty session create `) requires an **admin bootstrap credential** delivered to the operator at deploy time and never present in the PTY child's environment or filesystem view. Hardening alternative (non-MVP): a distinct privileged UID/group for the runtime with UDS peer-credential verification. Unauthenticated remote create/list/kill is NOT provided in MVP; the admin credential and the attach token are distinct planes -- an attach token can never create, list, or kill - - **One-time issuance at creation**: creating a session mints an immutable `generation`, signs one scoped attach token (claims: session ID + generation, audience, action scope = attach only, expiry), and returns it exactly once. **Issued once, not single-use**: the bearer token remains valid for reattach until its expiry or a generation bump -- reconnecting clients are not locked out; theft exposure is bounded by a short default TTL (well below the session TTL) and renewal happens only through the authenticated create/renew path + - **One-time issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, returned exactly once. **Issued once, not single-use**: the bearer token remains valid for reattach until its expiry or a generation bump -- reconnecting clients are not locked out; theft exposure is bounded by a short default TTL (well below the session TTL) and renewal happens only through the authenticated create/renew path - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface - - **Per-session revocation**: kill/recreate bumps the generation, immediately invalidating outstanding tokens for that session; signing-key rotation remains the global escape hatch and supports an old/new overlap window for zero-downtime rotation - - **Format direction**: HMAC-SHA256 opaque token; token expiry defaults shorter than the session TTL (re-issue via the authenticated create/renew path) + - **Per-session revocation**: kill/recreate bumps the generation and deletes the stored token hash, immediately invalidating outstanding tokens for that session; runtime restart clears all token state (sessions die with the process anyway, so this is not a loss) + - **Token format (MVP): no signing key exists.** Each attach token is a CSPRNG 256-bit opaque bearer value; the runtime stores only its hash together with `(session ID, generation, scope = attach-only, expiry)` in memory and deletes it on kill/expiry. Because sessions deliberately do not survive a runtime restart, self-contained signed tokens buy nothing in MVP -- and eliminating the signing key eliminates the minting authority a same-container shell could steal. Signed (HMAC) tokens are a later option and require either an external signer outside the PTY container or the runtime/child privilege boundary below - **Command authority**: the spawned command is operator configuration only; clients can never specify it. Session names are allowlist-validated (`[a-z0-9-]{1,32}`) - **Isolation**: the PTY container mounts only the workspace volume (workspace-scoped, never the broker HOME PVC) and its own config projection; no service-account token, no broker config, no platform secrets (see the deployment contract above). NetworkPolicy applies at pod scope: it can restrict the standalone profile's pod independently; in the colocated profile it cannot separate the two containers (see Isolation tiers) - **Container defaults**: the `openab-pty` image runs non-root (runtime UID 1000, PTY children under the distinct unprivileged UID per the separation MUST above), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable mount From 28ed8747c17492667920deb4ea317eeacf73bf6b Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 11:29:37 +0000 Subject: [PATCH 07/15] docs(adr): ground MVP containment in keyless model; demote child-UID separation to optional hardening --- docs/adr/openab-pty-runtime.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index a22c970eb..c694e5b6d 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -112,8 +112,8 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only - **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (set `automountServiceAccountToken: false` at **pod** level -- it is not per-container -- and, when the broker needs IRSA/configUrl credentials in the colocated profile, project an audience-scoped token volume **into the broker container only**), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) - **Credential-material delivery (MUST)**: MVP has no signing key to deliver (see Token format). The only secret the PTY runtime holds is the **hash of the admin bootstrap credential**, and its delivery follows this rule: **external tooling (operator/Helm/CI) resolves any logical `aws-sm://` reference at deploy time and materializes the literal value into the delivered projection -- `openab-pty` itself never resolves cloud references at runtime and holds no cloud identity.** Delivered material is owned by the runtime UID, mode 0400, never enters the child's environment, logs, or core dumps (dumpable disabled) -- **Runtime/child privilege separation (MUST)**: file permissions alone cannot protect credential material if the runtime and the shell child share a UID. The runtime process runs as its own UID (e.g. 1000) and **spawns every PTY child under a distinct unprivileged UID** (e.g. 1001); the admin-credential material, in-memory token state, and control socket are owned by the runtime UID with mode 0400/0700, and the runtime sets `PR_SET_DUMPABLE=0` so the child cannot read them via the filesystem or `/proc`. Deployments that cannot provide two UIDs in the container must move session creation to an external control-plane channel -- same-UID with path obscurity is NOT an accepted configuration -- `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY-only projection's `[secrets.refs]` contains exactly one entry -- the signing key +- **Same-container containment (MVP model)**: with the keyless token design, runtime state contains no minting authority to steal -- only non-reversible hashes. The security basis is: the admin bootstrap credential is CSPRNG high-entropy and the runtime stores **only its verifier hash** (reading the hash neither authorizes requests nor enables practical offline guessing); every control operation is authenticated by presenting the credential itself, even on a locally reachable socket; and the runtime sets `PR_SET_DUMPABLE=0`. **Optional hardening (not MVP)**: spawning PTY children under a distinct unprivileged UID gives defense-in-depth, but a non-root UID-1000 process cannot `setuid` without `CAP_SETUID` or a privileged launcher -- deployments that can provide a narrowly-scoped launcher may adopt it; the default container contract (non-root, all capabilities dropped, `allowPrivilegeEscalation: false`) is preserved either way +- `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY-only projection's `[secrets.refs]` contains exactly one entry -- the admin credential hash ```toml # one logical config.toml -- projected into two filtered views; @@ -155,7 +155,7 @@ admin_credential_hash = "${secrets.pty_admin_hash}" # verifies the admin bootst - **Token format (MVP): no signing key exists.** Each attach token is a CSPRNG 256-bit opaque bearer value; the runtime stores only its hash together with `(session ID, generation, scope = attach-only, expiry)` in memory and deletes it on kill/expiry. Because sessions deliberately do not survive a runtime restart, self-contained signed tokens buy nothing in MVP -- and eliminating the signing key eliminates the minting authority a same-container shell could steal. Signed (HMAC) tokens are a later option and require either an external signer outside the PTY container or the runtime/child privilege boundary below - **Command authority**: the spawned command is operator configuration only; clients can never specify it. Session names are allowlist-validated (`[a-z0-9-]{1,32}`) - **Isolation**: the PTY container mounts only the workspace volume (workspace-scoped, never the broker HOME PVC) and its own config projection; no service-account token, no broker config, no platform secrets (see the deployment contract above). NetworkPolicy applies at pod scope: it can restrict the standalone profile's pod independently; in the colocated profile it cannot separate the two containers (see Isolation tiers) -- **Container defaults**: the `openab-pty` image runs non-root (runtime UID 1000, PTY children under the distinct unprivileged UID per the separation MUST above), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable mount +- **Container defaults**: the `openab-pty` image runs as a non-root user (UID 1000), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable mount (child UID separation is optional hardening per Same-container containment above) - **Rate limiting in MVP**: per-IP WS-upgrade failure limits (e.g. 5 failures/min then a short ban) ship in Phase 1 -- audit is detection, rate limiting is prevention - **Audit in MVP**: attach/detach, session create/kill, and auth failures are logged from Phase 1; a leaked token must be observable - **Env**: the PTY child gets an explicit allowlist (TERM, LANG/LC_*, PATH, HOME, USER, SHELL) and nothing else; `OPENAB_*` and cloud-credential variables are never inherited From 7b54b30633648d6117cfa7f5b8a23c2aa96b11c8 Mon Sep 17 00:00:00 2001 From: chaodu-agent Date: Sat, 15 Aug 2026 16:09:04 +0000 Subject: [PATCH 08/15] docs(adr): address round-3 findings F1-F7 - F1: resolution asymmetry -- broker resolves [secrets.refs]; PTY runtime fails closed on any [secrets.refs] table or unresolved cloud reference; config example split into logical source vs delivered projection - F2: Phase 4 bridge redesigned as broker-pull over pod-local loopback -- no bridge secret ever enters the PTY container; events are display-only rate-limited hints; push+HMAC allowed only with an external signer or the runtime/child privilege boundary (non-MVP) - F3: hard kill domain MUST -- per-session cgroup.kill or pidfd descendant reaper; pgid is the first signal path, not the containment guarantee - F4: same-UID residual risks converted to a MUST checklist (read-only mounts, constant-time verify, token zeroize, 128-bit admin entropy floor, Linux-only PR_SET_DUMPABLE assumption, accepted SIGKILL risk) - F5: filesystem layout specified (/run/openab-pty runtime-only, /etc/openab-pty read-only projection, workspace as sole writable mount) - F6: session renew defined (admin-authenticated, process survives, generation bump, one-time token) and added to the Phase 1 CLI - F7: stale pre-keyless terms swept (PTY auth secret -> admin bootstrap credential; Phase 4 signing-key mention removed) --- docs/adr/openab-pty-runtime.md | 64 +++++++++++++++++++++++++--------- 1 file changed, 48 insertions(+), 16 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index c694e5b6d..b9a52acfe 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -38,7 +38,7 @@ Ship **`openab-pty`**: a separate binary that is an **independently runnable run | Profile | Processes | Use case | |---|---|---| | 1. ACP only (current default) | `openab` | Message-broker deployments; no change from today | -| 2. PTY only | `openab-pty` | Standalone remote terminal service: workspace PVC + `[pty]` config + PTY auth secret; no Discord/Slack tokens, no platform adapters, no ACP protocol | +| 2. PTY only | `openab-pty` | Standalone remote terminal service: workspace PVC + `[pty]` config + admin bootstrap credential; no Discord/Slack tokens, no platform adapters, no ACP protocol | | 3. ACP + PTY (colocated) | `openab` + `openab-pty` sidecar | Both in one pod sharing the workspace volume: drive a CLI by hand, let ACP agents continue in the same working tree from Discord | Deployment mechanics: @@ -62,7 +62,7 @@ Profile 3 (colocated) — K8s Pod | +------------+--------------+ | [pty] section only] | | | | +-------------+-------------+ | | | (colocated profile only, Phase 4)| | -| +<--- notification webhook ----------+ | +| +---- notification bridge (pull) --->+ | | | | Shared: workspace volume (PVC), | NOT shared: credentials, | | pod network namespace, pod fate | PID/cgroup, filesystem mounts| @@ -112,17 +112,30 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only - **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (set `automountServiceAccountToken: false` at **pod** level -- it is not per-container -- and, when the broker needs IRSA/configUrl credentials in the colocated profile, project an audience-scoped token volume **into the broker container only**), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) - **Credential-material delivery (MUST)**: MVP has no signing key to deliver (see Token format). The only secret the PTY runtime holds is the **hash of the admin bootstrap credential**, and its delivery follows this rule: **external tooling (operator/Helm/CI) resolves any logical `aws-sm://` reference at deploy time and materializes the literal value into the delivered projection -- `openab-pty` itself never resolves cloud references at runtime and holds no cloud identity.** Delivered material is owned by the runtime UID, mode 0400, never enters the child's environment, logs, or core dumps (dumpable disabled) +- **Filesystem layout (MUST)** -- the layout is what enforces "never in the child's filesystem view", so it is specified, not implied: + - `/run/openab-pty/` -- runtime-only directory holding the control socket and any runtime state; created by the runtime at startup, **never** exported to the child (not in its environment, not under its HOME or cwd) + - `/etc/openab-pty/` (read-only mount) -- the delivered config projection including the admin credential hash + - the workspace volume -- the child's HOME and cwd, and the **only writable mount** in the container (`readOnlyRootFilesystem: true`) + - the child never inherits a writable runtime directory; nothing under `/run/openab-pty/` or `/etc/openab-pty/` is reachable from the child's HOME/cwd tree - **Same-container containment (MVP model)**: with the keyless token design, runtime state contains no minting authority to steal -- only non-reversible hashes. The security basis is: the admin bootstrap credential is CSPRNG high-entropy and the runtime stores **only its verifier hash** (reading the hash neither authorizes requests nor enables practical offline guessing); every control operation is authenticated by presenting the credential itself, even on a locally reachable socket; and the runtime sets `PR_SET_DUMPABLE=0`. **Optional hardening (not MVP)**: spawning PTY children under a distinct unprivileged UID gives defense-in-depth, but a non-root UID-1000 process cannot `setuid` without `CAP_SETUID` or a privileged launcher -- deployments that can provide a narrowly-scoped launcher may adopt it; the default container contract (non-root, all capabilities dropped, `allowPrivilegeEscalation: false`) is preserved either way -- `${VAR}` interpolation and `[secrets.refs]` resolution behave identically in both binaries; the PTY-only projection's `[secrets.refs]` contains exactly one entry -- the admin credential hash +- **Same-UID residual-risk checklist (MUST)** -- because file modes are not a boundary between same-UID processes, the following are requirements, not recommendations: + - Config projection and any secret material are delivered on **read-only mounts** (tamper-proofing comes from the mount, not the file mode) + - Admin-credential verification is **constant-time** against the stored hash + - Attach-token plaintext is **zeroized promptly after hashing**; only the hash is retained + - The admin bootstrap credential has a stated **minimum entropy of 128 bits** (generated, never operator-chosen) + - `PR_SET_DUMPABLE=0` is a **Linux-specific** mitigation; non-Linux targets are out of scope for MVP and must not be assumed covered + - **Accepted risk, stated**: a same-UID child can signal (including SIGKILL) the runtime process; this is availability, not confidentiality -- sessions die with the runtime and no credential is exposed by the crash +- **Resolution asymmetry (deliberate)**: the broker resolves `${VAR}` interpolation and `[secrets.refs]` cloud references itself, as today. The PTY runtime accepts only literal values, `${VAR}` environment interpolation, and local file paths in its delivered projection -- it MUST NOT link or invoke a cloud secrets resolver at runtime. A delivered PTY projection that still contains a `[secrets.refs]` table or any unresolved cloud reference (`aws-sm://` etc.) is a **startup error** (fail closed): this guard prevents an implementer from re-importing a cloud fetch identity into the PTY trust boundary ```toml -# one logical config.toml -- projected into two filtered views; -# neither container ever mounts the other's sections - -[secrets.refs] # LOGICAL operator source only. Deploy tooling - # materializes the literal value into the delivered - # PTY projection -- openab-pty never resolves - # aws-sm:// itself and holds no cloud identity +# ---- LOGICAL operator source (what the operator maintains) ---- +# Deploy tooling projects this into two delivered views; neither +# container ever receives the other's sections. + +[secrets.refs] # broker view only. For the PTY projection, deploy + # tooling resolves this at deploy time and writes the + # literal value -- openab-pty never resolves aws-sm:// + # itself and holds no cloud identity pty_admin_hash = "aws-sm://openab/pty-admin#hash" [discord] # broker view only -- never delivered to the PTY runtime @@ -140,7 +153,24 @@ max_sessions = 4 absolute_session_ttl = "12h" # applies even while attached scrollback_kib = 1024 # in-memory only; cleared on teardown scrollback_replay = false # governs fresh-attach full-history dump only (see lifecycle) -admin_credential_hash = "${secrets.pty_admin_hash}" # verifies the admin bootstrap credential; attach tokens are in-memory only (no key) +admin_credential_hash = "${secrets.pty_admin_hash}" # logical reference in the source only +``` + +```toml +# ---- DELIVERED PTY projection (what openab-pty actually receives) ---- +# Generated outside the PTY trust boundary; contains no [secrets.refs], +# no cloud references, no broker sections. Anything else = startup error. + +[pty] +enabled = true +listen = "0.0.0.0:8090" +tls_terminated_upstream = false +command = "/bin/bash" +max_sessions = 4 +absolute_session_ttl = "12h" +scrollback_kib = 1024 +scrollback_replay = false +admin_credential_hash = "argon2id$..." # literal verifier hash, materialized at deploy time ``` ### Security model @@ -149,7 +179,8 @@ admin_credential_hash = "${secrets.pty_admin_hash}" # verifies the admin bootst - **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); origin policy and constant-time comparison carry over - **Token control plane** (MVP model; an identity layer remains explicitly out of scope per `identity-trust-none.md`): - **Create requires authentication -- and locality is not authentication**: the shell child may share the container (and potentially the UID) with the runtime, so a loopback/UDS endpoint alone is not an auth barrier. MVP mechanism: session creation (`openab-pty session create `) requires an **admin bootstrap credential** delivered to the operator at deploy time and never present in the PTY child's environment or filesystem view. Hardening alternative (non-MVP): a distinct privileged UID/group for the runtime with UDS peer-credential verification. Unauthenticated remote create/list/kill is NOT provided in MVP; the admin credential and the attach token are distinct planes -- an attach token can never create, list, or kill - - **One-time issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, returned exactly once. **Issued once, not single-use**: the bearer token remains valid for reattach until its expiry or a generation bump -- reconnecting clients are not locked out; theft exposure is bounded by a short default TTL (well below the session TTL) and renewal happens only through the authenticated create/renew path + - **One-time issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, returned exactly once. **Issued once, not single-use**: the bearer token remains valid for reattach until its expiry or a generation bump -- reconnecting clients are not locked out; theft exposure is bounded by a short default TTL (well below the session TTL) + - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. Renew is the re-issue path for expired or suspected-stolen tokens, and is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface - **Per-session revocation**: kill/recreate bumps the generation and deletes the stored token hash, immediately invalidating outstanding tokens for that session; runtime restart clears all token state (sessions die with the process anyway, so this is not a loss) - **Token format (MVP): no signing key exists.** Each attach token is a CSPRNG 256-bit opaque bearer value; the runtime stores only its hash together with `(session ID, generation, scope = attach-only, expiry)` in memory and deletes it on kill/expiry. Because sessions deliberately do not survive a runtime restart, self-contained signed tokens buy nothing in MVP -- and eliminating the signing key eliminates the minting authority a same-container shell could steal. Signed (HMAC) tokens are a later option and require either an external signer outside the PTY container or the runtime/child privilege boundary below @@ -168,6 +199,7 @@ admin_credential_hash = "${secrets.pty_admin_hash}" # verifies the admin bootst - **Reconnect**: monotonic byte cursor from day one -- the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes. The replay-to-live handoff is **atomic**: the subscriber registers under the buffer lock, captures the end offset, replays through it, then drains queued live bytes -- with connection-generation fencing so teardown of a replaced connection cannot affect its successor. On overflow the server sends an explicit `gap` control frame (bytes-dropped count) so the client can trigger a full clear/redraw instead of rendering a sliced ANSI stream - **`scrollback_replay` vs cursor semantics** (distinct controls): incremental `since` replay is always available within the ring buffer's retention; `scrollback_replay` governs only the cursor-less full-history dump on a fresh attach (default off -- secrets-safe); setting `scrollback_kib = 0` disables retention entirely, which also disables `since` replay (every reconnect starts with a `gap` + reset) - **Teardown**: setpgid on spawn; SIGTERM-grace-SIGKILL escalation on the process group; evict-while-attached order = notify client, close socket, kill group, close master fd, release slot; buffers cleared on teardown; scrollback never touches disk +- **Kill domain (MUST)**: the process group is only the first signal path, not the containment guarantee -- a child that calls `setsid` or double-forks escapes the pgid. Phase 1 MUST implement at least one hard kill boundary: a per-session cgroup killed via `cgroup.kill` (or freeze-then-kill), or a pidfd-based descendant reaper that tracks and kills all session descendants. Slot release and the absolute TTL are enforced against this boundary, never against the pgid alone - **Recovery taxonomy** (stated, not implied): detach/reattach survives (process alive); pod restart does not (process dead) -- reattach-to-dead returns a distinct error and offers **restart-in-place**: same session name, a fresh process and a new generation (old tokens invalid, empty scrollback). Pod-lifetime durability is out of scope and documented as such --- @@ -180,7 +212,7 @@ admin_credential_hash = "${secrets.pty_admin_hash}" # verifies the admin bootst - Fills the remote + sandboxed + raw-terminal quadrant with a real container boundary instead of a claimed one - Highest reversibility: default-off, separately versioned, separately deprecable - Coexistence where it matters (shared workspace) without shared process or credential-mount domains; network namespace and pod fate are shared only in the colocated profile (see Isolation tiers) -- The Phase 4 notification bridge (sidecar webhook -> broker -> Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes +- The Phase 4 notification bridge (broker pulls from the sidecar -> relays to Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes ### Negative @@ -224,7 +256,7 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Phase 1: `openab-pty` MVP (new crate, new binary) - Own session manager: named sessions, operator-configured command, allowlist-validated names -- **Session bootstrap**: sessions are created via the authenticated loopback/UDS operator CLI (`openab-pty session create `), which spawns the PTY and returns the one-time attach token; `GET /pty/{session}` is attach-only. No remote create/list/kill in Phase 1 (Phase 2 adds them behind admin auth) +- **Session bootstrap**: sessions are created via the authenticated loopback/UDS operator CLI (`openab-pty session create `; `session renew ` re-issues a token per the token control plane), which spawns the PTY and returns the one-time attach token; `GET /pty/{session}` is attach-only. No remote create/list/kill in Phase 1 (Phase 2 adds them behind admin auth) - portable-pty spawner with setpgid, escalating kill, and the teardown order above - `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`, `gap`, `ttl-warning`) with a defined close-code table. Frame validation is strict allowlist: bounded max frame size, unknown control types rejected, resize values bounds-checked; malformed frames count toward an abuse metric and can disconnect - Input backpressure: per-connection write watermark toward the PTY master; a client exceeding it is disconnected (fail closed) rather than growing unbounded queues or stalling the reader @@ -248,8 +280,8 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Phase 4: Messaging bridge (optional, colocated profile only) -- `openab-pty` posts a webhook to the broker when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); broker relays to the platform thread. Bridge is one-way and feature-gated -- **Bridge authentication is required by design, stated now**: the webhook carries an HMAC signature from a dedicated bridge secret delivered at deploy time (or travels over a loopback/UDS-only endpoint) -- separate from platform tokens and from the PTY signing key, and never usable as a PTY identity plane. A compromised PTY runtime must not be able to inject arbitrary notifications +- `openab-pty` exposes a pod-local, loopback-only notification stream; the **broker pulls** (long-poll/SSE on localhost) when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); the broker relays to the platform thread. Bridge is one-way and feature-gated +- **No bridge secret enters the PTY container -- by design, stated now**: a delivered HMAC key would recreate exactly the in-container authority the keyless token model eliminated (a same-UID child can read any file the runtime can read; 0400 at the same UID is not a boundary). The pull model removes the broker-side ingress entirely: there is no webhook endpoint to leave open and no shared key to steal. Residual risk, stated: a same-UID child that kills the runtime (an accepted same-UID risk) could bind the freed port and forge events -- therefore the broker treats bridge events as **display-only, rate-limited hints**: they never carry commands, never mutate broker state, and are labeled best-effort in the relayed message. A push/webhook variant with an HMAC secret is permitted only with an external signer outside the PTY container or the runtime/child privilege boundary from the Security model (non-MVP hardening) - **Not available in the PTY-only profile** — there is no broker to relay through, and `openab-pty` will not grow its own notifier (that would recreate the scope creep this ADR exists to avoid). Users who want notifications deploy profile 3 ### Later (demand-gated, explicitly deferred) From 27ee5b3c475408a21b6bfa27ce3c07d4f7f19a2b Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 21:46:56 +0000 Subject: [PATCH 09/15] docs(adr): address round-4 findings F1-F17 - F1 (critical): close all admin-credential delivery channels -- plaintext only on the operator side; in-container presentation via non-echoing stdin or UDS body only; never argv (/proc/cmdline), env, temp files, or logs; entropy floor raised to 256 bits matching attach tokens - F2: admin control plane throttled (failure backoff, bounded bodies, verification concurrency cap, audited failures) -- not just WS upgrades - F3: Origin policy decided explicitly: bearer-only trust boundary; the /acp Origin gate is keyless-loopback-only and does not carry over - F4: web client token storage contract: memory-only, never localStorage/URLs/cookies; CSP; Phase 2 acceptance criterion - F5: /run/openab-pty backed by dedicated tmpfs; 'only writable mount' corrected; path separation reworded as convention, not kernel boundary - F6: kill domain deployable: pidfd reaper + PR_SET_CHILD_SUBREAPER as MVP default; cgroup.kill gated on subtree delegation; fail-closed startup probe; added to the Phase 1 checklist - F7: startup guard explicitly rejects ${secrets.*} interpolation - F8: takeover preempts audited + rate-limited; revoke/attach/replay total order under one session lock - F9: threat model reworded (runtime is the mint; at-rest vs transient); PR_SET_DUMPABLE=0 load-bearing MUST with stated /proc+ptrace dependency; admin credential buffer zeroized after verification - F10: renew-while-attached evicts the active connection (defined) - F11: session restart added to the Phase 1 CLI - F12: output path bounded (handoff queue, slow-client backlog; gap or fail-closed disconnect) - F13: Phase 4 pull-stream resource contract (single stream, heartbeat, capped backoff, bounded coalescing queue, drop-oldest) - F14: admin-credential rotation = redeploy new hash + restart, stated - F15: sha256 verifier example with 256-bit-CSPRNG rationale (argon2 targets human-chosen secrets, forbidden here) - F16: delivered-projection example satisfies the fail-closed TLS guard; bridge diagram arrow reads as broker-initiated pull - F17: rollback contract detailed; Helm owns projection generation with a CI guard test (poisoned projection must be rejected) --- docs/adr/openab-pty-runtime.md | 55 +++++++++++++++++++++------------- 1 file changed, 34 insertions(+), 21 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index b9a52acfe..3d7a90560 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -62,7 +62,7 @@ Profile 3 (colocated) — K8s Pod | +------------+--------------+ | [pty] section only] | | | | +-------------+-------------+ | | | (colocated profile only, Phase 4)| | -| +---- notification bridge (pull) --->+ | +| +<--- events (broker-initiated pull) + | | | | Shared: workspace volume (PVC), | NOT shared: credentials, | | pod network namespace, pod fate | PID/cgroup, filesystem mounts| @@ -112,20 +112,21 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only - **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (set `automountServiceAccountToken: false` at **pod** level -- it is not per-container -- and, when the broker needs IRSA/configUrl credentials in the colocated profile, project an audience-scoped token volume **into the broker container only**), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) - **Credential-material delivery (MUST)**: MVP has no signing key to deliver (see Token format). The only secret the PTY runtime holds is the **hash of the admin bootstrap credential**, and its delivery follows this rule: **external tooling (operator/Helm/CI) resolves any logical `aws-sm://` reference at deploy time and materializes the literal value into the delivered projection -- `openab-pty` itself never resolves cloud references at runtime and holds no cloud identity.** Delivered material is owned by the runtime UID, mode 0400, never enters the child's environment, logs, or core dumps (dumpable disabled) -- **Filesystem layout (MUST)** -- the layout is what enforces "never in the child's filesystem view", so it is specified, not implied: - - `/run/openab-pty/` -- runtime-only directory holding the control socket and any runtime state; created by the runtime at startup, **never** exported to the child (not in its environment, not under its HOME or cwd) +- **Filesystem layout (MUST)** -- the layout is what enforces "never in the child's filesystem view" as an organizational contract, so it is specified, not implied: + - `/run/openab-pty/` -- runtime-only directory holding the control socket and any runtime state; backed by a **dedicated writable tmpfs/emptyDir mount** (required: `readOnlyRootFilesystem: true` makes the root filesystem unwritable); created by the runtime at startup, **never** exported to the child (not in its environment, not under its HOME or cwd) - `/etc/openab-pty/` (read-only mount) -- the delivered config projection including the admin credential hash - - the workspace volume -- the child's HOME and cwd, and the **only writable mount** in the container (`readOnlyRootFilesystem: true`) - - the child never inherits a writable runtime directory; nothing under `/run/openab-pty/` or `/etc/openab-pty/` is reachable from the child's HOME/cwd tree -- **Same-container containment (MVP model)**: with the keyless token design, runtime state contains no minting authority to steal -- only non-reversible hashes. The security basis is: the admin bootstrap credential is CSPRNG high-entropy and the runtime stores **only its verifier hash** (reading the hash neither authorizes requests nor enables practical offline guessing); every control operation is authenticated by presenting the credential itself, even on a locally reachable socket; and the runtime sets `PR_SET_DUMPABLE=0`. **Optional hardening (not MVP)**: spawning PTY children under a distinct unprivileged UID gives defense-in-depth, but a non-root UID-1000 process cannot `setuid` without `CAP_SETUID` or a privileged launcher -- deployments that can provide a narrowly-scoped launcher may adopt it; the default container contract (non-root, all capabilities dropped, `allowPrivilegeEscalation: false`) is preserved either way + - the workspace volume -- the child's HOME and cwd, and the only writable **persistent** mount (the `/run/openab-pty` tmpfs is the sole other writable mount, and it is runtime-scoped and non-persistent) + - **Scope of this boundary, stated honestly**: path separation is accident/organization prevention, not a kernel boundary -- a same-UID child in the same mount namespace can open these paths by absolute path. Confidentiality does not rest on filesystem invisibility; it rests on the hash-only design (nothing under these paths mints authority; see Same-container containment) +- **Same-container containment (MVP model)**: with the keyless token design, runtime **persistent state** contains no minting authority -- only non-reversible hashes. Stated precisely: the runtime *is* the mint, and freshly minted tokens and the presented admin credential transiently exist in its process memory; what the design removes is any *at-rest* stealable authority (no signing key, no stored plaintext). Protecting the transient window is therefore load-bearing: the runtime MUST set `PR_SET_DUMPABLE=0` **before any secret enters memory** -- this is what blocks same-UID `/proc//mem`, `/proc//fd`, and `ptrace` access on Linux; without it the containment claim does not hold. The remaining basis: the admin bootstrap credential is CSPRNG high-entropy and the runtime stores **only its verifier hash** (reading the hash neither authorizes requests nor enables practical offline guessing); every control operation is authenticated by presenting the credential itself, even on a locally reachable socket. **Optional hardening (not MVP)**: spawning PTY children under a distinct unprivileged UID gives defense-in-depth, but a non-root UID-1000 process cannot `setuid` without `CAP_SETUID` or a privileged launcher -- deployments that can provide a narrowly-scoped launcher may adopt it; the default container contract (non-root, all capabilities dropped, `allowPrivilegeEscalation: false`) is preserved either way - **Same-UID residual-risk checklist (MUST)** -- because file modes are not a boundary between same-UID processes, the following are requirements, not recommendations: - Config projection and any secret material are delivered on **read-only mounts** (tamper-proofing comes from the mount, not the file mode) - - Admin-credential verification is **constant-time** against the stored hash + - Admin-credential verification is **constant-time** against the stored hash, and the presented credential buffer is **zeroized immediately after verification** (the same discipline as attach tokens) - Attach-token plaintext is **zeroized promptly after hashing**; only the hash is retained - - The admin bootstrap credential has a stated **minimum entropy of 128 bits** (generated, never operator-chosen) + - The admin bootstrap credential is **generated, never operator-chosen**, with a minimum entropy of **256 bits** -- matching the attach-token strength, so the admin plane is never the weaker of the two credential planes + - **Rotation, stated**: `session renew` rotates attach tokens, never the admin credential. Rotating the admin credential = generating a new value, updating the delivered hash, and restarting the runtime -- which clears all sessions. This is acceptable (sessions are non-persistent by contract) and is the documented procedure - `PR_SET_DUMPABLE=0` is a **Linux-specific** mitigation; non-Linux targets are out of scope for MVP and must not be assumed covered - **Accepted risk, stated**: a same-UID child can signal (including SIGKILL) the runtime process; this is availability, not confidentiality -- sessions die with the runtime and no credential is exposed by the crash -- **Resolution asymmetry (deliberate)**: the broker resolves `${VAR}` interpolation and `[secrets.refs]` cloud references itself, as today. The PTY runtime accepts only literal values, `${VAR}` environment interpolation, and local file paths in its delivered projection -- it MUST NOT link or invoke a cloud secrets resolver at runtime. A delivered PTY projection that still contains a `[secrets.refs]` table or any unresolved cloud reference (`aws-sm://` etc.) is a **startup error** (fail closed): this guard prevents an implementer from re-importing a cloud fetch identity into the PTY trust boundary +- **Resolution asymmetry (deliberate)**: the broker resolves `${VAR}` interpolation and `[secrets.refs]` cloud references itself, as today. The PTY runtime accepts only literal values, `${VAR}` environment interpolation, and local file paths in its delivered projection -- it MUST NOT link or invoke a cloud secrets resolver at runtime. A delivered PTY projection that still contains a `[secrets.refs]` table, any unresolved cloud reference (`aws-sm://` etc.), **or any `${secrets.*}` interpolation** is a **startup error** (fail closed) -- `${secrets.*}` is enumerated explicitly because it shares the `${}` delimiters with the accepted `${VAR}` env form and must never be silently treated as an unset environment variable. This guard prevents an implementer from re-importing a cloud fetch identity into the PTY trust boundary ```toml # ---- LOGICAL operator source (what the operator maintains) ---- @@ -159,35 +160,44 @@ admin_credential_hash = "${secrets.pty_admin_hash}" # logical reference in the ```toml # ---- DELIVERED PTY projection (what openab-pty actually receives) ---- # Generated outside the PTY trust boundary; contains no [secrets.refs], -# no cloud references, no broker sections. Anything else = startup error. +# no cloud references, no ${secrets.*} interpolation, no broker sections. +# Anything else = startup error. [pty] enabled = true listen = "0.0.0.0:8090" -tls_terminated_upstream = false +tls_cert = "/etc/openab-pty/tls.crt" # termination (a): in-process TLS. Off-loopback +tls_key = "/etc/openab-pty/tls.key" # bind without TLS material fails closed unless +tls_terminated_upstream = false # tls_terminated_upstream = true (termination (b)) command = "/bin/bash" max_sessions = 4 absolute_session_ttl = "12h" scrollback_kib = 1024 scrollback_replay = false -admin_credential_hash = "argon2id$..." # literal verifier hash, materialized at deploy time +admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized at deploy time. + # SHA-256 suffices: the credential is generated + # 256-bit CSPRNG, so memory-hard hashing (argon2) + # adds nothing -- that defense targets low-entropy + # human-chosen secrets, which are forbidden here ``` ### Security model - **Transport / TLS contract**: WSS is mandatory for external clients. Two supported terminations, chosen explicitly per deployment: (a) `openab-pty` terminates TLS itself (cert mounted into the container), or (b) a trusted Ingress terminates TLS and forwards plain WS internally -- in which case the internal listener accepts non-loopback plain WS only when the deployment declares `tls_terminated_upstream = true`, and the residual internal-hop exposure is documented. Fail-closed in all cases: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) -- **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); origin policy and constant-time comparison carry over +- **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); constant-time comparison carries over. **Origin policy, decided explicitly (not "carried over")**: the as-built `/acp` consults `Origin` only on its keyless loopback path; its keyed bearer path never checks Origin -- and PTY has no keyless mode, so there is nothing to carry over. PTY's trust boundary is **bearer-only**: possession of a valid attach token is the sole authorization, and the `Origin` header is not consulted (it is attacker-controlled outside browsers and adds no strength to a keyed WebSocket). Browser-side token hygiene is governed by the client storage contract below - **Token control plane** (MVP model; an identity layer remains explicitly out of scope per `identity-trust-none.md`): - **Create requires authentication -- and locality is not authentication**: the shell child may share the container (and potentially the UID) with the runtime, so a loopback/UDS endpoint alone is not an auth barrier. MVP mechanism: session creation (`openab-pty session create `) requires an **admin bootstrap credential** delivered to the operator at deploy time and never present in the PTY child's environment or filesystem view. Hardening alternative (non-MVP): a distinct privileged UID/group for the runtime with UDS peer-credential verification. Unauthenticated remote create/list/kill is NOT provided in MVP; the admin credential and the attach token are distinct planes -- an attach token can never create, list, or kill + - **Admin-credential delivery channels (MUST)** -- the containment model rests on the plaintext credential never being observable by a same-UID shell child, so every channel is enumerated and closed, not just env and files: the plaintext exists **only on the operator's trusted side** (their terminal/secret manager); in-container presentation accepts it **only via short-lived non-echoing stdin or a UDS message body**; it is **never** accepted as an argv flag (`/proc//cmdline` is world-readable), never placed in any process environment inside the container, never written to temporary files, and never emitted to audit logs or errors (log a fingerprint of the verifier hash, not the credential). A same-UID shell observing one create/renew invocation must learn nothing reusable - **One-time issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, returned exactly once. **Issued once, not single-use**: the bearer token remains valid for reattach until its expiry or a generation bump -- reconnecting clients are not locked out; theft exposure is bounded by a short default TTL (well below the session TTL) - - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. Renew is the re-issue path for expired or suspected-stolen tokens, and is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface + - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. **Renew-while-attached, defined**: an actively attached connection is evicted via the standard evict order (notify, close with a distinct close code) before the fresh token is returned -- renew's use cases include suspected theft, so the possibly-hostile attached client must not linger under a revoked generation. Renew is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface - **Per-session revocation**: kill/recreate bumps the generation and deletes the stored token hash, immediately invalidating outstanding tokens for that session; runtime restart clears all token state (sessions die with the process anyway, so this is not a loss) - **Token format (MVP): no signing key exists.** Each attach token is a CSPRNG 256-bit opaque bearer value; the runtime stores only its hash together with `(session ID, generation, scope = attach-only, expiry)` in memory and deletes it on kill/expiry. Because sessions deliberately do not survive a runtime restart, self-contained signed tokens buy nothing in MVP -- and eliminating the signing key eliminates the minting authority a same-container shell could steal. Signed (HMAC) tokens are a later option and require either an external signer outside the PTY container or the runtime/child privilege boundary below - **Command authority**: the spawned command is operator configuration only; clients can never specify it. Session names are allowlist-validated (`[a-z0-9-]{1,32}`) - **Isolation**: the PTY container mounts only the workspace volume (workspace-scoped, never the broker HOME PVC) and its own config projection; no service-account token, no broker config, no platform secrets (see the deployment contract above). NetworkPolicy applies at pod scope: it can restrict the standalone profile's pod independently; in the colocated profile it cannot separate the two containers (see Isolation tiers) -- **Container defaults**: the `openab-pty` image runs as a non-root user (UID 1000), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable mount (child UID separation is optional hardening per Same-container containment above) -- **Rate limiting in MVP**: per-IP WS-upgrade failure limits (e.g. 5 failures/min then a short ban) ship in Phase 1 -- audit is detection, rate limiting is prevention +- **Container defaults**: the `openab-pty` image runs as a non-root user (UID 1000), `allowPrivilegeEscalation: false`, capabilities dropped, `readOnlyRootFilesystem` with the workspace as the only writable persistent mount plus the `/run/openab-pty` tmpfs (see Filesystem layout); child UID separation is optional hardening per Same-container containment above +- **Rate limiting in MVP**: per-IP WS-upgrade failure limits (e.g. 5 failures/min then a short ban) ship in Phase 1 -- audit is detection, rate limiting is prevention. **The admin control plane is throttled too, not just WS upgrades**: every admin-credential verifier (loopback/UDS `session create/renew/kill/restart` in Phase 1, remote admin endpoints in Phase 2) enforces a failure throttle with backoff, bounded request/body sizes, and a small concurrency cap on in-flight verifications (bounded work per attempt); admin auth failures are audited like attach failures +- **Client-side token storage (contract for the Phase 2 web client)**: the attach token is shell-equivalent, so the shipped client holds it **in memory only** -- never localStorage/sessionStorage, never in URLs (query strings leak via history, referrer, and proxy logs), never in cookies. Page reload = token gone = re-issue via renew, accepted UX. The served page sets a restrictive CSP; these rules are a Phase 2 acceptance criterion, stated now so the client is not designed around persistent storage - **Audit in MVP**: attach/detach, session create/kill, and auth failures are logged from Phase 1; a leaked token must be observable - **Env**: the PTY child gets an explicit allowlist (TERM, LANG/LC_*, PATH, HOME, USER, SHELL) and nothing else; `OPENAB_*` and cloud-credential variables are never inherited @@ -195,11 +205,12 @@ admin_credential_hash = "argon2id$..." # literal verifier hash, materialized at - **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong at a 15-30s interval; a half-open socket counts as detached after 2-3 missed pings -- exact values are Phase 1 config with these recommended defaults, balancing flaky mobile networks against dead-client slot pinning) - **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached -- capacity cannot be pinned forever by an open browser tab. Expiry is client-visible: a warning control frame precedes forced teardown, and the WebSocket closes with a distinct close code so clients surface "session expired" instead of retrying a network error -- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3) +- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3). **Takeover abuse controls**: every successful preempt is audited as an anomaly event (session, source address, count), and preempt frequency is rate-limited per session (e.g. max N takeovers/min, then attaches are rejected with a distinct close code) -- a stolen still-valid token must not be able to silently ping-pong the CAS and starve the legitimate client; the audit trail plus `session renew` is the recovery path. **Ordering contract**: token revocation (generation bump + stored-hash deletion), the attach CAS, and replay registration execute under one session lock in that total order -- no interleaving where a revoked token wins an attach or a replay registers against a stale generation - **Reconnect**: monotonic byte cursor from day one -- the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes. The replay-to-live handoff is **atomic**: the subscriber registers under the buffer lock, captures the end offset, replays through it, then drains queued live bytes -- with connection-generation fencing so teardown of a replaced connection cannot affect its successor. On overflow the server sends an explicit `gap` control frame (bytes-dropped count) so the client can trigger a full clear/redraw instead of rendering a sliced ANSI stream +- **Output-path bounds (MUST)**: every buffer on the PTY-to-client path is bounded, not just retained scrollback -- the replay-to-live handoff queue and the per-connection outbound backlog each have a fixed cap. A client too slow to drain its backlog gets a `gap` frame (drop-oldest, cursor advances) or, past a hard watermark, is disconnected with a distinct close code -- mirroring the input-side fail-closed backpressure so neither direction can grow unbounded memory - **`scrollback_replay` vs cursor semantics** (distinct controls): incremental `since` replay is always available within the ring buffer's retention; `scrollback_replay` governs only the cursor-less full-history dump on a fresh attach (default off -- secrets-safe); setting `scrollback_kib = 0` disables retention entirely, which also disables `since` replay (every reconnect starts with a `gap` + reset) - **Teardown**: setpgid on spawn; SIGTERM-grace-SIGKILL escalation on the process group; evict-while-attached order = notify client, close socket, kill group, close master fd, release slot; buffers cleared on teardown; scrollback never touches disk -- **Kill domain (MUST)**: the process group is only the first signal path, not the containment guarantee -- a child that calls `setsid` or double-forks escapes the pgid. Phase 1 MUST implement at least one hard kill boundary: a per-session cgroup killed via `cgroup.kill` (or freeze-then-kill), or a pidfd-based descendant reaper that tracks and kills all session descendants. Slot release and the absolute TTL are enforced against this boundary, never against the pgid alone +- **Kill domain (MUST)**: the process group is only the first signal path, not the containment guarantee -- a child that calls `setsid` or double-forks escapes the pgid. **MVP default (works under the stated container contract, no extra capabilities): a pidfd-based descendant reaper** -- the runtime sets `PR_SET_CHILD_SUBREAPER` so escaped descendants reparent to it, holds a pidfd per tracked process, and performs race-safe descendant discovery before killing (re-scan until stable, then kill via pidfds). **The cgroup path (`cgroup.kill`, or freeze-then-kill) is the stronger boundary but is gated on an explicit prerequisite**: cgroup v2 with subtree delegation to the container's UID, which the default non-root/no-capabilities contract does not provide -- deployments that arrange delegation SHOULD prefer it. **Startup probe, fail closed**: at startup the runtime verifies its configured kill mechanism is operational (subreaper flag set and pidfd support, or a writable delegated cgroup subtree) and refuses to serve sessions otherwise. Slot release and the absolute TTL are enforced against this hard boundary, never against the pgid alone - **Recovery taxonomy** (stated, not implied): detach/reattach survives (process alive); pod restart does not (process dead) -- reattach-to-dead returns a distinct error and offers **restart-in-place**: same session name, a fresh process and a new generation (old tokens invalid, empty scrollback). Pod-lifetime durability is out of scope and documented as such --- @@ -256,8 +267,8 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Phase 1: `openab-pty` MVP (new crate, new binary) - Own session manager: named sessions, operator-configured command, allowlist-validated names -- **Session bootstrap**: sessions are created via the authenticated loopback/UDS operator CLI (`openab-pty session create `; `session renew ` re-issues a token per the token control plane), which spawns the PTY and returns the one-time attach token; `GET /pty/{session}` is attach-only. No remote create/list/kill in Phase 1 (Phase 2 adds them behind admin auth) -- portable-pty spawner with setpgid, escalating kill, and the teardown order above +- **Session bootstrap**: sessions are created via the authenticated loopback/UDS operator CLI (`openab-pty session create `; `session renew ` re-issues a token per the token control plane; `session restart ` performs restart-in-place for reattach-to-dead per the recovery taxonomy), which spawns the PTY and returns the one-time attach token; `GET /pty/{session}` is attach-only. No remote create/list/kill in Phase 1 (Phase 2 adds them behind admin auth) +- portable-pty spawner with setpgid, escalating kill, the teardown order above, **and the hard kill boundary per the Kill domain MUST** (pidfd descendant reaper with `PR_SET_CHILD_SUBREAPER` as the default; delegated-cgroup kill where available; fail-closed startup probe) - `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`, `gap`, `ttl-warning`) with a defined close-code table. Frame validation is strict allowlist: bounded max frame size, unknown control types rejected, resize values bounds-checked; malformed frames count toward an abuse metric and can disconnect - Input backpressure: per-connection write watermark toward the PTY master; a client exceeding it is disconnected (fail closed) rather than growing unbounded queues or stalling the reader - Auth: the token control plane above (authenticated create, one-time issuance bound to session generation, attach-only verification); fail-closed off-loopback; `/acp`-style browser subprotocol transport; per-IP upgrade-failure rate limiting @@ -271,7 +282,8 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Helm: independent `openab.enabled` / `pty.enabled` toggles (or `--set profile=acp|pty|full`); standalone profile gets its own Service/Ingress (`/pty/*`) and NetworkPolicy example; config split documented per the configUrl pattern; `ghcr.io/openabdev/openab-pty` image published from the existing release pipeline - Minimal xterm.js page served by `openab-pty`; remote session list/create/kill endpoints gated by the **admin bootstrap credential** (the same control-plane contract as the Phase 1 CLI -- never attach tokens; attach-token scope remains attach-only) -- Rollback procedure: disabling the toggle drains (notify + grace) then kills sessions; broker unaffected +- Rollback contract (Phase 2 acceptance criterion): disabling the toggle drains (notify + grace, honoring `terminationGracePeriodSeconds`) then kills sessions; the broker container is unaffected and not restarted. Projection updates roll the PTY container only -- live sessions die on the rollout (non-persistent by contract) and the chart documents this; the workspace PVC is untouched by disable/re-enable, and re-enable is a fresh runtime with zero sessions +- Projection tooling has an owner (Phase 2 acceptance criterion): the Helm chart generates both config views, and CI runs a **guard test** -- the delivered PTY projection is fed to `openab-pty`'s startup validator, which must accept it and must reject a deliberately poisoned projection (embedded `[secrets.refs]`, `${secrets.*}`, or a broker section). The "never delivered" invariant is thereby enforced by mechanism, not operator discipline ### Phase 3: Lifecycle hardening @@ -283,6 +295,7 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - `openab-pty` exposes a pod-local, loopback-only notification stream; the **broker pulls** (long-poll/SSE on localhost) when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); the broker relays to the platform thread. Bridge is one-way and feature-gated - **No bridge secret enters the PTY container -- by design, stated now**: a delivered HMAC key would recreate exactly the in-container authority the keyless token model eliminated (a same-UID child can read any file the runtime can read; 0400 at the same UID is not a boundary). The pull model removes the broker-side ingress entirely: there is no webhook endpoint to leave open and no shared key to steal. Residual risk, stated: a same-UID child that kills the runtime (an accepted same-UID risk) could bind the freed port and forge events -- therefore the broker treats bridge events as **display-only, rate-limited hints**: they never carry commands, never mutate broker state, and are labeled best-effort in the relayed message. A push/webhook variant with an HMAC secret is permitted only with an external signer outside the PTY container or the runtime/child privilege boundary from the Security model (non-MVP hardening) - **Not available in the PTY-only profile** — there is no broker to relay through, and `openab-pty` will not grow its own notifier (that would recreate the scope creep this ADR exists to avoid). Users who want notifications deploy profile 3 +- **Pull-stream resource contract**: the notification stream is bounded like every other surface -- at most **one** concurrent broker stream (a new connection preempts the old), heartbeat with an idle timeout on both ends, reconnect with capped exponential backoff on the broker side, and a fixed-size event queue in `openab-pty` with coalescing (per-session dedupe: a newer idle event replaces an older undelivered one) and drop-oldest overflow. "Rate-limited hints" thus constrains retained resources, not only delivery semantics ### Later (demand-gated, explicitly deferred) From b931d9468231bf5420d0842bf5f974cb6cf27335 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 23:05:33 +0000 Subject: [PATCH 10/15] docs(adr): address round-6 group-review findings F1-F13 + O5/O7/O8 - F1 (critical): containment MUSTs now cover every in-container process touching the plaintext credential (dumpable-before-read, bounded buffers, zeroize-before-exit; stdin-splice preferred so only the runtime materializes it); Phase 1 gains same-UID adversary tests and a dumpable-stays-zero regression guard - F2 (critical): trusted-Ingress TLS is the MVP default and only same-UID mode; in-process TLS gated on a privilege/mount-namespace boundary; TLS-key vs admin-hash risk grading stated; config examples updated (no tls.key in container); 'no token-signing key' wording fixed - F3: kill domain restructured -- own-tree reaper scope, kill-and-rescan convergence invariant, pidfd/process caps with FD headroom, fail-closed - F4: takeover limiter generation-scoped (new-generation bypass); session-lock-before-buffer-lock total ordering - F5: named connection-evict vs session-teardown sequences; renew uses connection-evict with a renew-distinct close code and is documented as admin-initiated/disruptive; issuance phrasing de-ambiguated - F6: web client is attach-only; admin credential never reaches the browser; management UI deferred behind pairing/identity; testable CSP directives as Phase 2 acceptance criterion - F7: pull-stream incumbent-wins admission; live-hijack residual stated - F9: self-exit transition defined (session-ended close code, slot release after convergence, audit termination classes) - F10: read projections marked demand-gated deferral - F11: admin-credential rotation runbook as Phase 2 acceptance criterion - F12: sidecar terminology swept (reserved for profile-3 topology); containment wall of text split into sub-bullets; CSPRNG expanded - F13: secrets-management.md relevance scoped; projection mapping and #hash fragment semantics documented - O5: openab-pty --validate-projection subcommand in Phase 1 - O7: no-HA/process-bound availability trade added to Consequences - O8: chart never defaults to full; colocated labeled convenience-only - Pre-implementation go/no-go demand gate added to Section 5 (round-2 carry re-raised by the strategic lanes) PR body updated separately to match the keyless/pull architecture (F8). --- docs/adr/openab-pty-runtime.md | 81 +++++++++++++++++++++++----------- 1 file changed, 55 insertions(+), 26 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index 3d7a90560..3017a6af2 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -3,7 +3,7 @@ - **Status:** Proposed - **Date:** 2026-08-15 - **Author:** @pahud -- **Related:** [ADR: ACP Server with WebSocket Transport (base, as-built)](./acp-server-websocket-base.md), [ADR: Separate Binaries with Opt-In Unified Build](./unified-binary.md), [ADR: Secrets Management](./secrets-management.md), [ADR: Identity Trust None](./identity-trust-none.md) +- **Related:** [ADR: ACP Server with WebSocket Transport (base, as-built)](./acp-server-websocket-base.md), [ADR: Separate Binaries with Opt-In Unified Build](./unified-binary.md), [ADR: Secrets Management](./secrets-management.md) (applies to the deploy-time materialization of the admin credential hash only -- no PTY token-signing key exists in MVP), [ADR: Identity Trust None](./identity-trust-none.md) - **Supersedes:** the in-process "PTY Mode" proposal (PR #1477, closed) — group review verdict and rationale are preserved in that PR's consolidated review - **Implementation:** TBD @@ -77,12 +77,12 @@ Profile 2 makes `openab-pty` a small standalone product in OpenDray's category ( ### Why a separate runtime (and what it fixes) -| Review blocker (PR #1477) | How the sidecar form resolves it | +| Review blocker (PR #1477) | How the composable-runtime form resolves it | |---|---| | Positioning vs Thin Bridge | OAB binary is untouched; the broker stays a pure transport. `openab-pty` is an adjacent tool that shares deployment infrastructure only — no dual persona | | Same-pod blast radius | Separate container = separate PID namespace, cgroup, filesystem, and mounts: the shell user cannot signal the broker, exhaust its container cgroup, or read its credential files. Broker platform tokens are **never mounted** into the PTY container. Residual sharing in the colocated profile (pod network namespace, pod fate) is graded honestly in Isolation tiers below; full isolation = profiles 1+2 as separate pods | -| Auth below capability | The sidecar designs its token model from scratch for shell-equivalent trust (see Security model) with no ACP-key coupling | -| Pool incompatibility | The sidecar has its **own session manager** built for byte-stream lifecycle. No refactor of the shipped ACP pool; zero regression risk to the broker | +| Auth below capability | `openab-pty` designs its token model from scratch for shell-equivalent trust (see Security model) with no ACP-key coupling | +| Pool incompatibility | `openab-pty` has its **own session manager** built for byte-stream lifecycle. No refactor of the shipped ACP pool; zero regression risk to the broker | | Reversibility | Default-off runtime with its own image/release. If demand does not materialize, deprecate the image; nothing in the broker to unwind. If demand proves out, later extraction of a shared lifecycle crate — or even single-process merge — remains open | ### Coexistence with ACP @@ -111,13 +111,18 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - The broker reads its existing sections; it ignores `[pty]` - The PTY runtime receives **only** a pre-filtered `[pty]` projection. Self-filtering a shared config is NOT an accepted secure delivery: `--section pty` limits parsing, not access -- if the PTY container holds the source URL and fetch credentials, a shell user can fetch the full broker config directly. The sanitized projection MUST be generated outside the PTY trust boundary (CI, chart, or operator tooling) and delivered via its own object/URL with a fetch identity scoped to that object only - **Deployment contract -- the PTY container spec MUST NOT mount**: the ServiceAccount token (set `automountServiceAccountToken: false` at **pod** level -- it is not per-container -- and, when the broker needs IRSA/configUrl credentials in the colocated profile, project an audience-scoped token volume **into the broker container only**), the broker config or its source credentials, platform secrets, or any volume broader than the workspace (workspace-only volume or `subPath`; never the broker HOME PVC, which contains caches and session state) -- **Credential-material delivery (MUST)**: MVP has no signing key to deliver (see Token format). The only secret the PTY runtime holds is the **hash of the admin bootstrap credential**, and its delivery follows this rule: **external tooling (operator/Helm/CI) resolves any logical `aws-sm://` reference at deploy time and materializes the literal value into the delivered projection -- `openab-pty` itself never resolves cloud references at runtime and holds no cloud identity.** Delivered material is owned by the runtime UID, mode 0400, never enters the child's environment, logs, or core dumps (dumpable disabled) +- **Credential-material delivery (MUST)**: MVP has no **token-signing key** to deliver (see Token format), and in the same-UID MVP no TLS private key enters the container either (Ingress-terminated TLS is the default -- see the Transport/TLS contract). The only secret the PTY runtime holds is the **hash of the admin bootstrap credential**, and its delivery follows this rule: **external tooling (operator/Helm/CI) resolves any logical `aws-sm://` reference at deploy time and materializes the literal value into the delivered projection -- `openab-pty` itself never resolves cloud references at runtime and holds no cloud identity.** Delivered material is owned by the runtime UID, mode 0400, never enters the child's environment, logs, or core dumps (dumpable disabled) - **Filesystem layout (MUST)** -- the layout is what enforces "never in the child's filesystem view" as an organizational contract, so it is specified, not implied: - `/run/openab-pty/` -- runtime-only directory holding the control socket and any runtime state; backed by a **dedicated writable tmpfs/emptyDir mount** (required: `readOnlyRootFilesystem: true` makes the root filesystem unwritable); created by the runtime at startup, **never** exported to the child (not in its environment, not under its HOME or cwd) - `/etc/openab-pty/` (read-only mount) -- the delivered config projection including the admin credential hash - the workspace volume -- the child's HOME and cwd, and the only writable **persistent** mount (the `/run/openab-pty` tmpfs is the sole other writable mount, and it is runtime-scoped and non-persistent) - - **Scope of this boundary, stated honestly**: path separation is accident/organization prevention, not a kernel boundary -- a same-UID child in the same mount namespace can open these paths by absolute path. Confidentiality does not rest on filesystem invisibility; it rests on the hash-only design (nothing under these paths mints authority; see Same-container containment) -- **Same-container containment (MVP model)**: with the keyless token design, runtime **persistent state** contains no minting authority -- only non-reversible hashes. Stated precisely: the runtime *is* the mint, and freshly minted tokens and the presented admin credential transiently exist in its process memory; what the design removes is any *at-rest* stealable authority (no signing key, no stored plaintext). Protecting the transient window is therefore load-bearing: the runtime MUST set `PR_SET_DUMPABLE=0` **before any secret enters memory** -- this is what blocks same-UID `/proc//mem`, `/proc//fd`, and `ptrace` access on Linux; without it the containment claim does not hold. The remaining basis: the admin bootstrap credential is CSPRNG high-entropy and the runtime stores **only its verifier hash** (reading the hash neither authorizes requests nor enables practical offline guessing); every control operation is authenticated by presenting the credential itself, even on a locally reachable socket. **Optional hardening (not MVP)**: spawning PTY children under a distinct unprivileged UID gives defense-in-depth, but a non-root UID-1000 process cannot `setuid` without `CAP_SETUID` or a privileged launcher -- deployments that can provide a narrowly-scoped launcher may adopt it; the default container contract (non-root, all capabilities dropped, `allowPrivilegeEscalation: false`) is preserved either way + - **Scope of this boundary, stated honestly**: path separation is accident/organization prevention, not a kernel boundary -- a same-UID child in the same mount namespace can open these paths by absolute path. Confidentiality does not rest on filesystem invisibility; it rests on **no readable authority existing under these paths at all**: the config projection holds only the non-reversible verifier hash, and no TLS private key is present in the same-UID MVP (Ingress-terminated TLS -- see the Transport/TLS contract). Any future mounted secret that *is* authority (a TLS key, an HMAC key) requires the privilege/mount-namespace boundary first +- **Same-container containment (MVP model)**: + - **What the keyless design removes**: runtime **persistent state** contains no minting authority -- only non-reversible hashes. There is no signing key and no stored plaintext for a same-UID shell to steal at rest + - **What remains, stated precisely**: the runtime *is* the mint -- freshly minted tokens and the presented admin credential transiently exist in its process memory + - **Protecting the transient window is therefore load-bearing**: the runtime MUST set `PR_SET_DUMPABLE=0` **before any secret enters memory** -- this is what blocks same-UID `/proc//mem`, `/proc//fd`, and `ptrace` access on Linux; without it the containment claim does not hold + - **Credential strength basis**: the admin bootstrap credential is generated by a CSPRNG (cryptographically secure pseudorandom number generator) at high entropy, and the runtime stores **only its verifier hash** -- reading the hash neither authorizes requests nor enables practical offline guessing; every control operation is authenticated by presenting the credential itself, even on a locally reachable socket + - **Optional hardening (not MVP)**: spawning PTY children under a distinct unprivileged UID gives defense-in-depth, but a non-root UID-1000 process cannot `setuid` without `CAP_SETUID` or a privileged launcher -- deployments that can provide a narrowly-scoped launcher may adopt it; the default container contract (non-root, all capabilities dropped, `allowPrivilegeEscalation: false`) is preserved either way - **Same-UID residual-risk checklist (MUST)** -- because file modes are not a boundary between same-UID processes, the following are requirements, not recommendations: - Config projection and any secret material are delivered on **read-only mounts** (tamper-proofing comes from the mount, not the file mode) - Admin-credential verification is **constant-time** against the stored hash, and the presented credential buffer is **zeroized immediately after verification** (the same discipline as attach tokens) @@ -148,7 +153,9 @@ bot_token = "${DISCORD_BOT_TOKEN}" [pty] # PTY view only enabled = true listen = "0.0.0.0:8090" # own port; TLS contract below -tls_terminated_upstream = false # true only behind a trusted TLS-terminating Ingress +tls_terminated_upstream = true # MVP default: trusted Ingress terminates TLS. + # In-process TLS (a mounted key) is gated on a real + # runtime/child privilege boundary -- see Security model command = "/bin/bash" # operator-configured; never client-specified max_sessions = 4 absolute_session_ttl = "12h" # applies even while attached @@ -157,6 +164,8 @@ scrollback_replay = false # governs fresh-attach full-history dump onl admin_credential_hash = "${secrets.pty_admin_hash}" # logical reference in the source only ``` +Deploy tooling resolves `pty_admin_hash` from `[secrets.refs]` (the `#hash` fragment is the JSON key inside the secret, per the secrets-management ADR's `#` contract -- the stored value *is* the verifier hash) and writes the literal into the delivered projection: + ```toml # ---- DELIVERED PTY projection (what openab-pty actually receives) ---- # Generated outside the PTY trust boundary; contains no [secrets.refs], @@ -166,9 +175,8 @@ admin_credential_hash = "${secrets.pty_admin_hash}" # logical reference in the [pty] enabled = true listen = "0.0.0.0:8090" -tls_cert = "/etc/openab-pty/tls.crt" # termination (a): in-process TLS. Off-loopback -tls_key = "/etc/openab-pty/tls.key" # bind without TLS material fails closed unless -tls_terminated_upstream = false # tls_terminated_upstream = true (termination (b)) +tls_terminated_upstream = true # trusted Ingress terminates TLS (MVP default); + # no TLS private key exists in this container command = "/bin/bash" max_sessions = 4 absolute_session_ttl = "12h" @@ -183,13 +191,18 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized ### Security model -- **Transport / TLS contract**: WSS is mandatory for external clients. Two supported terminations, chosen explicitly per deployment: (a) `openab-pty` terminates TLS itself (cert mounted into the container), or (b) a trusted Ingress terminates TLS and forwards plain WS internally -- in which case the internal listener accepts non-loopback plain WS only when the deployment declares `tls_terminated_upstream = true`, and the residual internal-hop exposure is documented. Fail-closed in all cases: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) +- **Transport / TLS contract**: WSS is mandatory for external clients. **MVP default -- and the only supported mode while the runtime and PTY children share a UID: termination (b), a trusted Ingress terminates TLS** and forwards plain WS internally; the internal listener accepts non-loopback plain WS only when the deployment declares `tls_terminated_upstream = true`, and the residual internal-hop exposure is documented. **Termination (a), in-process TLS with a mounted certificate key, is gated on a real runtime/child privilege or mount-namespace boundary** (the same gating as HMAC bridge secrets and signed tokens): a `tls.key` readable by a same-UID child is persistent, stealable authority. **Risk grading, stated so the two secrets are not conflated**: a stolen TLS private key is *transport* authority (endpoint impersonation/MITM for that deployment); the admin credential hash is a *non-reversible verifier* (grants nothing when read) -- which is exactly why the key must stay outside the same-UID container while the hash may live inside it. Fail-closed in all cases: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) - **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); constant-time comparison carries over. **Origin policy, decided explicitly (not "carried over")**: the as-built `/acp` consults `Origin` only on its keyless loopback path; its keyed bearer path never checks Origin -- and PTY has no keyless mode, so there is nothing to carry over. PTY's trust boundary is **bearer-only**: possession of a valid attach token is the sole authorization, and the `Origin` header is not consulted (it is attacker-controlled outside browsers and adds no strength to a keyed WebSocket). Browser-side token hygiene is governed by the client storage contract below - **Token control plane** (MVP model; an identity layer remains explicitly out of scope per `identity-trust-none.md`): - **Create requires authentication -- and locality is not authentication**: the shell child may share the container (and potentially the UID) with the runtime, so a loopback/UDS endpoint alone is not an auth barrier. MVP mechanism: session creation (`openab-pty session create `) requires an **admin bootstrap credential** delivered to the operator at deploy time and never present in the PTY child's environment or filesystem view. Hardening alternative (non-MVP): a distinct privileged UID/group for the runtime with UDS peer-credential verification. Unauthenticated remote create/list/kill is NOT provided in MVP; the admin credential and the attach token are distinct planes -- an attach token can never create, list, or kill - - **Admin-credential delivery channels (MUST)** -- the containment model rests on the plaintext credential never being observable by a same-UID shell child, so every channel is enumerated and closed, not just env and files: the plaintext exists **only on the operator's trusted side** (their terminal/secret manager); in-container presentation accepts it **only via short-lived non-echoing stdin or a UDS message body**; it is **never** accepted as an argv flag (`/proc//cmdline` is world-readable), never placed in any process environment inside the container, never written to temporary files, and never emitted to audit logs or errors (log a fingerprint of the verifier hash, not the credential). A same-UID shell observing one create/renew invocation must learn nothing reusable - - **One-time issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, returned exactly once. **Issued once, not single-use**: the bearer token remains valid for reattach until its expiry or a generation bump -- reconnecting clients are not locked out; theft exposure is bounded by a short default TTL (well below the session TTL) - - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. **Renew-while-attached, defined**: an actively attached connection is evicted via the standard evict order (notify, close with a distinct close code) before the fresh token is returned -- renew's use cases include suspected theft, so the possibly-hostile attached client must not linger under a revoked generation. Renew is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface + - **Admin-credential delivery channels (MUST)** -- the containment model rests on the plaintext credential never being observable by a same-UID shell child, so every channel is enumerated and closed: + - The plaintext exists **only on the operator's trusted side** (their terminal/secret manager) + - In-container presentation accepts it **only via short-lived non-echoing stdin or a UDS message body**; it is **never** accepted as an argv flag (`/proc//cmdline` is world-readable), never placed in any process environment inside the container, never written to temporary files + - Audit logs and errors never contain it -- log a fingerprint of the verifier hash, not the credential + - **The containment discipline applies to every in-container process that touches the plaintext, not only the runtime**: any CLI/helper on the presentation path MUST set `PR_SET_DUMPABLE=0` **before reading any input**, use bounded buffers, and zeroize before exit -- otherwise the same-UID ptrace/`/proc` window merely moves from the runtime to the helper. Preferred implementation: the CLI never materializes the credential at all and splices its stdin directly into the runtime's UDS, so exactly one process (the runtime) ever holds the plaintext + - A same-UID shell observing one create/renew invocation must learn nothing reusable -- and Phase 1 carries a **same-UID adversary test** (ptrace//proc probes against every credential-handling process yield `EPERM`) plus a **dumpable regression guard** (assert dumpability stays 0 after startup; a dependency calling `prctl(PR_SET_DUMPABLE, 1)` must fail the test, because it silently collapses this model) + - **Issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, **minted and returned exactly once at creation -- and valid for multiple reattaches** until its expiry or a generation bump (not single-use); reconnecting clients are not locked out, and theft exposure is bounded by a short default TTL (well below the session TTL) + - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. **Renew-while-attached, defined**: an actively attached connection is terminated via **connection-evict** (see the named sequences in Session lifecycle -- never session-teardown) with a **renew-distinct close code**, so an evicted client can tell renewal from takeover. **Renew is admin-initiated and disruptive by design**: it may cut an active session's connection -- including the admin's own if they renew while attached -- which is the correct behavior for its primary use cases (expired or suspected-stolen tokens). Renew is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface - **Per-session revocation**: kill/recreate bumps the generation and deletes the stored token hash, immediately invalidating outstanding tokens for that session; runtime restart clears all token state (sessions die with the process anyway, so this is not a loss) - **Token format (MVP): no signing key exists.** Each attach token is a CSPRNG 256-bit opaque bearer value; the runtime stores only its hash together with `(session ID, generation, scope = attach-only, expiry)` in memory and deletes it on kill/expiry. Because sessions deliberately do not survive a runtime restart, self-contained signed tokens buy nothing in MVP -- and eliminating the signing key eliminates the minting authority a same-container shell could steal. Signed (HMAC) tokens are a later option and require either an external signer outside the PTY container or the runtime/child privilege boundary below @@ -205,12 +218,21 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized - **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong at a 15-30s interval; a half-open socket counts as detached after 2-3 missed pings -- exact values are Phase 1 config with these recommended defaults, balancing flaky mobile networks against dead-client slot pinning) - **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached -- capacity cannot be pinned forever by an open browser tab. Expiry is client-visible: a warning control frame precedes forced teardown, and the WebSocket closes with a distinct close code so clients surface "session expired" instead of retrying a network error -- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3). **Takeover abuse controls**: every successful preempt is audited as an anomaly event (session, source address, count), and preempt frequency is rate-limited per session (e.g. max N takeovers/min, then attaches are rejected with a distinct close code) -- a stolen still-valid token must not be able to silently ping-pong the CAS and starve the legitimate client; the audit trail plus `session renew` is the recovery path. **Ordering contract**: token revocation (generation bump + stored-hash deletion), the attach CAS, and replay registration execute under one session lock in that total order -- no interleaving where a revoked token wins an attach or a replay registers against a stale generation +- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3). **Takeover abuse controls**: every successful preempt is audited as an anomaly event (session, source address, count), and preempt frequency is rate-limited per session (e.g. max N takeovers/min, then attaches are rejected with a distinct close code) -- a stolen still-valid token must not be able to silently ping-pong the CAS and starve the legitimate client. **The limiter is scoped to the session generation**: a generation bump (renew/recreate) resets the bucket, and the first attach under a new generation always bypasses an exhausted bucket -- so a thief who exhausted the budget can never lock the victim out of the `session renew` recovery path. **Lock ordering (total)**: token revocation (generation bump + stored-hash deletion), the attach CAS, and replay registration execute in that order under the session lock; where the buffer lock is also needed (replay registration), **the session lock is always acquired before the buffer lock and never the reverse** -- no interleaving where a revoked token wins an attach, a replay registers against a stale generation, or two paths deadlock across the two locks - **Reconnect**: monotonic byte cursor from day one -- the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes. The replay-to-live handoff is **atomic**: the subscriber registers under the buffer lock, captures the end offset, replays through it, then drains queued live bytes -- with connection-generation fencing so teardown of a replaced connection cannot affect its successor. On overflow the server sends an explicit `gap` control frame (bytes-dropped count) so the client can trigger a full clear/redraw instead of rendering a sliced ANSI stream - **Output-path bounds (MUST)**: every buffer on the PTY-to-client path is bounded, not just retained scrollback -- the replay-to-live handoff queue and the per-connection outbound backlog each have a fixed cap. A client too slow to drain its backlog gets a `gap` frame (drop-oldest, cursor advances) or, past a hard watermark, is disconnected with a distinct close code -- mirroring the input-side fail-closed backpressure so neither direction can grow unbounded memory - **`scrollback_replay` vs cursor semantics** (distinct controls): incremental `since` replay is always available within the ring buffer's retention; `scrollback_replay` governs only the cursor-less full-history dump on a fresh attach (default off -- secrets-safe); setting `scrollback_kib = 0` disables retention entirely, which also disables `since` replay (every reconnect starts with a `gap` + reset) -- **Teardown**: setpgid on spawn; SIGTERM-grace-SIGKILL escalation on the process group; evict-while-attached order = notify client, close socket, kill group, close master fd, release slot; buffers cleared on teardown; scrollback never touches disk -- **Kill domain (MUST)**: the process group is only the first signal path, not the containment guarantee -- a child that calls `setsid` or double-forks escapes the pgid. **MVP default (works under the stated container contract, no extra capabilities): a pidfd-based descendant reaper** -- the runtime sets `PR_SET_CHILD_SUBREAPER` so escaped descendants reparent to it, holds a pidfd per tracked process, and performs race-safe descendant discovery before killing (re-scan until stable, then kill via pidfds). **The cgroup path (`cgroup.kill`, or freeze-then-kill) is the stronger boundary but is gated on an explicit prerequisite**: cgroup v2 with subtree delegation to the container's UID, which the default non-root/no-capabilities contract does not provide -- deployments that arrange delegation SHOULD prefer it. **Startup probe, fail closed**: at startup the runtime verifies its configured kill mechanism is operational (subreaper flag set and pidfd support, or a writable delegated cgroup subtree) and refuses to serve sessions otherwise. Slot release and the absolute TTL are enforced against this hard boundary, never against the pgid alone +- **Two named termination sequences** (they are different operations and must never be conflated): + - **Connection-evict** -- ends a *connection*, the session process survives: notify the client, close the socket with the operation-specific close code (takeover, renew, TTL warning). Used by attach takeover and renew-while-attached + - **Session-teardown** -- ends the *session*: setpgid on spawn; SIGTERM-grace-SIGKILL escalation; evict-while-attached order = notify client, close socket, kill (per the Kill domain below), close master fd, release slot; buffers cleared on teardown; scrollback never touches disk +- **Kill domain (MUST)** -- the process group is only the first signal path, not the containment guarantee (a child that calls `setsid` or double-forks escapes the pgid): + - **MVP default (works under the stated container contract, no extra capabilities): a pidfd-based descendant reaper.** The runtime sets `PR_SET_CHILD_SUBREAPER` so escaped descendants reparent to it, and holds a pidfd per tracked process + - **Reaper scope**: discovery and kill are bounded to the runtime's **own spawned tree** (processes it created directly or via PTY children) -- never a blanket `/proc` sweep. Being a subreaper reparents *all* container orphans to the runtime; reparented processes outside a session's tree are reaped (waited on) but never killed + - **Convergence invariant**: teardown is a **kill-and-rescan loop until no session descendants remain** -- a one-shot scan-then-kill is not race-free (a tracked process can fork between the final scan and the kills). Subreaper reparent-and-reap is the convergence guarantee; the session slot is released only after the invariant holds + - **Resource budget**: tracked processes and their pidfds are capped per session and globally, with reserved FD headroom for control/WebSocket sockets; hitting tracking capacity is fail-closed (the session is killed, never left partially tracked) + - **The cgroup path (`cgroup.kill`, or freeze-then-kill) is the stronger boundary, gated on an explicit prerequisite**: cgroup v2 with subtree delegation to the container's UID, which the default non-root/no-capabilities contract does not provide -- deployments that arrange delegation SHOULD prefer it + - **Startup probe, fail closed**: at startup the runtime verifies its configured kill mechanism is operational (subreaper flag set and pidfd support, or a writable delegated cgroup subtree) and refuses to serve sessions otherwise. Slot release and the absolute TTL are enforced against this hard boundary, never against the pgid alone +- **Self-exit, defined (Phase 1 behavior, not deferred with the full state machine)**: when the child exits on its own, the runtime reaps it (kill-domain convergence still runs for surviving descendants), sends any attached client a final output flush plus a **session-ended close code** (distinct from TTL expiry and eviction), releases the slot after convergence, and deletes the session's token state -- the name then behaves exactly like reattach-to-dead: a distinct error offering restart-in-place. Termination classes (user-kill / self-exit / runtime-shutdown) are tagged in audit from Phase 1; the richer state machine remains Phase 3 - **Recovery taxonomy** (stated, not implied): detach/reattach survives (process alive); pod restart does not (process dead) -- reattach-to-dead returns a distinct error and offers **restart-in-place**: same session name, a fresh process and a new generation (old tokens invalid, empty scrollback). Pod-lifetime durability is out of scope and documented as such --- @@ -223,11 +245,12 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized - Fills the remote + sandboxed + raw-terminal quadrant with a real container boundary instead of a claimed one - Highest reversibility: default-off, separately versioned, separately deprecable - Coexistence where it matters (shared workspace) without shared process or credential-mount domains; network namespace and pod fate are shared only in the colocated profile (see Isolation tiers) -- The Phase 4 notification bridge (broker pulls from the sidecar -> relays to Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes +- The Phase 4 notification bridge (broker pulls from `openab-pty` -> relays to Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes ### Negative - A second binary and image to build, test, and release (mitigated by the existing multi-binary workspace and release pipeline) +- **No HA or horizontal scale by design**: the keyless in-memory token model binds every session and token to one runtime process -- no multi-replica serving, no failover; a runtime crash or OOM invalidates all sessions and tokens simultaneously. This is consistent with the sessions-die-with-the-process contract, but it is a real availability trade accepted for the elimination of at-rest minting authority - Cross-container coordination (notification bridge, future shared-crate extraction) is more ceremony than in-process calls - Some duplication with the ACP pool (capacity accounting, pgid kill) until a shared lifecycle crate is justified by real usage @@ -258,12 +281,14 @@ Zero code and genuinely useful for cluster admins — but it requires kubectl cr ### E. Do nothing / remain ACP-only (rejected) -Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's host blast radius. The sidecar form lets OAB serve it without betting the broker's identity. +Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's host blast radius. The composable-runtime form lets OAB serve it without betting the broker's identity. --- ## 5. Implementation Plan +**Pre-implementation gate (go/no-go)**: Phase 1 starts only after a demand check with measurable criteria set by the maintainers -- at minimum, linked user requests beyond the originating discussion thread and a maintainer-agreed operating-cost budget. Acceptance of this ADR records the *design*, not a commitment to build on a schedule; the 12-month adoption review below is the post-ship counterpart of this gate. + ### Phase 1: `openab-pty` MVP (new crate, new binary) - Own session manager: named sessions, operator-configured command, allowlist-validated names @@ -275,15 +300,19 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Monotonic cursor reconnect with atomic replay/live handoff and gap signaling; scrollback in-memory, off-by-default fresh-attach replay, cleared on teardown - Detached-idle TTL + absolute lifetime cap (with client-visible expiry warning + close code); single-attach exclusive - Audit log (attach/detach/create/kill/auth-failure) and basic metrics +- `openab-pty --validate-projection `: the fail-closed startup guard exposed as a standalone subcommand, so operators can verify a hand-generated projection from day one (Phase 2 CI reuses it as the guard test) +- **Same-UID adversary tests**: ptrace/`/proc//{mem,fd}` probes against every credential-handling process return `EPERM`; dumpability asserted to remain 0 after startup (regression guard against a dependency re-enabling it); kill-domain convergence test with a `setsid`/double-fork escapee and a SIGTERM-trapping stubborn child (zero orphans, zero FD leaks) - Resize propagation (TIOCSWINSZ) including attach-time initial size - Terminal-capability response filtering at the PTY boundary (known Ink-CLI startup breakage) ### Phase 2: Deployment + web client -- Helm: independent `openab.enabled` / `pty.enabled` toggles (or `--set profile=acp|pty|full`); standalone profile gets its own Service/Ingress (`/pty/*`) and NetworkPolicy example; config split documented per the configUrl pattern; `ghcr.io/openabdev/openab-pty` image published from the existing release pipeline -- Minimal xterm.js page served by `openab-pty`; remote session list/create/kill endpoints gated by the **admin bootstrap credential** (the same control-plane contract as the Phase 1 CLI -- never attach tokens; attach-token scope remains attach-only) +- Helm: independent `openab.enabled` / `pty.enabled` toggles (or `--set profile=acp|pty|full`); standalone profile gets its own Service/Ingress (`/pty/*`) and NetworkPolicy example; config split documented per the configUrl pattern; `ghcr.io/openabdev/openab-pty` image published from the existing release pipeline. **The chart never defaults to `full`**: the colocated profile is opt-in only and its values file is labeled convenience-only, pointing at the Isolation tiers table -- the default demo/production path is separate pods +- **Web client is attach-only (browser management deferred)**: the minimal xterm.js page served by `openab-pty` accepts an attach token and connects -- nothing else. Remote list/create/kill/renew endpoints exist for *non-browser* admin tooling only, gated by the admin bootstrap credential; **the web client never receives, stores, or transmits the admin credential** -- delivering the global management credential to a browser would turn one XSS into administration of every session. A browser management UI requires an operator-mediated pairing / one-time scoped-issuance flow or the identity layer, and is explicitly deferred until one exists +- **Client-page acceptance criteria (testable)**: attach token held in memory only (never localStorage/sessionStorage/cookies/URLs); page served with CSP enforcing at minimum `script-src 'self'` (no inline/eval), `object-src 'none'`, `base-uri 'none'`, `frame-ancestors 'none'`, and `connect-src` limited to the PTY origin; no third-party runtime scripts +- **Admin-credential rotation runbook (acceptance criterion)**: documented steps, blast radius, and expected downtime for rotation (generate new value -> update delivered hash -> restart runtime -> all sessions cleared) -- operators must not discover mid-incident that rotation kills every session - Rollback contract (Phase 2 acceptance criterion): disabling the toggle drains (notify + grace, honoring `terminationGracePeriodSeconds`) then kills sessions; the broker container is unaffected and not restarted. Projection updates roll the PTY container only -- live sessions die on the rollout (non-persistent by contract) and the chart documents this; the workspace PVC is untouched by disable/re-enable, and re-enable is a fresh runtime with zero sessions -- Projection tooling has an owner (Phase 2 acceptance criterion): the Helm chart generates both config views, and CI runs a **guard test** -- the delivered PTY projection is fed to `openab-pty`'s startup validator, which must accept it and must reject a deliberately poisoned projection (embedded `[secrets.refs]`, `${secrets.*}`, or a broker section). The "never delivered" invariant is thereby enforced by mechanism, not operator discipline +- Projection tooling has an owner (Phase 2 acceptance criterion): the Helm chart generates both config views, and CI runs a **guard test** -- the delivered PTY projection is fed to `openab-pty --validate-projection` (the Phase 1 startup validator exposed as a subcommand), which must accept it and must reject a deliberately poisoned projection (embedded `[secrets.refs]`, `${secrets.*}`, or a broker section). The "never delivered" invariant is thereby enforced by mechanism, not operator discipline ### Phase 3: Lifecycle hardening @@ -295,13 +324,13 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - `openab-pty` exposes a pod-local, loopback-only notification stream; the **broker pulls** (long-poll/SSE on localhost) when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); the broker relays to the platform thread. Bridge is one-way and feature-gated - **No bridge secret enters the PTY container -- by design, stated now**: a delivered HMAC key would recreate exactly the in-container authority the keyless token model eliminated (a same-UID child can read any file the runtime can read; 0400 at the same UID is not a boundary). The pull model removes the broker-side ingress entirely: there is no webhook endpoint to leave open and no shared key to steal. Residual risk, stated: a same-UID child that kills the runtime (an accepted same-UID risk) could bind the freed port and forge events -- therefore the broker treats bridge events as **display-only, rate-limited hints**: they never carry commands, never mutate broker state, and are labeled best-effort in the relayed message. A push/webhook variant with an HMAC secret is permitted only with an external signer outside the PTY container or the runtime/child privilege boundary from the Security model (non-MVP hardening) - **Not available in the PTY-only profile** — there is no broker to relay through, and `openab-pty` will not grow its own notifier (that would recreate the scope creep this ADR exists to avoid). Users who want notifications deploy profile 3 -- **Pull-stream resource contract**: the notification stream is bounded like every other surface -- at most **one** concurrent broker stream (a new connection preempts the old), heartbeat with an idle timeout on both ends, reconnect with capped exponential backoff on the broker side, and a fixed-size event queue in `openab-pty` with coalescing (per-session dedupe: a newer idle event replaces an older undelivered one) and drop-oldest overflow. "Rate-limited hints" thus constrains retained resources, not only delivery semantics +- **Pull-stream resource contract**: the notification stream is bounded like every other surface -- at most **one** concurrent broker stream with **incumbent-wins admission**: while a healthy stream exists (heartbeating within its idle timeout), new connection attempts are rejected; a replacement is admitted only after heartbeat timeout declares the incumbent dead. This closes both the churn vector (forced close/reconnect work) and the live-hijack vector -- **stated residual**: the stream is unauthenticated pod-local, so a same-pod process that connects *first* (or after killing the runtime and binding the freed port) can occupy it; display-only rate-limited hints bound the impact in every case. Heartbeat with an idle timeout on both ends, reconnect with capped exponential backoff on the broker side, and a fixed-size event queue in `openab-pty` with coalescing (per-session dedupe: a newer idle event replaces an older undelivered one) and drop-oldest overflow. "Rate-limited hints" thus constrains retained resources, not only delivery semantics ### Later (demand-gated, explicitly deferred) - Shared lifecycle crate extraction (if the ACP pool and PTY manager converge naturally). Candidate shared surface: spawn mechanics, env-allowlist construction, pgid kill/escalation; deliberately NOT shared: liveness definitions, TTL/eviction policy, persistence - **Adoption review point**: 12 months after the standalone profile ships, review its usage; below a threshold the maintainers set then, consider deprecating the standalone image or folding PTY back to colocate-only -- Single-process merge (only if operations prove the sidecar split is more cost than benefit) +- Single-process merge (only if operations prove the runtime split is more cost than benefit) - Identity layer for PTY tokens; semantic agent-state detection; JSONL transcript channel (see Alternative B) --- @@ -329,7 +358,7 @@ The full survey from the superseded proposal carries over unchanged in substance | Semantic agent state detection | Per-agent detection manifests classify panes as working/blocked/idle/done, with an explain API for rule provenance | Later (demand-gated) | | Race-safe waits | Server-owned event-driven waits pinned to the pane occupant; atomic prompt+wait | Later (demand-gated) | | Layered restore taxonomy | Live persistence / live handoff / native session restore / history replay (off by default: secrets) / layout-only snapshot | Phase 1 adopts the secrets-safe default and the recovery taxonomy | -| Multiple read projections | `visible` / `recent` / `recent-unwrapped` / `detection` views of one PTY | Phase 2/3 | +| Multiple read projections | `visible` / `recent` / `recent-unwrapped` / `detection` views of one PTY | Later (demand-gated) -- no phase deliverable exists; if adopted, views are lazy, bounded, and charged to the owning session's memory envelope | | Callback env injection | Spawned processes receive the runtime's socket path so in-pane agents can drive it | Later (demand-gated; A2A needs its own ADR) | ### Claude Code cross-session messaging (v2.1.224+) From 7c09cdb530dca99999c31517a3614c599317656f Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 23:08:51 +0000 Subject: [PATCH 11/15] docs(adr): address late round-6 safety findings S3/S9/S10 (B7) - S3 (critical): admin plane is unreachable from managed sessions -- admin operations from any process in a managed session's tree are denied (lineage/peer-cred + pgid check), closing the server-side token-into-scrollback exfiltration channel; tokens are returned only to the external control client and never written to a PTY master; from-child create test added to Phase 1 - S9: prctl(PR_SET_DUMPABLE, 0) failure is fail-closed -- the runtime refuses to serve and helpers exit before reading input; the containment model never degrades silently - S10: the session lock covers state only, never I/O -- socket/PTY work executes after lock drop under generation fencing; kill/TTL paths cannot be blocked by a slow or non-reading client --- docs/adr/openab-pty-runtime.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index 3017a6af2..47e1f2630 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -120,7 +120,7 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - **Same-container containment (MVP model)**: - **What the keyless design removes**: runtime **persistent state** contains no minting authority -- only non-reversible hashes. There is no signing key and no stored plaintext for a same-UID shell to steal at rest - **What remains, stated precisely**: the runtime *is* the mint -- freshly minted tokens and the presented admin credential transiently exist in its process memory - - **Protecting the transient window is therefore load-bearing**: the runtime MUST set `PR_SET_DUMPABLE=0` **before any secret enters memory** -- this is what blocks same-UID `/proc//mem`, `/proc//fd`, and `ptrace` access on Linux; without it the containment claim does not hold + - **Protecting the transient window is therefore load-bearing**: the runtime MUST set `PR_SET_DUMPABLE=0` **before any secret enters memory** -- this is what blocks same-UID `/proc//mem`, `/proc//fd`, and `ptrace` access on Linux; without it the containment claim does not hold. **`prctl` failure is fail-closed**: if `PR_SET_DUMPABLE=0` cannot be set (seccomp policy, kernel restriction), the runtime refuses to serve and any credential-handling helper exits non-zero *before reading input* -- the same fail-closed style as the kill-domain startup probe; the containment model must never degrade silently - **Credential strength basis**: the admin bootstrap credential is generated by a CSPRNG (cryptographically secure pseudorandom number generator) at high entropy, and the runtime stores **only its verifier hash** -- reading the hash neither authorizes requests nor enables practical offline guessing; every control operation is authenticated by presenting the credential itself, even on a locally reachable socket - **Optional hardening (not MVP)**: spawning PTY children under a distinct unprivileged UID gives defense-in-depth, but a non-root UID-1000 process cannot `setuid` without `CAP_SETUID` or a privileged launcher -- deployments that can provide a narrowly-scoped launcher may adopt it; the default container contract (non-root, all capabilities dropped, `allowPrivilegeEscalation: false`) is preserved either way - **Same-UID residual-risk checklist (MUST)** -- because file modes are not a boundary between same-UID processes, the following are requirements, not recommendations: @@ -203,6 +203,7 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized - A same-UID shell observing one create/renew invocation must learn nothing reusable -- and Phase 1 carries a **same-UID adversary test** (ptrace//proc probes against every credential-handling process yield `EPERM`) plus a **dumpable regression guard** (assert dumpability stays 0 after startup; a dependency calling `prctl(PR_SET_DUMPABLE, 1)` must fail the test, because it silently collapses this model) - **Issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, **minted and returned exactly once at creation -- and valid for multiple reattaches** until its expiry or a generation bump (not single-use); reconnecting clients are not locked out, and theft exposure is bounded by a short default TTL (well below the session TTL) - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. **Renew-while-attached, defined**: an actively attached connection is terminated via **connection-evict** (see the named sequences in Session lifecycle -- never session-teardown) with a **renew-distinct close code**, so an evicted client can tell renewal from takeover. **Renew is admin-initiated and disruptive by design**: it may cut an active session's connection -- including the admin's own if they renew while attached -- which is the correct behavior for its primary use cases (expired or suspected-stolen tokens). Renew is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface + - **Admin plane is unreachable from managed sessions (MUST)**: the runtime rejects admin operations (create/renew/kill/restart) originating from any process inside a managed session's tree -- verified via the kill-domain lineage tracking (subreaper/pidfd ancestry) or UDS peer-credential plus session-pgid check. Rationale: running `session create/renew` *inside* a managed PTY would print the one-time token into the PTY byte stream, landing it in the scrollback ring and every attach/replay client -- a server-side exfiltration channel no client-storage rule can close. Tokens are returned only to the external control client and are never written to any PTY master. Phase 1 test: `session create` invoked from a managed child is denied and audited - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface - **Per-session revocation**: kill/recreate bumps the generation and deletes the stored token hash, immediately invalidating outstanding tokens for that session; runtime restart clears all token state (sessions die with the process anyway, so this is not a loss) - **Token format (MVP): no signing key exists.** Each attach token is a CSPRNG 256-bit opaque bearer value; the runtime stores only its hash together with `(session ID, generation, scope = attach-only, expiry)` in memory and deletes it on kill/expiry. Because sessions deliberately do not survive a runtime restart, self-contained signed tokens buy nothing in MVP -- and eliminating the signing key eliminates the minting authority a same-container shell could steal. Signed (HMAC) tokens are a later option and require either an external signer outside the PTY container or the runtime/child privilege boundary below @@ -218,7 +219,7 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized - **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong at a 15-30s interval; a half-open socket counts as detached after 2-3 missed pings -- exact values are Phase 1 config with these recommended defaults, balancing flaky mobile networks against dead-client slot pinning) - **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached -- capacity cannot be pinned forever by an open browser tab. Expiry is client-visible: a warning control frame precedes forced teardown, and the WebSocket closes with a distinct close code so clients surface "session expired" instead of retrying a network error -- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3). **Takeover abuse controls**: every successful preempt is audited as an anomaly event (session, source address, count), and preempt frequency is rate-limited per session (e.g. max N takeovers/min, then attaches are rejected with a distinct close code) -- a stolen still-valid token must not be able to silently ping-pong the CAS and starve the legitimate client. **The limiter is scoped to the session generation**: a generation bump (renew/recreate) resets the bucket, and the first attach under a new generation always bypasses an exhausted bucket -- so a thief who exhausted the budget can never lock the victim out of the `session renew` recovery path. **Lock ordering (total)**: token revocation (generation bump + stored-hash deletion), the attach CAS, and replay registration execute in that order under the session lock; where the buffer lock is also needed (replay registration), **the session lock is always acquired before the buffer lock and never the reverse** -- no interleaving where a revoked token wins an attach, a replay registers against a stale generation, or two paths deadlock across the two locks +- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3). **Takeover abuse controls**: every successful preempt is audited as an anomaly event (session, source address, count), and preempt frequency is rate-limited per session (e.g. max N takeovers/min, then attaches are rejected with a distinct close code) -- a stolen still-valid token must not be able to silently ping-pong the CAS and starve the legitimate client. **The limiter is scoped to the session generation**: a generation bump (renew/recreate) resets the bucket, and the first attach under a new generation always bypasses an exhausted bucket -- so a thief who exhausted the budget can never lock the victim out of the `session renew` recovery path. **Lock ordering (total)**: token revocation (generation bump + stored-hash deletion), the attach CAS, and replay registration execute in that order under the session lock; where the buffer lock is also needed (replay registration), **the session lock is always acquired before the buffer lock and never the reverse** -- no interleaving where a revoked token wins an attach, a replay registers against a stale generation, or two paths deadlock across the two locks. **The session lock covers state only, never I/O**: it protects the state machine (generation, token hashes, `owner_conn_generation`, subscriber registration metadata); all socket and PTY I/O -- including notifying/closing a preempted connection and draining replay bytes -- executes after the lock is dropped, fenced by generation so stale work is ignored. Kill and TTL paths are never blocked by a per-connection drain (bounded wait or lock-free signal), so a slow or malicious non-reading client cannot delay renew, expiry, or teardown - **Reconnect**: monotonic byte cursor from day one -- the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes. The replay-to-live handoff is **atomic**: the subscriber registers under the buffer lock, captures the end offset, replays through it, then drains queued live bytes -- with connection-generation fencing so teardown of a replaced connection cannot affect its successor. On overflow the server sends an explicit `gap` control frame (bytes-dropped count) so the client can trigger a full clear/redraw instead of rendering a sliced ANSI stream - **Output-path bounds (MUST)**: every buffer on the PTY-to-client path is bounded, not just retained scrollback -- the replay-to-live handoff queue and the per-connection outbound backlog each have a fixed cap. A client too slow to drain its backlog gets a `gap` frame (drop-oldest, cursor advances) or, past a hard watermark, is disconnected with a distinct close code -- mirroring the input-side fail-closed backpressure so neither direction can grow unbounded memory - **`scrollback_replay` vs cursor semantics** (distinct controls): incremental `since` replay is always available within the ring buffer's retention; `scrollback_replay` governs only the cursor-less full-history dump on a fresh attach (default off -- secrets-safe); setting `scrollback_kib = 0` disables retention entirely, which also disables `since` replay (every reconnect starts with a `gap` + reset) @@ -301,7 +302,7 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Detached-idle TTL + absolute lifetime cap (with client-visible expiry warning + close code); single-attach exclusive - Audit log (attach/detach/create/kill/auth-failure) and basic metrics - `openab-pty --validate-projection `: the fail-closed startup guard exposed as a standalone subcommand, so operators can verify a hand-generated projection from day one (Phase 2 CI reuses it as the guard test) -- **Same-UID adversary tests**: ptrace/`/proc//{mem,fd}` probes against every credential-handling process return `EPERM`; dumpability asserted to remain 0 after startup (regression guard against a dependency re-enabling it); kill-domain convergence test with a `setsid`/double-fork escapee and a SIGTERM-trapping stubborn child (zero orphans, zero FD leaks) +- **Same-UID adversary tests**: ptrace/`/proc//{mem,fd}` probes against every credential-handling process return `EPERM`; dumpability asserted to remain 0 after startup (regression guard against a dependency re-enabling it); `prctl(PR_SET_DUMPABLE, 0)` failure refuses service (fail-closed); `session create` invoked from inside a managed child is denied and audited; kill-domain convergence test with a `setsid`/double-fork escapee and a SIGTERM-trapping stubborn child (zero orphans, zero FD leaks) - Resize propagation (TIOCSWINSZ) including attach-time initial size - Terminal-capability response filtering at the PTY boundary (known Ink-CLI startup breakage) From 9df59bf7aecfb12d996043a1f2707a68e75d3643 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 23:10:42 +0000 Subject: [PATCH 12/15] docs(adr): consolidate keyless-model trade-offs in Consequences (B11) One bullet now covers the shared root of O7 and F18: no HA/failover, and crash/restart/rollout/rotation all clear every session and token -- the deliberate exchange for eliminating at-rest minting authority. --- docs/adr/openab-pty-runtime.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index 47e1f2630..811a01627 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -251,7 +251,7 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized ### Negative - A second binary and image to build, test, and release (mitigated by the existing multi-binary workspace and release pipeline) -- **No HA or horizontal scale by design**: the keyless in-memory token model binds every session and token to one runtime process -- no multi-replica serving, no failover; a runtime crash or OOM invalidates all sessions and tokens simultaneously. This is consistent with the sessions-die-with-the-process contract, but it is a real availability trade accepted for the elimination of at-rest minting authority +- **The keyless in-memory model's cost, consolidated**: every session and token is bound to one runtime process -- no HA, no multi-replica serving, no failover; a crash, OOM, restart, projection rollout, or admin-credential rotation (which requires a restart) invalidates **all** sessions and tokens simultaneously. This is the deliberate exchange for eliminating at-rest minting authority, and it is why the rotation runbook (Phase 2) must state the blast radius up front - Cross-container coordination (notification bridge, future shared-crate extraction) is more ceremony than in-process calls - Some duplication with the ACP pool (capacity accounting, pgid kill) until a shared lifecycle crate is justified by real usage From b205928c826137c39863b360ea96bd00e264bf07 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 23:12:15 +0000 Subject: [PATCH 13/15] docs(adr): dumpable must precede FD possession, splice-only CLI included (B7 F1') PR_SET_DUMPABLE=0 is required before a credential-handling process holds any FD that may carry the credential, not merely before the first userspace read -- with dumpable=1 a same-UID process can drain an inherited pipe via /proc//fd/*. A splice-only CLI holds the carrying FDs and therefore remains subject to the containment MUST and the Phase 1 same-UID adversary tests. --- docs/adr/openab-pty-runtime.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index 811a01627..3110775dd 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -199,7 +199,7 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized - The plaintext exists **only on the operator's trusted side** (their terminal/secret manager) - In-container presentation accepts it **only via short-lived non-echoing stdin or a UDS message body**; it is **never** accepted as an argv flag (`/proc//cmdline` is world-readable), never placed in any process environment inside the container, never written to temporary files - Audit logs and errors never contain it -- log a fingerprint of the verifier hash, not the credential - - **The containment discipline applies to every in-container process that touches the plaintext, not only the runtime**: any CLI/helper on the presentation path MUST set `PR_SET_DUMPABLE=0` **before reading any input**, use bounded buffers, and zeroize before exit -- otherwise the same-UID ptrace/`/proc` window merely moves from the runtime to the helper. Preferred implementation: the CLI never materializes the credential at all and splices its stdin directly into the runtime's UDS, so exactly one process (the runtime) ever holds the plaintext + - **The containment discipline applies to every in-container process that touches the plaintext, not only the runtime**: any CLI/helper on the presentation path MUST set `PR_SET_DUMPABLE=0` **before holding any file descriptor that may carry the credential -- not merely before the first userspace read** (with dumpable=1, a same-UID process can drain an inherited pipe via `/proc//fd/*` even if the helper never copies the bytes itself), use bounded buffers, and zeroize before exit -- otherwise the same-UID ptrace/`/proc` window merely moves from the runtime to the helper. Preferred implementation: the CLI never materializes the credential at all and splices its stdin directly into the runtime's UDS, so exactly one process (the runtime) ever holds the plaintext in userspace -- **a splice-only CLI still counts as credential-handling** (it holds the carrying FDs) and remains subject to this MUST and the Phase 1 adversary tests - A same-UID shell observing one create/renew invocation must learn nothing reusable -- and Phase 1 carries a **same-UID adversary test** (ptrace//proc probes against every credential-handling process yield `EPERM`) plus a **dumpable regression guard** (assert dumpability stays 0 after startup; a dependency calling `prctl(PR_SET_DUMPABLE, 1)` must fail the test, because it silently collapses this model) - **Issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, **minted and returned exactly once at creation -- and valid for multiple reattaches** until its expiry or a generation bump (not single-use); reconnecting clients are not locked out, and theft exposure is bounded by a short default TTL (well below the session TTL) - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. **Renew-while-attached, defined**: an actively attached connection is terminated via **connection-evict** (see the named sequences in Session lifecycle -- never session-teardown) with a **renew-distinct close code**, so an evicted client can tell renewal from takeover. **Renew is admin-initiated and disruptive by design**: it may cut an active session's connection -- including the admin's own if they renew while attached -- which is the correct behavior for its primary use cases (expired or suspected-stolen tokens). Renew is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface From 3baa5578b5fcb9bd411e464c5f39e5a43ce7b979 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 23:12:54 +0000 Subject: [PATCH 14/15] docs(adr): adversary tests explicitly cover the splice CLI's FDs (B7 F1' item 3) --- docs/adr/openab-pty-runtime.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index 3110775dd..e71c20809 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -302,7 +302,7 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Detached-idle TTL + absolute lifetime cap (with client-visible expiry warning + close code); single-attach exclusive - Audit log (attach/detach/create/kill/auth-failure) and basic metrics - `openab-pty --validate-projection `: the fail-closed startup guard exposed as a standalone subcommand, so operators can verify a hand-generated projection from day one (Phase 2 CI reuses it as the guard test) -- **Same-UID adversary tests**: ptrace/`/proc//{mem,fd}` probes against every credential-handling process return `EPERM`; dumpability asserted to remain 0 after startup (regression guard against a dependency re-enabling it); `prctl(PR_SET_DUMPABLE, 0)` failure refuses service (fail-closed); `session create` invoked from inside a managed child is denied and audited; kill-domain convergence test with a `setsid`/double-fork escapee and a SIGTERM-trapping stubborn child (zero orphans, zero FD leaks) +- **Same-UID adversary tests**: ptrace/`/proc//{mem,fd}` probes against every credential-handling process return `EPERM` -- **explicitly including the splice-only CLI's file descriptors, not just the runtime**; dumpability asserted to remain 0 after startup (regression guard against a dependency re-enabling it); `prctl(PR_SET_DUMPABLE, 0)` failure refuses service (fail-closed); `session create` invoked from inside a managed child is denied and audited; kill-domain convergence test with a `setsid`/double-fork escapee and a SIGTERM-trapping stubborn child (zero orphans, zero FD leaks) - Resize propagation (TIOCSWINSZ) including attach-time initial size - Terminal-capability response filtering at the PTY boundary (known Ink-CLI startup breakage) From a70b41f336bdf598d983ef050b3f2f029a7fb6d6 Mon Sep 17 00:00:00 2001 From: chaodu-agent <274062505+chaodu-agent@users.noreply.github.com> Date: Sat, 15 Aug 2026 21:46:55 -0400 Subject: [PATCH 15/15] docs(adr): scope MVP to separate pods; cgroup-per-session kill domain; resolve round-7 F1-F8 Owner decision: MVP ships profiles 1+2 only (separate pods, full-isolation tier); colocated profile 3 and the Phase 4 bridge move behind an explicit demand gate. - F1: per-session delegated cgroup (cgroup.kill) is the authoritative attribution primitive and a stated Phase 1 prerequisite; subreaper/pidfd demoted to reaping only; fail-safe unattributed-orphan policy; the feasibility gate can fail on delegation - F2: credential presentation narrowed to UDS body or interactive TTY (isatty; piped stdin rejected) -- dumpable-before-FD cannot hold for an inherited pipe - F3: admin_credential_hash validator is literal-only, non-empty, format-validated (sha256:64hex); both poisoning cases join the CI guard - F4: memory admission formula ties per-session bounds to the container budget; Phase 2 observability named - F5: seccomp/delegation conditional folded into the fail-closed startup probe diagnostics - F6: executable gate numbers (3 independent users / 90 days) plus a <=1-week feasibility spike that can fail the gate - F7: attach-semantics and transport/TLS bullets restructured into named sub-bullets - F8: MUST inventories marked as target contracts; extraction into docs/pty-security-contract.md added as a Phase 1 deliverable --- docs/adr/openab-pty-runtime.md | 82 +++++++++++++++++++++++----------- 1 file changed, 55 insertions(+), 27 deletions(-) diff --git a/docs/adr/openab-pty-runtime.md b/docs/adr/openab-pty-runtime.md index e71c20809..5b17dd1e8 100644 --- a/docs/adr/openab-pty-runtime.md +++ b/docs/adr/openab-pty-runtime.md @@ -31,24 +31,26 @@ This ADR proposes the same capability in a shape that answers all five. ## 2. Decision -Ship **`openab-pty`**: a separate binary that is an **independently runnable runtime** — deployable standalone or colocated with the OAB broker. Not deployed by default. OAB remains a pure ACP broker; `openab-pty` owns everything terminal. +Ship **`openab-pty`**: a separate binary that is an **independently runnable runtime** — deployable standalone or colocated with the OAB broker. Not deployed by default. OAB remains a pure ACP broker; `openab-pty` owns everything terminal. **MVP deploys it standalone only (separate pods, profiles 1+2); colocation (profile 3) is a demand-gated later profile.** -**One codebase, two composable runtimes, three deployment modes:** +**One codebase, two composable runtimes, three deployment profiles — MVP ships two:** | Profile | Processes | Use case | |---|---|---| | 1. ACP only (current default) | `openab` | Message-broker deployments; no change from today | | 2. PTY only | `openab-pty` | Standalone remote terminal service: workspace PVC + `[pty]` config + admin bootstrap credential; no Discord/Slack tokens, no platform adapters, no ACP protocol | -| 3. ACP + PTY (colocated) | `openab` + `openab-pty` sidecar | Both in one pod sharing the workspace volume: drive a CLI by hand, let ACP agents continue in the same working tree from Discord | +| 3. ACP + PTY (colocated) — **demand-gated, not in MVP** | `openab` + `openab-pty` sidecar | Both in one pod sharing the workspace volume: drive a CLI by hand, let ACP agents continue in the same working tree from Discord | + +**MVP scope (decision): profiles 1 and 2 only — always separate pods, the full-isolation tier.** The colocated profile 3 is design-recorded in this ADR (it is the shape that answers the #1477 coexistence question) but is **demand-gated**: it ships only if the standalone profile proves demand and a maintainer explicitly re-opens it (see Section 5, Later). Everything colocated-specific below — the sidecar diagram, the partial-isolation tier, the Phase 4 notification bridge — is therefore a **target design recorded for that gate, not an MVP commitment**. In MVP, workspace sharing between an ACP pod and a PTY pod is possible only where the storage class supports RWX (`ReadWriteMany`); the chart documents this instead of offering the colocated convenience. Deployment mechanics: - **Own image**: `ghcr.io/openabdev/openab-pty` — smaller than the broker image (no platform adapter dependencies) - **Own Service/Ingress**: `/pty/*` routes to the `openab-pty` port in both profile 2 and 3; the broker listener never serves terminal traffic -- **Helm UX**: independent toggles (`openab.enabled` / `pty.enabled`) or a convenience `--set profile=acp|pty|full` +- **Helm UX (MVP)**: independent toggles (`openab.enabled` / `pty.enabled`) rendering **separate pods** — convenience form `--set profile=acp|pty`. The colocated `full` profile is **not rendered by the MVP chart**; it ships (opt-in only, never default) only if the profile-3 demand gate opens ``` -Profile 3 (colocated) — K8s Pod +Profile 3 (colocated; demand-gated target design — not in MVP) — K8s Pod +--------------------------------------------------------------------+ | | | Container: openab (broker) Container: openab-pty | @@ -80,16 +82,16 @@ Profile 2 makes `openab-pty` a small standalone product in OpenDray's category ( | Review blocker (PR #1477) | How the composable-runtime form resolves it | |---|---| | Positioning vs Thin Bridge | OAB binary is untouched; the broker stays a pure transport. `openab-pty` is an adjacent tool that shares deployment infrastructure only — no dual persona | -| Same-pod blast radius | Separate container = separate PID namespace, cgroup, filesystem, and mounts: the shell user cannot signal the broker, exhaust its container cgroup, or read its credential files. Broker platform tokens are **never mounted** into the PTY container. Residual sharing in the colocated profile (pod network namespace, pod fate) is graded honestly in Isolation tiers below; full isolation = profiles 1+2 as separate pods | +| Same-pod blast radius | Separate container = separate PID namespace, cgroup, filesystem, and mounts: the shell user cannot signal the broker, exhaust its container cgroup, or read its credential files. Broker platform tokens are **never mounted** into the PTY container. Residual sharing in the colocated profile (pod network namespace, pod fate) is graded honestly in Isolation tiers below; full isolation = profiles 1+2 as separate pods, **which is all MVP ships** | | Auth below capability | `openab-pty` designs its token model from scratch for shell-equivalent trust (see Security model) with no ACP-key coupling | | Pool incompatibility | `openab-pty` has its **own session manager** built for byte-stream lifecycle. No refactor of the shipped ACP pool; zero regression risk to the broker | | Reversibility | Default-off runtime with its own image/release. If demand does not materialize, deprecate the image; nothing in the broker to unwind. If demand proves out, later extraction of a shared lifecycle crate — or even single-process merge — remains open | ### Coexistence with ACP -ACP and PTY coexist per deployment, not per process: +ACP and PTY coexist per deployment, not per process. **In MVP, coexistence means two separate pods** (workspace sharing via RWX PVC where the storage class supports it); the same-pod form below is the demand-gated profile-3 target design: -- **Same pod, two containers (profile 3)** — one Helm toggle (`pty.enabled=true`) adds the sidecar; the broker container is byte-identical across all three profiles +- **Same pod, two containers (profile 3, demand-gated)** — one Helm toggle (`pty.enabled=true`) adds the sidecar; the broker container is byte-identical across all three profiles - **Shared workspace volume (opt-in)** — the PTY shell and ACP agents can see the same working tree (same PVC mount), which is the practical point of coexistence: drive a CLI by hand in the terminal, then let ACP agents continue in the same workspace from Discord. This sharing is an **explicit cross-runtime trust and concurrency bridge**, stated rather than implied: - *Trust*: the workspace is a single trust zone. Workspace-resident credentials (`.git/credentials`, `.env` files, agent OAuth stores) are readable by the shell regardless of mount hygiene, and either side can plant content (hooks, PATH-shadowing binaries) the other later executes. Treat ACP and PTY principals as sharing workspace authority when sharing is enabled - *Concurrency*: concurrent writes are best-effort and uncoordinated (a runtime non-goal). Recommended convention: separate git worktrees or session directories per principal; document RWO/RWX PVC implications in the chart @@ -101,8 +103,8 @@ ACP and PTY coexist per deployment, not per process: | Profile | Isolation tier | |---|---| -| 1 + 2 as separate pods | Full: independent network identity, NetworkPolicy, failure domains -- **recommended for production when strong isolation is required** | -| 3 (colocated sidecar) | Partial: process/filesystem/credential-mount separation only; shared pod network and fate; the per-session token requirement and auth on every broker listener are the remaining intra-pod barriers. A convenience tier for teams that accept this trade for same-workspace ergonomics | +| 1 + 2 as separate pods (**the MVP scope**) | Full: independent network identity, NetworkPolicy, failure domains -- **recommended for production when strong isolation is required** | +| 3 (colocated sidecar, **demand-gated — not in MVP**) | Partial: process/filesystem/credential-mount separation only; shared pod network and fate; the per-session token requirement and auth on every broker listener are the remaining intra-pod barriers. A convenience tier for teams that accept this trade for same-workspace ergonomics | ### Configuration: one source, two projected views @@ -131,7 +133,7 @@ Operators keep a **single logical `config.toml`** (the existing `configUrl` flow - **Rotation, stated**: `session renew` rotates attach tokens, never the admin credential. Rotating the admin credential = generating a new value, updating the delivered hash, and restarting the runtime -- which clears all sessions. This is acceptable (sessions are non-persistent by contract) and is the documented procedure - `PR_SET_DUMPABLE=0` is a **Linux-specific** mitigation; non-Linux targets are out of scope for MVP and must not be assumed covered - **Accepted risk, stated**: a same-UID child can signal (including SIGKILL) the runtime process; this is availability, not confidentiality -- sessions die with the runtime and no credential is exposed by the crash -- **Resolution asymmetry (deliberate)**: the broker resolves `${VAR}` interpolation and `[secrets.refs]` cloud references itself, as today. The PTY runtime accepts only literal values, `${VAR}` environment interpolation, and local file paths in its delivered projection -- it MUST NOT link or invoke a cloud secrets resolver at runtime. A delivered PTY projection that still contains a `[secrets.refs]` table, any unresolved cloud reference (`aws-sm://` etc.), **or any `${secrets.*}` interpolation** is a **startup error** (fail closed) -- `${secrets.*}` is enumerated explicitly because it shares the `${}` delimiters with the accepted `${VAR}` env form and must never be silently treated as an unset environment variable. This guard prevents an implementer from re-importing a cloud fetch identity into the PTY trust boundary +- **Resolution asymmetry (deliberate)**: the broker resolves `${VAR}` interpolation and `[secrets.refs]` cloud references itself, as today. The PTY runtime accepts only literal values, `${VAR}` environment interpolation, and local file paths in its delivered projection -- it MUST NOT link or invoke a cloud secrets resolver at runtime. A delivered PTY projection that still contains a `[secrets.refs]` table, any unresolved cloud reference (`aws-sm://` etc.), **or any `${secrets.*}` interpolation** is a **startup error** (fail closed) -- `${secrets.*}` is enumerated explicitly because it shares the `${}` delimiters with the accepted `${VAR}` env form and must never be silently treated as an unset environment variable. This guard prevents an implementer from re-importing a cloud fetch identity into the PTY trust boundary. **The validator is also fail-closed on the verifier itself**: in the delivered projection, `admin_credential_hash` MUST be **literal-only, non-empty, and format-validated** (`sha256:` followed by exactly 64 lowercase hex characters) -- a plain `${VAR}` interpolation on this key, an empty value, or a malformed hash is a startup error, never a silently-accepted config (a fail-open here would let a poisoned projection disable admin auth). Both poisoning cases -- unresolved interpolation and malformed/empty hash -- are added to the Phase 2 CI guard test alongside the `[secrets.refs]` / `${secrets.*}` cases ```toml # ---- LOGICAL operator source (what the operator maintains) ---- @@ -191,19 +193,25 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized ### Security model -- **Transport / TLS contract**: WSS is mandatory for external clients. **MVP default -- and the only supported mode while the runtime and PTY children share a UID: termination (b), a trusted Ingress terminates TLS** and forwards plain WS internally; the internal listener accepts non-loopback plain WS only when the deployment declares `tls_terminated_upstream = true`, and the residual internal-hop exposure is documented. **Termination (a), in-process TLS with a mounted certificate key, is gated on a real runtime/child privilege or mount-namespace boundary** (the same gating as HMAC bridge secrets and signed tokens): a `tls.key` readable by a same-UID child is persistent, stealable authority. **Risk grading, stated so the two secrets are not conflated**: a stolen TLS private key is *transport* authority (endpoint impersonation/MITM for that deployment); the admin credential hash is a *non-reversible verifier* (grants nothing when read) -- which is exactly why the key must stay outside the same-UID container while the hash may live inside it. Fail-closed in all cases: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) +> **Granularity note — target contracts**: the MUST inventories in this section and in Session lifecycle are **target contracts**: they bind the Phase 1 acceptance tests, and implementation evidence may revise them via PRs against this ADR (each revision recorded, never silent). Phase 1 includes extracting them into **`docs/pty-security-contract.md`** as given/when/assert acceptance criteria — after which this ADR records *decisions and rationale* and the contract document owns the *testable clauses*. This resolves the ADR-to-spec granularity drift deliberately rather than letting the two genres blur. + +- **Transport / TLS contract** -- WSS is mandatory for external clients: + - **MVP default -- termination (b), trusted Ingress**: a trusted Ingress terminates TLS and forwards plain WS internally. This is **the only supported mode while the runtime and PTY children share a UID**; the internal listener accepts non-loopback plain WS only when the deployment declares `tls_terminated_upstream = true`, and the residual internal-hop exposure is documented + - **Termination (a), in-process TLS -- gated on a real boundary**: a mounted certificate key requires a runtime/child privilege or mount-namespace boundary first (the same gating as HMAC bridge secrets and signed tokens): a `tls.key` readable by a same-UID child is persistent, stealable authority + - **Risk grading, stated so the two secrets are not conflated**: a stolen TLS private key is *transport* authority (endpoint impersonation/MITM for that deployment); the admin credential hash is a *non-reversible verifier* (grants nothing when read) -- which is exactly why the key must stay outside the same-UID container while the hash may live inside it + - **Fail-closed in all cases**: the listener refuses to bind off-loopback without auth material configured (same guard the `/acp` endpoint enforces) - **Browser credential transport**: reuse the validated `/acp` scheme -- `Authorization: Bearer` for non-browser clients, `Sec-WebSocket-Protocol: openab.bearer.` for browsers (browsers cannot set the Authorization header on upgrade); constant-time comparison carries over. **Origin policy, decided explicitly (not "carried over")**: the as-built `/acp` consults `Origin` only on its keyless loopback path; its keyed bearer path never checks Origin -- and PTY has no keyless mode, so there is nothing to carry over. PTY's trust boundary is **bearer-only**: possession of a valid attach token is the sole authorization, and the `Origin` header is not consulted (it is attacker-controlled outside browsers and adds no strength to a keyed WebSocket). Browser-side token hygiene is governed by the client storage contract below - **Token control plane** (MVP model; an identity layer remains explicitly out of scope per `identity-trust-none.md`): - **Create requires authentication -- and locality is not authentication**: the shell child may share the container (and potentially the UID) with the runtime, so a loopback/UDS endpoint alone is not an auth barrier. MVP mechanism: session creation (`openab-pty session create `) requires an **admin bootstrap credential** delivered to the operator at deploy time and never present in the PTY child's environment or filesystem view. Hardening alternative (non-MVP): a distinct privileged UID/group for the runtime with UDS peer-credential verification. Unauthenticated remote create/list/kill is NOT provided in MVP; the admin credential and the attach token are distinct planes -- an attach token can never create, list, or kill - **Admin-credential delivery channels (MUST)** -- the containment model rests on the plaintext credential never being observable by a same-UID shell child, so every channel is enumerated and closed: - The plaintext exists **only on the operator's trusted side** (their terminal/secret manager) - - In-container presentation accepts it **only via short-lived non-echoing stdin or a UDS message body**; it is **never** accepted as an argv flag (`/proc//cmdline` is world-readable), never placed in any process environment inside the container, never written to temporary files + - In-container presentation accepts it **only via** (a) **a UDS message body**, or (b) **an interactive non-echoing terminal prompt whose credential bytes arrive only after the prompting process has set dumpable=0**. **Piped or pre-filled stdin is rejected (`isatty` check) -- stated as the F2 resolution**: "dumpable before FD possession" cannot be satisfied for a credential-bearing pipe inherited at exec, because the FD and its buffered bytes exist before the new program can call `prctl`; a launcher mechanism that sets dumpable pre-exec could re-admit that channel and is non-MVP hardening. The credential is **never** accepted as an argv flag (`/proc//cmdline` is world-readable), never placed in any process environment inside the container, never written to temporary files - Audit logs and errors never contain it -- log a fingerprint of the verifier hash, not the credential - **The containment discipline applies to every in-container process that touches the plaintext, not only the runtime**: any CLI/helper on the presentation path MUST set `PR_SET_DUMPABLE=0` **before holding any file descriptor that may carry the credential -- not merely before the first userspace read** (with dumpable=1, a same-UID process can drain an inherited pipe via `/proc//fd/*` even if the helper never copies the bytes itself), use bounded buffers, and zeroize before exit -- otherwise the same-UID ptrace/`/proc` window merely moves from the runtime to the helper. Preferred implementation: the CLI never materializes the credential at all and splices its stdin directly into the runtime's UDS, so exactly one process (the runtime) ever holds the plaintext in userspace -- **a splice-only CLI still counts as credential-handling** (it holds the carrying FDs) and remains subject to this MUST and the Phase 1 adversary tests - A same-UID shell observing one create/renew invocation must learn nothing reusable -- and Phase 1 carries a **same-UID adversary test** (ptrace//proc probes against every credential-handling process yield `EPERM`) plus a **dumpable regression guard** (assert dumpability stays 0 after startup; a dependency calling `prctl(PR_SET_DUMPABLE, 1)` must fail the test, because it silently collapses this model) - **Issuance at creation**: creating a session mints an immutable `generation` and a fresh attach token, **minted and returned exactly once at creation -- and valid for multiple reattaches** until its expiry or a generation bump (not single-use); reconnecting clients are not locked out, and theft exposure is bounded by a short default TTL (well below the session TTL) - **Renewal (`openab-pty session renew `)**: admin-authenticated like create; the session **process survives** (scrollback and state intact), the generation is bumped (all outstanding tokens for the session become invalid immediately), and a fresh attach token is returned exactly once. **Renew-while-attached, defined**: an actively attached connection is terminated via **connection-evict** (see the named sequences in Session lifecycle -- never session-teardown) with a **renew-distinct close code**, so an evicted client can tell renewal from takeover. **Renew is admin-initiated and disruptive by design**: it may cut an active session's connection -- including the admin's own if they renew while attached -- which is the correct behavior for its primary use cases (expired or suspected-stolen tokens). Renew is distinct from **restart-in-place** (which replaces the process). MVP tokens are otherwise valid until expiry or kill; there is no client-side refresh on the attach surface - - **Admin plane is unreachable from managed sessions (MUST)**: the runtime rejects admin operations (create/renew/kill/restart) originating from any process inside a managed session's tree -- verified via the kill-domain lineage tracking (subreaper/pidfd ancestry) or UDS peer-credential plus session-pgid check. Rationale: running `session create/renew` *inside* a managed PTY would print the one-time token into the PTY byte stream, landing it in the scrollback ring and every attach/replay client -- a server-side exfiltration channel no client-storage rule can close. Tokens are returned only to the external control client and are never written to any PTY master. Phase 1 test: `session create` invoked from a managed child is denied and audited + - **Admin plane is unreachable from managed sessions (MUST)**: the runtime rejects admin operations (create/renew/kill/restart) originating from any process inside a managed session -- verified via **session-cgroup membership of the UDS peer** (peer pid resolved via pidfd to defeat recycling; see the Kill domain), which is authoritative where ancestry and pgid checks are not (`setsid` escapes the pgid, and adopted double-fork orphans have no recoverable lineage). Rationale: running `session create/renew` *inside* a managed PTY would print the one-time token into the PTY byte stream, landing it in the scrollback ring and every attach/replay client -- a server-side exfiltration channel no client-storage rule can close. Tokens are returned only to the external control client and are never written to any PTY master. Phase 1 test: `session create` invoked from a managed child is denied and audited - **Attach only verifies, never issues**: `GET /pty/{session}` validates the presented token; there is no minting path on the attach surface - **Per-session revocation**: kill/recreate bumps the generation and deletes the stored token hash, immediately invalidating outstanding tokens for that session; runtime restart clears all token state (sessions die with the process anyway, so this is not a loss) - **Token format (MVP): no signing key exists.** Each attach token is a CSPRNG 256-bit opaque bearer value; the runtime stores only its hash together with `(session ID, generation, scope = attach-only, expiry)` in memory and deletes it on kill/expiry. Because sessions deliberately do not survive a runtime restart, self-contained signed tokens buy nothing in MVP -- and eliminating the signing key eliminates the minting authority a same-container shell could steal. Signed (HMAC) tokens are a later option and require either an external signer outside the PTY container or the runtime/child privilege boundary below @@ -219,20 +227,29 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized - **Liveness**: activity = client input OR PTY output OR a live attached socket (WS ping/pong at a 15-30s interval; a half-open socket counts as detached after 2-3 missed pings -- exact values are Phase 1 config with these recommended defaults, balancing flaky mobile networks against dead-client slot pinning) - **TTLs**: detached-idle TTL (default 30m) plus an absolute session lifetime cap (default 12h) that applies even while attached -- capacity cannot be pinned forever by an open browser tab. Expiry is client-visible: a warning control frame precedes forced teardown, and the WebSocket closes with a distinct close code so clients surface "session expired" instead of retrying a network error -- **Attach semantics (MVP)**: single-attach exclusive, enforced by a session-level `owner_conn_generation` compare-and-swap: only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes, the PTY writer task honors only the current generation, and teardown of a replaced connection can never affect its successor. A second attach with a valid token takes over via this CAS (documented; multi-viewer is Phase 3). **Takeover abuse controls**: every successful preempt is audited as an anomaly event (session, source address, count), and preempt frequency is rate-limited per session (e.g. max N takeovers/min, then attaches are rejected with a distinct close code) -- a stolen still-valid token must not be able to silently ping-pong the CAS and starve the legitimate client. **The limiter is scoped to the session generation**: a generation bump (renew/recreate) resets the bucket, and the first attach under a new generation always bypasses an exhausted bucket -- so a thief who exhausted the budget can never lock the victim out of the `session renew` recovery path. **Lock ordering (total)**: token revocation (generation bump + stored-hash deletion), the attach CAS, and replay registration execute in that order under the session lock; where the buffer lock is also needed (replay registration), **the session lock is always acquired before the buffer lock and never the reverse** -- no interleaving where a revoked token wins an attach, a replay registers against a stale generation, or two paths deadlock across the two locks. **The session lock covers state only, never I/O**: it protects the state machine (generation, token hashes, `owner_conn_generation`, subscriber registration metadata); all socket and PTY I/O -- including notifying/closing a preempted connection and draining replay bytes -- executes after the lock is dropped, fenced by generation so stale work is ignored. Kill and TTL paths are never blocked by a per-connection drain (bounded wait or lock-free signal), so a slow or malicious non-reading client cannot delay renew, expiry, or teardown +- **Attach semantics (MVP)** -- single-attach exclusive; multi-viewer is Phase 3: + - **Exclusivity mechanism**: a session-level `owner_conn_generation` compare-and-swap -- only the connection that wins the CAS holds the PTY write end; the replaced connection's write path is dropped before its socket closes; the PTY writer task honors only the current generation; teardown of a replaced connection can never affect its successor + - **Takeover**: a second attach with a valid token takes over via this CAS (documented behavior) + - **Takeover abuse controls**: every successful preempt is audited as an anomaly event (session, source address, count), and preempt frequency is rate-limited per session (e.g. max N takeovers/min, then attaches are rejected with a distinct close code) -- a stolen still-valid token must not be able to silently ping-pong the CAS and starve the legitimate client + - **Limiter scope (generation-fenced)**: a generation bump (renew/recreate) resets the bucket, and the first attach under a new generation always bypasses an exhausted bucket -- so a thief who exhausted the budget can never lock the victim out of the `session renew` recovery path + - **Lock ordering (total)**: token revocation (generation bump + stored-hash deletion), the attach CAS, and replay registration execute in that order under the session lock; where the buffer lock is also needed (replay registration), the session lock is always acquired before the buffer lock and never the reverse -- no interleaving where a revoked token wins an attach, a replay registers against a stale generation, or two paths deadlock across the two locks + - **Locks cover state, never I/O**: the session lock protects the state machine (generation, token hashes, `owner_conn_generation`, subscriber registration metadata); all socket and PTY I/O -- including notifying/closing a preempted connection and draining replay bytes -- executes after the lock is dropped, fenced by generation so stale work is ignored + - **Teardown is never client-blockable**: kill and TTL paths are never blocked by a per-connection drain (bounded wait or lock-free signal), so a slow or malicious non-reading client cannot delay renew, expiry, or teardown - **Reconnect**: monotonic byte cursor from day one -- the ring buffer tracks total bytes written; clients reconnect with `since=` and receive only missed bytes. The replay-to-live handoff is **atomic**: the subscriber registers under the buffer lock, captures the end offset, replays through it, then drains queued live bytes -- with connection-generation fencing so teardown of a replaced connection cannot affect its successor. On overflow the server sends an explicit `gap` control frame (bytes-dropped count) so the client can trigger a full clear/redraw instead of rendering a sliced ANSI stream - **Output-path bounds (MUST)**: every buffer on the PTY-to-client path is bounded, not just retained scrollback -- the replay-to-live handoff queue and the per-connection outbound backlog each have a fixed cap. A client too slow to drain its backlog gets a `gap` frame (drop-oldest, cursor advances) or, past a hard watermark, is disconnected with a distinct close code -- mirroring the input-side fail-closed backpressure so neither direction can grow unbounded memory +- **Memory admission formula (MUST)**: the per-session bounds compose into an explicit container budget -- `max_sessions × (scrollback_kib + replay/backlog queue caps + fixed per-session overhead) + runtime baseline ≤ the container memory request`. This matters more here than in a stateful service: the keyless in-memory model makes an OOM a **total-loss event** (every session and token invalidated at once, per Consequences), so admission control is load-bearing, not tuning. The chart derives a recommended memory request from the configured `[pty]` values and refuses obviously-oversubscribed combinations; Phase 2 ships the observability for it (per-session buffer occupancy, global tracked-process and FD counts, admission-rejection metrics) plus capacity guidance - **`scrollback_replay` vs cursor semantics** (distinct controls): incremental `since` replay is always available within the ring buffer's retention; `scrollback_replay` governs only the cursor-less full-history dump on a fresh attach (default off -- secrets-safe); setting `scrollback_kib = 0` disables retention entirely, which also disables `since` replay (every reconnect starts with a `gap` + reset) - **Two named termination sequences** (they are different operations and must never be conflated): - **Connection-evict** -- ends a *connection*, the session process survives: notify the client, close the socket with the operation-specific close code (takeover, renew, TTL warning). Used by attach takeover and renew-while-attached - **Session-teardown** -- ends the *session*: setpgid on spawn; SIGTERM-grace-SIGKILL escalation; evict-while-attached order = notify client, close socket, kill (per the Kill domain below), close master fd, release slot; buffers cleared on teardown; scrollback never touches disk -- **Kill domain (MUST)** -- the process group is only the first signal path, not the containment guarantee (a child that calls `setsid` or double-forks escapes the pgid): - - **MVP default (works under the stated container contract, no extra capabilities): a pidfd-based descendant reaper.** The runtime sets `PR_SET_CHILD_SUBREAPER` so escaped descendants reparent to it, and holds a pidfd per tracked process - - **Reaper scope**: discovery and kill are bounded to the runtime's **own spawned tree** (processes it created directly or via PTY children) -- never a blanket `/proc` sweep. Being a subreaper reparents *all* container orphans to the runtime; reparented processes outside a session's tree are reaped (waited on) but never killed - - **Convergence invariant**: teardown is a **kill-and-rescan loop until no session descendants remain** -- a one-shot scan-then-kill is not race-free (a tracked process can fork between the final scan and the kills). Subreaper reparent-and-reap is the convergence guarantee; the session slot is released only after the invariant holds +- **Kill domain (MUST)** -- the process group is only the first signal path, not the containment guarantee (a child that calls `setsid` or double-forks escapes the pgid). Two distinct problems are named separately and solved separately: **attribution** (which session does a live process belong to?) and **reaping** (collecting exited children). Ancestry-based tracking cannot solve attribution: `PR_SET_CHILD_SUBREAPER` guarantees reaping, not lineage -- a double-fork whose intermediate exits before the first discovery scan yields a live adopted process whose session membership is unrecoverable, and no reparent notification exists: + - **Phase 1 prerequisite -- authoritative per-session membership: one delegated cgroup per session, torn down via `cgroup.kill`** (or freeze-then-kill). Each PTY child is spawned into its session's cgroup; membership is read from the kernel (`cgroup.procs`), never inferred from ancestry. This requires cgroup v2 with subtree delegation to the container's UID, which the default non-root/no-capabilities contract does not provide by itself -- the chart MUST document the delegation arrangement, and **the pre-implementation feasibility gate (Section 5) can fail on this prerequisite**: if delegation proves impractical across target environments, this ADR returns to review with the spike evidence and an alternative attribution boundary is designed explicitly -- the mechanism never silently degrades to ancestry tracking + - **Subreaper + pidfd as the reaping layer, never the attribution layer**: the runtime still sets `PR_SET_CHILD_SUBREAPER` and holds pidfds for spawn-tracked processes -- to collect zombies, detect self-exit, and signal as defense-in-depth -- but session teardown convergence is defined over cgroup membership, never over the reaper's view of ancestry + - **Unattributed-orphan policy (stated)**: a process that reparents to the runtime and belongs to no session cgroup cannot exist under the cgroup-membership design (a fork inside a session cgroup stays in it); observing one therefore indicates a mechanism fault, and the policy is fail-safe -- it is killed (not merely reaped), audited as an anomaly with a `/proc` identity snapshot, and counted on a global orphan metric. It is never charged to a specific session's teardown, and it never blocks slot release (the session-cgroup-empty condition alone governs release) + - **Convergence invariant**: teardown = `cgroup.kill`, then wait until the session cgroup is empty. Forking races are closed by the kernel (a fork racing `cgroup.kill` is itself killed); no userspace kill-and-rescan loop is load-bearing + - **Admin-plane-unreachable rides on the same primitive**: admin operations are rejected when the UDS peer (resolved via pidfd to defeat pid recycling) is a member of any session cgroup -- ancestry and pgid checks are not load-bearing for that MUST either - **Resource budget**: tracked processes and their pidfds are capped per session and globally, with reserved FD headroom for control/WebSocket sockets; hitting tracking capacity is fail-closed (the session is killed, never left partially tracked) - - **The cgroup path (`cgroup.kill`, or freeze-then-kill) is the stronger boundary, gated on an explicit prerequisite**: cgroup v2 with subtree delegation to the container's UID, which the default non-root/no-capabilities contract does not provide -- deployments that arrange delegation SHOULD prefer it - - **Startup probe, fail closed**: at startup the runtime verifies its configured kill mechanism is operational (subreaper flag set and pidfd support, or a writable delegated cgroup subtree) and refuses to serve sessions otherwise. Slot release and the absolute TTL are enforced against this hard boundary, never against the pgid alone + - **Startup probe, fail closed**: at startup the runtime verifies its kill mechanism end-to-end -- create, populate-check, `cgroup.kill`, and remove a probe cgroup in the delegated subtree, plus subreaper flag and pidfd support -- and refuses to serve sessions otherwise. **Environmental conditional, stated**: even though the pidfd path needs no extra *capabilities*, seccomp profiles commonly filter `prctl`/`pidfd_open`, and delegation may be absent -- the startup probe is the intended detection for all such environments, and its failure diagnostics name the missing prerequisite (blocked syscall or non-writable subtree). Slot release and the absolute TTL are enforced against the cgroup boundary, never against the pgid alone - **Self-exit, defined (Phase 1 behavior, not deferred with the full state machine)**: when the child exits on its own, the runtime reaps it (kill-domain convergence still runs for surviving descendants), sends any attached client a final output flush plus a **session-ended close code** (distinct from TTL expiry and eviction), releases the slot after convergence, and deletes the session's token state -- the name then behaves exactly like reattach-to-dead: a distinct error offering restart-in-place. Termination classes (user-kill / self-exit / runtime-shutdown) are tagged in audit from Phase 1; the richer state machine remains Phase 3 - **Recovery taxonomy** (stated, not implied): detach/reattach survives (process alive); pod restart does not (process dead) -- reattach-to-dead returns a distinct error and offers **restart-in-place**: same session name, a fresh process and a new generation (old tokens invalid, empty scrollback). Pod-lifetime durability is out of scope and documented as such @@ -245,12 +262,14 @@ admin_credential_hash = "sha256:9f2c..." # literal verifier hash, materialized - OAB keeps its thin-broker identity untouched — zero changes to the shipped binary, pool, or ACP path - Fills the remote + sandboxed + raw-terminal quadrant with a real container boundary instead of a claimed one - Highest reversibility: default-off, separately versioned, separately deprecable -- Coexistence where it matters (shared workspace) without shared process or credential-mount domains; network namespace and pod fate are shared only in the colocated profile (see Isolation tiers) -- The Phase 4 notification bridge (broker pulls from `openab-pty` -> relays to Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes +- Coexistence where it matters (shared workspace) without shared process or credential-mount domains; in MVP coexistence is two separate pods (RWX workspace where the storage class supports it), so the security story is single-tier: full isolation only +- The Phase 4 notification bridge (broker pulls from `openab-pty` -> relays to Discord) later reconnects the feature to OAB's messaging strength without merging the runtimes -- demand-gated together with the colocated profile ### Negative - A second binary and image to build, test, and release (mitigated by the existing multi-binary workspace and release pipeline) +- **MVP defers the headline same-pod ergonomics**: shared working tree without RWX storage requires the colocated profile, which is demand-gated -- accepted deliberately to keep the MVP security story single-tier (full isolation) and the review/attack surface small; users who need same-workspace coexistence on RWO-only storage wait for the profile-3 gate +- **The kill-domain prerequisite adds deployment friction**: per-session delegated cgroups require cgroup v2 subtree delegation, which the default non-root container contract does not provide by itself -- the chart must document the arrangement, some environments may not support it (the runtime then refuses to serve, fail-closed), and the feasibility spike can fail the gate on it - **The keyless in-memory model's cost, consolidated**: every session and token is bound to one runtime process -- no HA, no multi-replica serving, no failover; a crash, OOM, restart, projection rollout, or admin-credential rotation (which requires a restart) invalidates **all** sessions and tokens simultaneously. This is the deliberate exchange for eliminating at-rest minting authority, and it is why the rotation runbook (Phase 2) must state the blast radius up front - Cross-container coordination (notification bridge, future shared-crate extraction) is more ceremony than in-process calls - Some duplication with the ACP pool (capacity accounting, pgid kill) until a shared lifecycle crate is justified by real usage @@ -288,13 +307,18 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ## 5. Implementation Plan -**Pre-implementation gate (go/no-go)**: Phase 1 starts only after a demand check with measurable criteria set by the maintainers -- at minimum, linked user requests beyond the originating discussion thread and a maintainer-agreed operating-cost budget. Acceptance of this ADR records the *design*, not a commitment to build on a schedule; the 12-month adoption review below is the post-ship counterpart of this gate. +**Pre-implementation gate (go/no-go, executable)**: Phase 1 starts only after BOTH components pass: + +- **Demand component (numbers stated)**: at least **3 independent user requests** (distinct users; issues, discussions, or support threads — the originating discussion thread and its participants count as one) within **90 days** of this ADR merging, plus a maintainer-agreed operating-cost budget (image build/release + support surface) recorded on the tracking issue. These defaults may be revised by a maintainer on the tracking issue *before* the gate is evaluated, never retroactively +- **Feasibility component (can fail the gate)**: a **time-boxed spike (≤1 week)** in a non-root, capabilities-dropped container proving (a) per-session delegated-cgroup creation and `cgroup.kill` teardown under the kill-domain adversary cases (`setsid` escapee, double-fork with early-exiting intermediate, SIGTERM-trapping child — zero unattributed survivors), and (b) the dumpable-before-FD adversary tests against every credential-handling process. A failed spike returns this ADR to review with the evidence attached; the design does not proceed with a degraded mechanism + +Acceptance of this ADR records the *design*, not a commitment to build on a schedule; the 12-month adoption review below is the post-ship counterpart of this gate. ### Phase 1: `openab-pty` MVP (new crate, new binary) - Own session manager: named sessions, operator-configured command, allowlist-validated names - **Session bootstrap**: sessions are created via the authenticated loopback/UDS operator CLI (`openab-pty session create `; `session renew ` re-issues a token per the token control plane; `session restart ` performs restart-in-place for reattach-to-dead per the recovery taxonomy), which spawns the PTY and returns the one-time attach token; `GET /pty/{session}` is attach-only. No remote create/list/kill in Phase 1 (Phase 2 adds them behind admin auth) -- portable-pty spawner with setpgid, escalating kill, the teardown order above, **and the hard kill boundary per the Kill domain MUST** (pidfd descendant reaper with `PR_SET_CHILD_SUBREAPER` as the default; delegated-cgroup kill where available; fail-closed startup probe) +- portable-pty spawner with setpgid, escalating kill, the teardown order above, **and the hard kill boundary per the Kill domain MUST** (per-session delegated cgroup with `cgroup.kill` as the authoritative attribution and teardown boundary -- the stated Phase 1 prerequisite; subreaper + pidfd for reaping and self-exit detection; fail-closed startup probe) - `GET /pty/{session}` WSS endpoint: binary frames = PTY bytes; text frames = versioned control schema (`resize`, `ping`, `detach`, `gap`, `ttl-warning`) with a defined close-code table. Frame validation is strict allowlist: bounded max frame size, unknown control types rejected, resize values bounds-checked; malformed frames count toward an abuse metric and can disconnect - Input backpressure: per-connection write watermark toward the PTY master; a client exceeding it is disconnected (fail closed) rather than growing unbounded queues or stalling the reader - Auth: the token control plane above (authenticated create, one-time issuance bound to session generation, attach-only verification); fail-closed off-loopback; `/acp`-style browser subprotocol transport; per-IP upgrade-failure rate limiting @@ -302,13 +326,14 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Detached-idle TTL + absolute lifetime cap (with client-visible expiry warning + close code); single-attach exclusive - Audit log (attach/detach/create/kill/auth-failure) and basic metrics - `openab-pty --validate-projection `: the fail-closed startup guard exposed as a standalone subcommand, so operators can verify a hand-generated projection from day one (Phase 2 CI reuses it as the guard test) +- **Extract the security/lifecycle MUST inventories into `docs/pty-security-contract.md`** (given/when/assert form, one clause per MUST) so the adversary tests below are CI-linkable to named contract clauses; the ADR's MUST sections then become pointers into that contract (see the Granularity note in the Security model) - **Same-UID adversary tests**: ptrace/`/proc//{mem,fd}` probes against every credential-handling process return `EPERM` -- **explicitly including the splice-only CLI's file descriptors, not just the runtime**; dumpability asserted to remain 0 after startup (regression guard against a dependency re-enabling it); `prctl(PR_SET_DUMPABLE, 0)` failure refuses service (fail-closed); `session create` invoked from inside a managed child is denied and audited; kill-domain convergence test with a `setsid`/double-fork escapee and a SIGTERM-trapping stubborn child (zero orphans, zero FD leaks) - Resize propagation (TIOCSWINSZ) including attach-time initial size - Terminal-capability response filtering at the PTY boundary (known Ink-CLI startup breakage) ### Phase 2: Deployment + web client -- Helm: independent `openab.enabled` / `pty.enabled` toggles (or `--set profile=acp|pty|full`); standalone profile gets its own Service/Ingress (`/pty/*`) and NetworkPolicy example; config split documented per the configUrl pattern; `ghcr.io/openabdev/openab-pty` image published from the existing release pipeline. **The chart never defaults to `full`**: the colocated profile is opt-in only and its values file is labeled convenience-only, pointing at the Isolation tiers table -- the default demo/production path is separate pods +- Helm: independent `openab.enabled` / `pty.enabled` toggles rendering **separate pods** (`--set profile=acp|pty`); standalone profile gets its own Service/Ingress (`/pty/*`) and NetworkPolicy example; config split documented per the configUrl pattern; `ghcr.io/openabdev/openab-pty` image published from the existing release pipeline. **The MVP chart does not render the colocated `full` profile at all** -- profile 3 ships only if its demand gate opens (see Later), and when it does it is opt-in only, never the default, with its values file labeled convenience-only and pointing at the Isolation tiers table. RWX-based cross-pod workspace sharing is documented as the MVP coexistence path - **Web client is attach-only (browser management deferred)**: the minimal xterm.js page served by `openab-pty` accepts an attach token and connects -- nothing else. Remote list/create/kill/renew endpoints exist for *non-browser* admin tooling only, gated by the admin bootstrap credential; **the web client never receives, stores, or transmits the admin credential** -- delivering the global management credential to a browser would turn one XSS into administration of every session. A browser management UI requires an operator-mediated pairing / one-time scoped-issuance flow or the identity layer, and is explicitly deferred until one exists - **Client-page acceptance criteria (testable)**: attach token held in memory only (never localStorage/sessionStorage/cookies/URLs); page served with CSP enforcing at minimum `script-src 'self'` (no inline/eval), `object-src 'none'`, `base-uri 'none'`, `frame-ancestors 'none'`, and `connect-src` limited to the PTY origin; no third-party runtime scripts - **Admin-credential rotation runbook (acceptance criterion)**: documented steps, blast radius, and expected downtime for rotation (generate new value -> update delivered hash -> restart runtime -> all sessions cleared) -- operators must not discover mid-incident that rotation kills every session @@ -320,7 +345,9 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho - Multi-viewer (one writer, N readers) with writer-lease semantics and read-only token scope - Reconnect backoff, richer capacity controls (per-token limits) -### Phase 4: Messaging bridge (optional, colocated profile only) +### Phase 4: Messaging bridge (demand-gated with profile 3; colocated profile only) + +> This phase exists only if the profile-3 demand gate opens (see Later): the bridge is colocated-only by design, so it inherits profile 3's gate. Recorded here as the target design. - `openab-pty` exposes a pod-local, loopback-only notification stream; the **broker pulls** (long-poll/SSE on localhost) when a detached session emits no output for N seconds after a prompt-like burst (stated heuristic, not magic); the broker relays to the platform thread. Bridge is one-way and feature-gated - **No bridge secret enters the PTY container -- by design, stated now**: a delivered HMAC key would recreate exactly the in-container authority the keyless token model eliminated (a same-UID child can read any file the runtime can read; 0400 at the same UID is not a boundary). The pull model removes the broker-side ingress entirely: there is no webhook endpoint to leave open and no shared key to steal. Residual risk, stated: a same-UID child that kills the runtime (an accepted same-UID risk) could bind the freed port and forge events -- therefore the broker treats bridge events as **display-only, rate-limited hints**: they never carry commands, never mutate broker state, and are labeled best-effort in the relayed message. A push/webhook variant with an HMAC secret is permitted only with an external signer outside the PTY container or the runtime/child privilege boundary from the Security model (non-MVP hardening) @@ -329,6 +356,7 @@ Leaves the need unserved; users accept Herdr's laptop fragility or OpenDray's ho ### Later (demand-gated, explicitly deferred) +- **Colocated profile 3 + the Phase 4 bridge**: gated on the standalone profile proving demand *and* an explicit maintainer decision to re-open; the sidecar design, partial-isolation tier, and pull-bridge contract recorded in this ADR are the target design for that gate. Until then the chart renders separate pods only - Shared lifecycle crate extraction (if the ACP pool and PTY manager converge naturally). Candidate shared surface: spawn mechanics, env-allowlist construction, pgid kill/escalation; deliberately NOT shared: liveness definitions, TTL/eviction policy, persistence - **Adoption review point**: 12 months after the standalone profile ships, review its usage; below a threshold the maintainers set then, consider deprecating the standalone image or folding PTY back to colocate-only - Single-process merge (only if operations prove the runtime split is more cost than benefit)