feat: support compose secrets via podman secret store

architecture/supported-subset.md

@@ -9,7 +9,7 @@ warns (ignored, behavior-neutral inside a single pod) or raises
 ## Top-level keys
 
 - **Supported:** `services` (required, non-empty), `version`, `name`,
-  `networks`.
+  `networks`, `secrets`.
 - **Ignored (warns):** `networks` — all services share the pod's single
   network namespace, so top-level network definitions have no effect.
 - **Extension fields:** any key prefixed `x-` is accepted and ignored
@@ -21,9 +21,10 @@ warns (ignored, behavior-neutral inside a single pod) or raises
 
 - **Supported:** `image`, `build`, `command`, `entrypoint`, `environment`,
   `env_file`, `volumes`, `healthcheck`, `depends_on`, `networks`, `hostname`,
-  `container_name`, `tmpfs`, `user`, `working_dir`, `group_add`, `labels`,
-  `read_only`, `init`, `privileged`, `cap_add`, `cap_drop`, `security_opt`,
-  `platform`, `devices`, `annotations`, `extra_hosts`, `pull_policy`, `ulimits`.
+  `container_name`, `tmpfs`, `secrets`, `user`, `working_dir`, `group_add`,
+  `labels`, `read_only`, `init`, `privileged`, `cap_add`, `cap_drop`,
+  `security_opt`, `platform`, `devices`, `annotations`, `extra_hosts`,
+  `pull_policy`, `ulimits`.
   compose2pod never builds: a `build` section is accepted but its contents
   (context, dockerfile, args) are not read — `image_for` (`compose2pod/emit.py`)
   runs the CI image supplied via `--image` for any service that has one.
@@ -162,6 +163,58 @@ common way to shadow a subdirectory of a bind mount). No host-path translation
 is applied, since the entry names a container path, not a host source. A
 colon-less entry that is not absolute (e.g. `./cache`) is malformed and raises.
 
+## Secrets
+
+- **Top-level `secrets:` definitions:** each entry must be a mapping with
+  exactly one of `file:` (a host path, resolved against `--project-dir` when
+  relative) or `environment:` (a host environment variable name), both as a
+  plain string. `external: true` gets its own rejection message (compose's
+  "must already exist" secrets have no analogue here); any other unrecognized
+  key raises generically (`_validate_secret_def`, `compose2pod/secrets.py`).
+- **Per-service `secrets:` references:** short form (`- name`) or long form
+  (a mapping with `source` and optionally `target`, `uid`, `gid`, `mode`).
+  `source` must name a top-level secret; an unknown `source` raises at
+  `validate()` time (`_ref_source`/`validate_secrets`,
+  `compose2pod/secrets.py`).
+- **Closure-scoped creation:** only secrets referenced (by `source`) from
+  somewhere in the target service's dependency closure are ever created, so a
+  top-level secret nothing in the closure references never becomes a
+  `podman secret create` call (`referenced_secret_names`, driven by the same
+  `startup_order` closure used to decide which services run at all).
+- **Creation:** each referenced secret becomes one pod-namespaced
+  `podman secret create <pod>-<name> ...` line, emitted right after
+  `podman pod create` and before any `podman run` (`emit_script`,
+  `compose2pod/emit.py`). A `file:` source resolves
+  `Path(project_dir, file)` through `to_shell()`, so a `${VAR}` in the path
+  expands live when the script runs, exactly like other interpolated fields.
+  An `environment:` source instead pipes `printf '%s' "${VAR-}"` into
+  `podman secret create ... -`, where the `${VAR-}` means an *unset* host
+  variable yields an empty secret rather than failing the script
+  (`secret_create_lines`).
+- **Mounting:** each service reference becomes a
+  `--secret source=<pod>-<name>,target=<target>` flag on that service's
+  `podman run`, where `target` defaults to the secret's own name when the
+  reference doesn't give one (short form, or long form without `target`).
+  `uid`/`gid`/`mode` are only added when the long form gives them explicitly;
+  `mode` renders as a 4-digit octal string when given as a Python int
+  (`0o400` becomes `"0400"`) and passes through verbatim when given as a
+  string (`secret_flags`). When `uid`/`gid`/`mode` are omitted, podman itself
+  applies its own defaults: the secret is mounted at `/run/secrets/<target>`,
+  owned `0:0`, mode `0444`.
+- **Teardown:** the EXIT trap that force-removes the pod also runs
+  `podman secret rm <pod>-<name> ...` for every referenced secret, so the
+  store never outlives the pod even when the script exits abnormally
+  (`emit_script`).
+- **Variable interpolation:** an `environment:` source's variable name, and
+  any `${VAR}` inside a `file:` path, both count toward the CLI's
+  informational stderr note of variables the generated script expands at run
+  time (`secret_referenced_variables`, folded into `referenced_variables()`
+  in `compose2pod/emit.py`); see Variable interpolation, below, for the note
+  itself.
+- Everything else raises `UnsupportedComposeError` rather than silently doing
+  nothing: `external: true`, an unknown `source`, a definition with neither
+  or both of `file`/`environment`, and an unrecognized long-form key.
+
 ## Variable interpolation
 
 compose2pod does not resolve Compose Spec `${VAR}` references at generation

compose2pod/emit.py

@@ -8,6 +8,7 @@
 from compose2pod.graph import depends_on, hostnames, startup_order
 from compose2pod.healthcheck import health_cmd, interval_seconds
 from compose2pod.keys import SERVICE_KEYS, Token, _Expand, _key_value_pairs
+from compose2pod.secrets import referenced_secret_names, secret_create_lines, secret_flags, secret_referenced_variables
 from compose2pod.shell import to_shell, variable_names
 
 
@@ -98,6 +99,7 @@ def run_flags(name: str, svc: dict[str, Any], pod: str, hosts: list[str], projec
     for key, spec in SERVICE_KEYS.items():
         if key in svc:
             flags += spec.emit(svc[key])
+    flags += secret_flags(svc, pod)
     return flags
 
 
@@ -184,6 +186,7 @@ def emit_script(compose: dict[str, Any], options: EmitOptions) -> str:
     services = compose["services"]
     hosts = hostnames(services)
     order = startup_order(services, options.target)
+    secret_names = referenced_secret_names(services, order)
     completion_gated = {
         dep
         for svc in services.values()
@@ -192,8 +195,13 @@ def emit_script(compose: dict[str, Any], options: EmitOptions) -> str:
     }
 
     lines = [_SCRIPT_HEADER]
-    lines.append(f"trap 'podman pod rm -f {shlex.quote(options.pod)} >/dev/null 2>&1 || true' EXIT")
+    teardown = f"podman pod rm -f {shlex.quote(options.pod)} >/dev/null 2>&1 || true"
+    if secret_names:
+        stores = " ".join(f"{options.pod}-{name}" for name in secret_names)
+        teardown += f"; podman secret rm {stores} >/dev/null 2>&1 || true"
+    lines.append(f"trap '{teardown}' EXIT")
     lines.append(f"podman pod create --name {shlex.quote(options.pod)}")
+    lines.extend(secret_create_lines(compose, options.pod, options.project_dir, secret_names))
     waited: set[str] = set()
     for name in order:
         for dep, condition in depends_on(services[name]).items():
@@ -227,4 +235,5 @@ def referenced_variables(compose: dict[str, Any], options: EmitOptions) -> list[
         for token in _run_tokens(name, services, options, hosts):
             if isinstance(token, _Expand):
                 names.update(variable_names(token.value))
+    names |= secret_referenced_variables(compose, options.project_dir, referenced_secret_names(services, order))
     return sorted(names)

compose2pod/keys.py

@@ -196,4 +196,5 @@ def _emit_ulimits(value: Any) -> list[Token]:  # noqa: ANN401 - Compose values a
     "networks",
     "hostname",
     "container_name",
+    "secrets",
 }

compose2pod/parsing.py

@@ -6,12 +6,13 @@
 from compose2pod.graph import depends_on, hostnames
 from compose2pod.healthcheck import has_healthcheck, interval_seconds
 from compose2pod.keys import SERVICE_KEYS, STRUCTURAL_KEYS
+from compose2pod.secrets import validate_secrets
 
 
 SUPPORTED_SERVICE_KEYS = set(SERVICE_KEYS) | STRUCTURAL_KEYS
 IGNORED_SERVICE_KEYS = {"ports", "restart", "stdin_open", "tty", "stop_signal", "stop_grace_period"}
 SUPPORTED_HEALTHCHECK_KEYS = {"test", "interval", "timeout", "retries", "start_period"}
-SUPPORTED_TOP_LEVEL_KEYS = {"services", "version", "name", "networks", "volumes"}
+SUPPORTED_TOP_LEVEL_KEYS = {"services", "version", "name", "networks", "volumes", "secrets"}
 DEPENDS_ON_CONDITIONS = {"service_started", "service_healthy", "service_completed_successfully"}
 
 
@@ -126,4 +127,5 @@ def validate(compose: dict[str, Any]) -> list[str]:
         warnings.extend(_validate_service(name, svc))
     hostnames(services)  # validate hostname/container_name/networks shapes at the gate
     _validate_depends_on(services)
+    validate_secrets(compose)
     return warnings

compose2pod/secrets.py

@@ -0,0 +1,141 @@
+"""Translate compose secrets into podman secret store create / mount / remove."""
+
+import re
+from pathlib import Path
+from typing import Any
+
+from compose2pod.exceptions import UnsupportedComposeError
+from compose2pod.keys import Token
+from compose2pod.shell import to_shell, variable_names
+
+
+_LONG_FORM_KEYS = {"source", "target", "uid", "gid", "mode"}
+_SECRET_NAME = re.compile(r"^[a-zA-Z0-9][a-zA-Z0-9_.-]*$")
+_ENV_NAME = re.compile(r"^[a-zA-Z_][a-zA-Z0-9_]*$")
+
+
+def _validate_secret_def(name: str, definition: Any) -> None:  # noqa: ANN401 - Compose values are untyped
+    if not _SECRET_NAME.fullmatch(name):
+        msg = f"secret name {name!r} must match [a-zA-Z0-9][a-zA-Z0-9_.-]*"
+        raise UnsupportedComposeError(msg)
+    if not isinstance(definition, dict):
+        msg = f"secret {name!r} must be a mapping"
+        raise UnsupportedComposeError(msg)
+    unknown = set(definition) - {"file", "environment"}
+    if unknown:
+        if "external" in unknown:
+            msg = f"secret {name!r}: external secrets are not supported (use a file: or environment: source)"
+            raise UnsupportedComposeError(msg)
+        msg = f"secret {name!r}: unsupported keys {sorted(unknown)}"
+        raise UnsupportedComposeError(msg)
+    sources = [key for key in ("file", "environment") if isinstance(definition.get(key), str)]
+    if len(sources) != 1:
+        msg = f"secret {name!r} must have exactly one of 'file' or 'environment' (a string)"
+        raise UnsupportedComposeError(msg)
+    if sources[0] == "environment" and not _ENV_NAME.fullmatch(definition["environment"]):
+        msg = f"secret {name!r}: environment variable name {definition['environment']!r} is not a valid identifier"
+        raise UnsupportedComposeError(msg)
+
+
+def _check_long_form_scalars(name: str, ref: dict[str, Any]) -> None:
+    for key in ("uid", "gid", "mode"):
+        if key in ref and (isinstance(ref[key], bool) or not isinstance(ref[key], int | str)):
+            msg = f"service {name!r}: secret {key!r} must be an int or string"
+            raise UnsupportedComposeError(msg)
+
+
+def _ref_source(name: str, ref: Any) -> str:  # noqa: ANN401 - Compose values are untyped
+    if isinstance(ref, str):
+        return ref
+    if not isinstance(ref, dict):
+        msg = f"service {name!r}: secret entry must be a string or mapping"
+        raise UnsupportedComposeError(msg)
+    unknown = set(ref) - _LONG_FORM_KEYS
+    if unknown:
+        msg = f"service {name!r}: unsupported secret keys {sorted(unknown)}"
+        raise UnsupportedComposeError(msg)
+    source = ref.get("source")
+    if not isinstance(source, str):
+        msg = f"service {name!r}: secret entry 'source' must be a string"
+        raise UnsupportedComposeError(msg)
+    _check_long_form_scalars(name, ref)
+    return source
+
+
+def validate_secrets(compose: dict[str, Any]) -> None:
+    """Validate top-level secret definitions and every service's references to them."""
+    defs = compose.get("secrets")
+    if defs is not None and not isinstance(defs, dict):
+        msg = "top-level 'secrets' must be a mapping"
+        raise UnsupportedComposeError(msg)
+    defs = defs or {}
+    for name, definition in defs.items():
+        _validate_secret_def(name, definition)
+    for name, svc in (compose.get("services") or {}).items():
+        refs = svc.get("secrets")
+        if refs is None:
+            continue
+        if not isinstance(refs, list):
+            msg = f"service {name!r}: secrets must be a list"
+            raise UnsupportedComposeError(msg)
+        for ref in refs:
+            source = _ref_source(name, ref)
+            if source not in defs:
+                msg = f"service {name!r}: unknown secret {source!r}"
+                raise UnsupportedComposeError(msg)
+
+
+def _mode_str(mode: Any) -> str:  # noqa: ANN401 - Compose values are untyped
+    return format(mode, "04o") if isinstance(mode, int) else str(mode)
+
+
+def referenced_secret_names(services: dict[str, Any], order: list[str]) -> list[str]:
+    """Secret names referenced by services in `order`, unique in first-seen order."""
+    seen: dict[str, None] = {}
+    for name in order:
+        for ref in services[name].get("secrets") or []:
+            seen[ref if isinstance(ref, str) else ref["source"]] = None
+    return list(seen)
+
+
+def secret_flags(svc: dict[str, Any], pod: str) -> list[Token]:
+    """Per-service `--secret source=<pod>-<name>,target=...` flag tokens."""
+    tokens: list[Token] = []
+    for ref in svc.get("secrets") or []:
+        opts = {} if isinstance(ref, str) else ref
+        source = ref if isinstance(ref, str) else ref["source"]
+        parts = [f"source={pod}-{source}", f"target={opts.get('target', source)}"]
+        parts += [f"{key}={opts[key]}" for key in ("uid", "gid") if key in opts]
+        if "mode" in opts:
+            parts.append(f"mode={_mode_str(opts['mode'])}")
+        tokens += ["--secret", ",".join(parts)]
+    return tokens
+
+
+def secret_create_lines(compose: dict[str, Any], pod: str, project_dir: str, names: list[str]) -> list[str]:
+    """`podman secret create` lines for the referenced secrets (file or environment source)."""
+    defs = compose.get("secrets") or {}
+    lines: list[str] = []
+    for name in names:
+        definition = defs[name]
+        store = f"{pod}-{name}"
+        if isinstance(definition.get("file"), str):
+            path = to_shell(str(Path(project_dir, definition["file"])))
+            lines.append(f"podman secret create {store} {path}")
+        else:
+            var = definition["environment"]
+            lines.append(f"printf '%s' \"${{{var}-}}\" | podman secret create {store} -")
+    return lines
+
+
+def secret_referenced_variables(compose: dict[str, Any], project_dir: str, names: list[str]) -> set[str]:
+    """Run-time variable names the secret create lines expand (env-source vars + file-path vars)."""
+    defs = compose.get("secrets") or {}
+    result: set[str] = set()
+    for name in names:
+        definition = defs[name]
+        if isinstance(definition.get("file"), str):
+            result |= variable_names(str(Path(project_dir, definition["file"])))
+        else:
+            result.add(definition["environment"])
+    return result

planning/changes/2026-07-10.02-secrets.md

@@ -0,0 +1,96 @@
+---
+summary: Support compose secrets (first Bucket B feature) via a new secrets.py module — top-level file:/environment: definitions become pod-namespaced `podman secret create` + `--secret` mounts with full long-form target/uid/gid/mode; external:true is rejected.
+---
+
+# Design: compose secrets
+
+## Summary
+
+Translate compose `secrets` into the single-pod script using podman's own secret
+store. A new `compose2pod/secrets.py` module owns the translation; `parsing.py`
+gates it and `emit.py` weaves the store lifecycle into the script. Top-level
+`file:`/`environment:` secret definitions become `podman secret create` calls,
+service-level references become `podman run --secret` mounts at
+`/run/secrets/<target>`, and the store is torn down in the existing EXIT trap.
+This is the first Bucket B feature (real translation, not a flag pass-through).
+
+## Motivation
+
+Secrets are a core CI need — a test service reads a DB password or an API key
+from `/run/secrets/<name>`. Podman `--secret` maps compose secrets almost 1:1
+(same default mount `/run/secrets/<name>`, mode `0444`, uid/gid `0`; `target`/
+`uid`/`gid`/`mode` are direct opts), and secrets mount as per-container files, so
+they are genuinely per-container-supportable in the pod (unlike `dns`/`sysctls`).
+
+## Design
+
+**`secrets.py`** owns: the set of secret names referenced by services in the
+target's closure; the `podman secret create` lines (file vs env source); and the
+per-service `--secret` flag tokens. It is a deep module — `emit.py` calls it for
+the create/rm lines and flags; `parsing.py` calls it (or a sibling validator)
+for the shape checks.
+
+**Generated script** (pod `test-pod`; a service references a `file:` secret
+`db_password` and an `environment: API_KEY` secret `api_key`):
+
+```sh
+trap 'podman pod rm -f test-pod ... ; podman secret rm test-pod-db_password test-pod-api_key >/dev/null 2>&1 || true' EXIT
+podman pod create --name test-pod
+podman secret create test-pod-db_password "/builds/proj/db_pw.txt"
+printf '%s' "${API_KEY-}" | podman secret create test-pod-api_key -
+...
+podman run -d ... --secret source=test-pod-db_password,target=db_password ...
+```
+
+- **Pod-namespaced store names** (`<pod>-<name>`) isolate concurrent CI runs
+  (podman secrets are per-user global); `target=` maps back to
+  `/run/secrets/<compose-target-or-name>`.
+- `create` lines emit right after `podman pod create` (secrets must exist before
+  `--secret`); the EXIT **trap** removes them alongside the pod, so a failed
+  create under `set -e` tears everything down.
+- `file:` path resolves against `--project-dir` and is `_Expand`-rendered (like
+  volumes). `environment:` uses `${VAR-}` (empty-if-unset, survives `set -u`);
+  `VAR` flows into the `referenced_variables()` stderr note. Long-form
+  `target`/`uid`/`gid`/`mode` become literal `--secret` opts; `mode: 0440`
+  (PyYAML parses as octal int `288`) renders back to `0440`, a string `"0440"`
+  passes through.
+
+**Validation** (`parsing.py`, a cross-cutting `_validate_secrets(compose)` — a
+service ref points at a top-level def, so it needs the whole document): top-level
+`secrets:` is a mapping of name -> definition with **exactly one** of `file:` /
+`environment:` (a string); `external: true`, multiple sources, unknown source
+keys, or a non-mapping def -> reject. Service `secrets:` is a list of short
+strings or long mappings (`source` required + optional `target`/`uid`/`gid`/
+`mode`); an unknown source name -> `unknown secret 'X'`. `secrets` joins
+`SUPPORTED_TOP_LEVEL_KEYS` and, as a structural key, `SUPPORTED_SERVICE_KEYS`.
+
+## Non-goals
+
+- `external: true` -- rejected (v1 boundary; compose2pod creates the store from
+  the file, and referencing a maybe-absent podman secret fails late). Revisit if
+  a real pre-created-podman-secret need appears; own decision file.
+- Creating secrets not referenced in the target's closure (matches how emit only
+  runs the closure).
+- Matching compose's "skip an empty env-source secret": an unset var yields an
+  empty secret (`${VAR-}`), the simplest faithful-enough v1 behavior.
+
+## Testing
+
+`just test-ci` at 100%. Validation accept/reject matrix (file/env accepted;
+`external`, multiple sources, unknown source key, non-mapping def, unknown
+referenced secret, malformed long form rejected). Emission: file `create`, env
+`printf | create -`, `--secret source/target/uid/gid/mode` opts, `mode` int->
+octal string and string passthrough, pod-namespacing, the trap `secret rm`, and
+`referenced_variables()` collecting env-source vars. An `emit_script` integration
+with a secret referenced across the closure. `just lint-ci` clean (watch `C901`
+on the new translation functions -- keep them small/split).
+
+## Risk
+
+- **Store-name collision on reused pod names** (low x med): namespacing is
+  `<pod>-<name>`; two runs sharing a pod name collide, same class as an existing
+  pod-name collision — the user's responsibility.
+- **mode representation** (med x low): PyYAML octal-int vs string; mitigated by
+  handling both and testing `0440` int and `"0440"` string.
+- **New-module scope** (low x low): `secrets.py` spans validate + emit; kept a
+  deep module with a small interface (referenced set, create lines, flags).