brownjuly2003-code · brownjuly2003-code · Jul 6, 2026 · Jul 6, 2026 · Jul 6, 2026 · Jul 6, 2026
diff --git a/docs/clickhouse-cutover-plan.md b/docs/clickhouse-cutover-plan.md
@@ -130,14 +130,39 @@ executed accidentally.
       unless `controlPlane.store=postgres` and `serving.backend=clickhouse` —
       see ADR 0010. Slice 6 (2026-07-04) released the enum ratchet to
       `[embedded, postgres]`.)*
-- [ ] **LIVE verify (kind, Docker — pending Mac/CI, E4 tail):**
-      `scripts/k8s_staging_up.sh` on kind with the scale overlay,
-      `k8s_smoke_test.sh` green, `replicaCount: 2` schedules without PVC
-      contention; **plus the ADR 0010 replica-correctness checks** — exactly one
-      delivery per (webhook, event) across two pods, one alert page per incident,
-      a webhook registered via either pod visible to both. See the recipe below.
-      *(Chart-side render verified locally 2026-07-04 via `helm template`/`lint`;
-      the two-real-pods run needs Docker, unavailable on the authoring host.)*
+- [ ] **LIVE verify (kind, Docker — attempted on the Mac 2026-07-06, genuinely
+      incomplete):** `scripts/k8s_staging_up.sh` on kind with the scale
+      overlay, `k8s_smoke_test.sh` green, `replicaCount: 2` schedules without
+      PVC contention; **plus the ADR 0010 replica-correctness checks** —
+      exactly one delivery per (webhook, event) across two pods, one alert
+      page per incident, a webhook registered via either pod visible to both.
+      See the recipe below.
+      *(Chart-side render verified locally 2026-07-04 via `helm template`/
+      `lint`, and re-verified live-adjacent on the Mac 2026-07-06: `helm lint`
+      clean; `helm template --set replicaCount=2 --set persistence.enabled=false`
+      **without** `controlPlane.store=postgres`/`serving.backend=clickhouse`
+      correctly fails at render time with the exact ADR 0010 gate error
+      (`"Multi-replica requires BOTH an external serving engine ... AND an
+      external control-plane store ..."`); the same render **with both halves
+      set** succeeds with `replicas: 2` and the correct
+      `AGENTFLOW_CONTROLPLANE_STORE`/`SERVING_BACKEND` env vars. The
+      **two-real-pods live run** (building `agentflow/api`, deploying 2 pods,
+      exercising `scripts/k8s_replica_correctness_verify.sh` against them) was
+      attempted on the Mac kind stand but could not complete: this same
+      session hit severe, sustained shared-host resource contention (the
+      Mac runs this kind cluster alongside another project's ~9 containers;
+      `/proc/loadavg` read 40–78 for extended stretches, `docker stats` showed
+      a co-tenant's Temporal engine alone at up to 80% CPU, and this cluster's
+      own control plane was crash-looping — `kube-apiserver` had restarted 10+
+      times, `kube-controller-manager` 46+ times, confirmed via `crictl ps -a`
+      directly on the node). Two separate `docker build` attempts for the
+      fresh `agentflow/api` image (needed since the cached image predates
+      ADR 0010 slice 5) stalled at near-zero CPU progress even after killing
+      a competing orphaned build process. This is an honest "attempted,
+      blocked by host resource crisis" outcome — not fabricated, and not
+      abandoned without cause; the render-gate logic itself is now more
+      thoroughly verified than before. Re-attempt on a quieter window on this
+      host, or a dedicated instance.)*
 
 ### Phase 3 replica-correctness verify recipe (ADR 0010 slice 6)
 

diff --git a/docs/decisions/0010-control-plane-externalization-postgres.md b/docs/decisions/0010-control-plane-externalization-postgres.md
@@ -208,13 +208,23 @@ ClickHouse is consumed), and `psycopg` joins the optional dependencies.
    `AGENTFLOW_CONTROLPLANE_STORE` + `AGENTFLOW_CONTROLPLANE_PG_DSN`; the scale
    profile is an overlay (`k8s/staging/values-staging-scale.yaml.example`), not
    a change to the zero-dependency default. Render side verified locally
-   (`helm template`/`lint` + `tests/unit/test_helm_values_contract.py`). The
-   **two-real-pods live run** (kind at `replicaCount=2` + the replica-correctness
-   checks above) needs Docker and is the deferred Mac/CI tail —
-   `scripts/k8s_replica_correctness_verify.sh` automates the pod-count and
-   cross-pod registration-visibility checks; the delivery/alert checks follow
-   the recipe in the cutover plan (their store-level guarantee is already
-   live-verified by slice 5's 31/31 PG probes).
+   (`helm template`/`lint` + `tests/unit/test_helm_values_contract.py`), and
+   re-verified 2026-07-06 on the Mac kind stand: the render gate correctly
+   rejects `replicaCount=2` without both halves of the gate (the exact ADR
+   error text) and correctly admits it once both are set (`replicas: 2`,
+   correct env vars). The **two-real-pods live run** (kind at
+   `replicaCount=2` + the replica-correctness checks above) needs Docker and
+   was attempted 2026-07-06 on the Mac stand — genuinely incomplete: the
+   shared host hit severe resource contention this session (co-tenant CPU
+   spikes up to 80%, this cluster's own control plane crash-looping,
+   `kube-apiserver` 10+ restarts) that stalled the `agentflow/api` image build
+   itself at near-zero CPU progress across two attempts. See
+   `docs/clickhouse-cutover-plan.md` Phase 3 for the full honest account.
+   `scripts/k8s_replica_correctness_verify.sh` remains ready to automate the
+   pod-count and cross-pod registration-visibility checks whenever the live
+   run can complete; the delivery/alert checks follow the recipe in the
+   cutover plan (their store-level guarantee is already live-verified by
+   slice 5's 31/31 PG probes).
 
 ## Consequences
 

diff --git a/docs/dv2-multi-branch/demo.cast b/docs/dv2-multi-branch/demo.cast
@@ -1,22 +1,24 @@
-{"version": 2, "width": 130, "height": 35, "timestamp": 1779531590, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}, "title": "DV2.0 multi-branch live demo"}
-[0.01269, "o", "\r\n\u001b[1;33m# === Beat 1 — Cluster topology (00:25) ===\u001b[0m\r\n"]
-[1.520249, "o", "\u001b[1;36m$\u001b[0m kubectl get nodes --show-labels | head -5\r\n"]
-[2.411272, "o", "NAME                    STATUS   ROLES           AGE     VERSION   LABELS\r\nhq-demo-control-plane   Ready    control-plane   6h18m   v1.35.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,nodepool=hq-control\r\nhq-demo-worker          Ready    <none>          6h18m   v1.35.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker,kubernetes.io/os=linux,nodepool=hq-data-tier-a,workload=postgres\r\nhq-demo-worker2         Ready    <none>          6h18m   v1.35.0   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker2,kubernetes.io/os=linux,nodepool=hq-data-tier-b,workload=clickhouse\r\n"]
-[4.921556, "o", "\r\n\u001b[1;33m# === Beat 2 — DV2.0 model surface (00:45) ===\u001b[0m\r\n"]
-[6.434524, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('_', name)[1] AS kind, count() FROM system.tables WHERE database='rv' GROUP BY kind ORDER BY kind\"\r\n"]
-[7.428901, "o", "bv\t6\r\nhub\t8\r\nlnk\t8\r\nsat\t39\r\n"]
-[9.95663, "o", "\r\n\u001b[1;33m# === Beat 3 — Multi-branch distribution (01:05) ===\u001b[0m\r\n"]
-[11.465034, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('__', record_source)[2] AS branch, count() AS orders, round(count()*100.0/(SELECT count() FROM rv.hub_order),1) AS pct FROM rv.hub_order GROUP BY branch ORDER BY pct DESC FORMAT PrettyCompact\"\r\n"]
-[12.475313, "o", "   ┌─branch─┬─orders─┬──pct─┐\r\n1. │ msk    │   4200 │ 40.5 │\r\n2. │ spb    │   2500 │ 24.1 │\r\n3. │ ekb    │   1500 │ 14.5 │\r\n4. │ dxb    │   1160 │ 11.2 │\r\n5. │ ala    │   1000 │  9.7 │\r\n   └────────┴────────┴──────┘\r\n"]
-[14.995744, "o", "\r\n\u001b[1;33m# === Beat 4 — Business Vault MDM (01:25) ===\u001b[0m\r\n"]
-[16.50705, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --multiline --query \"SELECT 'msk' AS branch, count() AS rows, countIf(first_name != '') AS with_pii, countIf(loyalty_segment != '') AS with_loyalty FROM rv.bv_customer_mdm__msk UNION ALL SELECT 'dxb', count(), countIf(first_name != ''), countIf(loyalty_segment IS NOT NULL AND loyalty_segment != '') FROM rv.bv_customer_mdm__dxb FORMAT PrettyCompact\"\r\n"]
-[17.550896, "o", "   ┌─branch─┬─rows─┬─with_pii─┬─with_loyalty─┐\r\n1. │ dxb    │  200 │      200 │            0 │\r\n2. │ msk    │  800 │      800 │          640 │\r\n   └────────┴──────┴──────────┴──────────────┘\r\n"]
-[20.072279, "o", "\r\n\u001b[1;33m# === Beat 5a — Cold offload (01:50) ===\u001b[0m\r\n"]
-[21.576438, "o", "\u001b[1;36m$\u001b[0m kubectl create job --from=cronjob/dv2-cold-offload-msk cold-demo-3455 -n dv2\r\n"]
-[22.483005, "o", "job.batch/cold-demo-3455 created\r\n"]
-[32.996093, "o", "\r\n\u001b[1;33m# === Beat 5b — MinIO parquet listing ===\u001b[0m\r\n"]
-[34.50554, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 minio-0 -- mc ls -r local/cold-tier | tail -5\r\n"]
-[35.495859, "o", "[2026-05-23 09:44:37 UTC] 6.7KiB STANDARD branch=ala/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 09:44:32 UTC] 6.7KiB STANDARD branch=dxb/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 09:44:37 UTC] 8.6KiB STANDARD branch=ekb/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 10:20:13 UTC]  20KiB STANDARD branch=msk/year=2026/month=05/customers_anon.parquet\r\n[2026-05-23 09:44:32 UTC]  13KiB STANDARD branch=spb/year=2026/month=05/customers_anon.parquet\r\n"]
-[38.005981, "o", "\r\n\u001b[1;33m# === Closing (02:00) — Single bootstrap reproduces everything ===\u001b[0m\r\n"]
-[39.516429, "o", "\u001b[1;36m$\u001b[0m bash infrastructure/dv2/bootstrap.sh\r\n"]
-[41.521865, "o", "\r\n\u001b[1;33mDemo complete. See docs/dv2-multi-branch/demo_evidence.md for the full breakdown.\u001b[0m\r\n"]
+{"version": 2, "width": 130, "height": 35, "timestamp": 1783334766, "env": {"SHELL": "/bin/bash", "TERM": "xterm-256color"}, "title": "DV2.0 multi-branch live demo"}
+[0.033084, "o", "\r\n\u001b[1;33m# === Beat 1 — Cluster topology (00:25) ===\u001b[0m\r\n"]
+[1.554022, "o", "\u001b[1;36m$\u001b[0m kubectl get nodes --show-labels | head -5\r\n"]
+[2.805528, "o", "NAME                    STATUS   ROLES           AGE     VERSION   LABELS\r\nhq-demo-control-plane   Ready    control-plane   5h30m   v1.32.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=,nodepool=hq-control\r\nhq-demo-worker          Ready    <none>          5h29m   v1.32.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker,kubernetes.io/os=linux,nodepool=hq-data-tier-a,workload=postgres\r\nhq-demo-worker2         Ready    <none>          5h29m   v1.32.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,branch=msk,kubernetes.io/arch=amd64,kubernetes.io/hostname=hq-demo-worker2,kubernetes.io/os=linux,nodepool=hq-data-tier-b,workload=clickhouse\r\n"]
+[5.31246, "o", "\r\n\u001b[1;33m# === Beat 2 — DV2.0 model surface (00:45) ===\u001b[0m\r\n"]
+[6.826426, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('_', name)[1] AS kind, count() FROM system.tables WHERE database='rv' GROUP BY kind ORDER BY kind\"\r\n"]
+[13.226603, "o", "bv\t6\r\nhub\t8\r\nlnk\t8\r\nsat\t48\r\n"]
+[17.616754, "o", "\r\n\u001b[1;33m# === Beat 3 — Multi-branch distribution (01:05) ===\u001b[0m\r\n"]
+[19.126624, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --query \"SELECT splitByString('__', record_source)[2] AS branch, count() AS orders, round(count()*100.0/(SELECT count() FROM rv.hub_order),1) AS pct FROM rv.hub_order GROUP BY branch ORDER BY pct DESC FORMAT PrettyCompact\"\r\n"]
+[21.499089, "o", "   ┌─branch─┬─orders─┬──pct─┐\r\n1. │ msk    │   9740 │ 94.7 │\r\n2. │ spb    │    180 │  1.8 │\r\n3. │ dxb    │    155 │  1.5 │\r\n4. │ ekb    │    130 │  1.3 │\r\n5. │ ala    │     75 │  0.7 │\r\n   └────────┴────────┴──────┘\r\n"]
+[24.128329, "o", "\r\n\u001b[1;33m# === Beat 4 — Business Vault MDM (01:25) ===\u001b[0m\r\n"]
+[25.635284, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 clickhouse-0 -- clickhouse-client --user default --password demo --multiline --query \"SELECT 'msk' AS branch, count() AS rows, countIf(first_name != '') AS with_pii, countIf(loyalty_segment != '') AS with_loyalty FROM rv.bv_customer_mdm__msk UNION ALL SELECT 'dxb', count(), countIf(first_name != ''), countIf(loyalty_segment IS NOT NULL AND loyalty_segment != '') FROM rv.bv_customer_mdm__dxb FORMAT PrettyCompact\"\r\n"]
+[37.092843, "o", "   ┌─branch─┬─rows─┬─with_pii─┬─with_loyalty─┐\r\n1. │ dxb    │   80 │       80 │            0 │\r\n2. │ msk    │ 2240 │     2240 │          152 │\r\n   └────────┴──────┴──────────┴──────────────┘\r\n"]
+[40.056267, "o", "\r\n"]
+[40.056349, "o", "\u001b[1;33m# === Beat 5a — Cold offload (01:50) ===\u001b[0m\r\n"]
+[41.566622, "o", "\u001b[1;36m$\u001b[0m kubectl create job --from=cronjob/dv2-cold-offload-msk cold-demo-5534 -n dv2\r\n"]
+[42.711019, "o", "job.batch/cold-demo-5534 created\r\n"]
+[53.229094, "o", "\r\n\u001b[1;33m# === Beat 5b — MinIO parquet listing ===\u001b[0m\r\n"]
+[54.745112, "o", "\u001b[1;36m$\u001b[0m kubectl exec -n dv2 minio-0 -- mc ls -r local/cold-tier | tail -5\r\n"]
+[62.23601, "o", "[2026-07-06 06:00:06 UTC] 4.1KiB STANDARD branch=ala/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 05:59:53 UTC] 3.7KiB STANDARD branch=dxb/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 05:59:49 UTC] 3.9KiB STANDARD branch=ekb/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 10:44:10 UTC]  50KiB STANDARD branch=msk/year=2026/month=07/customers_anon.parquet\r\n[2026-07-06 05:59:48 UTC] 4.6KiB STANDARD branch=spb/year=2026/month=07/customers_anon.parquet\r\n"]
+[64.753412, "o", "\r\n\u001b[1;33m# === Closing (02:00) — Single bootstrap reproduces everything ===\u001b[0m\r\n"]
+[66.268043, "o", "\u001b[1;36m$\u001b[0m bash infrastructure/dv2/bootstrap.sh\r\n"]
+[68.281547, "o", "\r\n"]
+[68.281596, "o", "\u001b[1;33mDemo complete. See docs/dv2-multi-branch/demo_evidence.md for the full breakdown.\u001b[0m\r\n"]