One-command Kubernetes + AWS deployment, multi-arch images

[EdgarBabajanyan]

Problem

"Deploy Compass serverless on AWS" was a docs page (docs/deployment.md), not an artifact. A stranger had no manifests, no Terraform, and — silently worse — no public container image at all: the release workflow publishes binaries, nothing publishes to a registry, and arm64 images were impossible (bookworm's gcc-12 fails on the numkong dependency's ARM feature probes):

error: failed to run custom build command for `numkong v7.6.0`
cc1: error: invalid feature modifier 'sve2p1' in '-march=armv8.2-a+sve2p1'

What this ships

deploy/kubernetes — kustomize tree for the full serverless topology on any cluster:

• Serving StatefulSet with a PVC per replica — warm disks make restarts a seconds-long delta catch-up, and two replicas must never share one volume. A startupProbe (up to 60 min) covers the real boot behavior: the engine binds its port only after loading collections, so without it a big cold rebuild gets liveness-killed into a crash loop.
• Stateless writer Deployment (emptyDir scratch, kill-any-time) and an optional cold-serving component (serve-from-storage tier, warm promotion off).
• Hardened by default: runAsNonRoot as the image's uid 10001, seccomp RuntimeDefault, all capabilities dropped, fsGroupChangePolicy: OnRootMismatch so a large warm cache isn't recursively chowned on every mount.
• Overlays: minio-dev (the whole topology self-contained on kind/k3s/minikube in one kubectl apply -k) and aws (real S3; IRSA or access-key auth; the AWS secret is optional so IRSA needs none).

deploy/terraform/aws — the storage half: private S3 bucket (public-access block, KMS encryption, TLS-only bucket policy, incomplete-multipart cleanup, versioning deliberately off — the LSM manages object lifecycle and versioning would resurrect deleted fragments, force_destroy=false so destroy refuses while data exists) + least-privilege IAM (object CRUD/list/multipart only) as an IRSA role or an opt-in access key.

.github/workflows/docker.yml — publishes ghcr.io/runcaptain/compass (amd64+arm64) on release tags; the overlays reference it. Builder base → rust:1.88-trixie (same toolchain pin; gcc-14 unblocks arm64).

Tested, not just written

• Live k3s cluster (in Docker): minio-dev overlay applied → all tiers 1/1 Running, PVC Bound, security context verified applied. 8/8 in-cluster checks through the Services: health on all tiers, writer-tier ingest → serving-tier min_seq read-your-writes, writer role refusal, cold tier answering from the bucket. Data survives serving-pod deletion (PVC warm restart).
• Terraform against an AWS API emulator (moto): apply succeeds; public-access block 4/4, KMS on; re-apply is a 0-change no-op; destroy with data in the bucket refuses (BucketNotEmpty) exactly as designed.
• Both image architectures build from the new base (arm64 verified by running the full topology on it; amd64 rebuilt as a regression check).
• Iterated through real failures the live test caught: image pull policy vs imported images, containerd k8s.io namespace, uid mismatch (65532 vs the image's 10001), and the boot-order/probe interaction.

Notes

• Deliberately not included (documented in deploy/README.md): NetworkPolicy (cluster-specific), PodDisruptionBudget (single-replica default), tenant-affinity routing (roadmap).
• After merge, run the Docker workflow once (workflow_dispatch) or wait for the next tag so ghcr.io/runcaptain/compass exists before announcing the overlays.
• writer refuses reads returns HTTP 500 today; 403 would be cleaner — engine nit, noted for a follow-up, not deploy scope.