Feature/kubernetes job runner

.dockerignore

@@ -0,0 +1,7 @@
+.git
+.github
+.venv
+__pycache__
+tests
+deploy
+*.pyc

.github/workflows/container.yml

@@ -0,0 +1,105 @@
+name: Container
+
+# Publishes a PUBLIC multi-architecture image to GHCR.
+#
+# Public on purpose: a private image forces every operator to copy a registry
+# credential into the customer's namespace as an imagePullSecret and remember to
+# delete it afterwards. A public image removes that step entirely.
+#
+# Multi-arch on purpose: ClickHouse node pools are frequently arm64 (one customer
+# cluster is 5x arm64 + 1x amd64), and an amd64-only image can only ever schedule
+# on a fraction of such a pool.
+
+on:
+  pull_request:
+  push:
+    branches: [master]
+    tags: ['v*.*.*']
+
+env:
+  REGISTRY: ghcr.io
+  IMAGE_NAME: altinity/s3gc
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+      - run: python -m pip install --disable-pip-version-check -r requirements.txt -r requirements-dev.txt
+      # Development-cluster tests require explicit credentials and are never
+      # part of the offline pull-request gate.
+      - run: pytest -v -m "not dev_cluster"
+      # The renderer is part of the contract: it refuses images that are not
+      # digest-pinned, so a broken template breaks every deployment.
+      - run: python deploy/kubernetes/render.py deploy/kubernetes/example.env > "$GITHUB_WORKSPACE/.s3gc-job.yaml"
+      # Client-side kubectl validation requires a Kubernetes API server for
+      # OpenAPI discovery. Kubeconform validates the rendered standard-resource
+      # manifest against pinned Kubernetes schemas without cluster access.
+      - name: Rendered manifest must conform to Kubernetes schemas
+        uses: docker://ghcr.io/yannh/kubeconform@sha256:85dbef6b4b312b99133decc9c6fc9495e9fc5f92293d4ff3b7e1b30f5611823c
+        with:
+          entrypoint: /kubeconform
+          args: -strict -summary /github/workspace/.s3gc-job.yaml
+
+  publish:
+    needs: test
+    # Fork pull requests cannot write packages, and there is nothing to publish
+    # from them anyway.
+    if: github.event_name == 'push'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: docker/setup-qemu-action@v3
+
+      - uses: docker/setup-buildx-action@v3
+
+      - name: Log in to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          # Automatic token: this repository lives in the Altinity organisation,
+          # so it already grants packages:write for ghcr.io/altinity/*.
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Docker metadata
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+          tags: |
+            type=semver,pattern={{version}}
+            type=sha,format=long
+            type=raw,value=latest,enable={{is_default_branch}}
+
+      - name: Build and push
+        id: image
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          file: docker/Dockerfile
+          platforms: linux/amd64,linux/arm64
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          annotations: ${{ steps.meta.outputs.annotations }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Record immutable image reference
+        # render.py requires IMAGE to be digest-pinned, so this is the exact
+        # string an operator pastes into their .env file.
+        run: |
+          {
+            echo "### Image published"
+            echo '```'
+            echo "IMAGE=${REGISTRY}/${IMAGE_NAME}@${{ steps.image.outputs.digest }}"
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"

.gitignore

@@ -1 +1,5 @@
+__pycache__/
+*.pyc
+options.lst
+.pytest_cache/
 .venv

.python-version

@@ -0,0 +1 @@
+3.11

AGENTS.md

@@ -0,0 +1,38 @@
+# Codex guide — s3gc
+
+Read `CLAUDE.md`, `CHANGELOG.md`, and `TODO.md` before making substantive
+changes in this repository. `CLAUDE.md` is the full contributor guide and the
+primary source of truth; the changelog provides recent historical context and
+the TODO file records possible, interesting, and deliberately deferred
+improvements.
+
+This file is intentionally short so agent tooling can find the critical rules
+quickly, then defer to `CLAUDE.md` for complete repository guidance.
+
+## Critical rules
+
+1. **Read `CLAUDE.md` first, then `CHANGELOG.md` and `TODO.md`.** Treat them
+   as required repository context, not optional background reading.
+2. **Protect destructive-operation safeguards.** `s3gc` deletes orphaned S3
+   objects only after collection, dry-run review, explicit confirmation, and
+   the applicable ClickHouse cluster/replica preflight. Do not weaken these
+   controls without explicit approval and matching tests and documentation.
+3. **Test coverage and required checks are mandatory.** For every feature,
+   bug fix, or material behaviour change, add a focused test when no existing
+   test covers it; use test-driven development for behaviour changes. Run the
+   relevant offline pytest suite and Kubernetes renderer/manifest dry-run
+   checks described in `CLAUDE.md`. Automated tests must not contact live
+   ClickHouse or object storage, or delete objects.
+4. **No secrets or customer data in Git.** Never commit credentials, customer
+   configuration, target-cluster details, or rendered customer manifests. Use
+   Kubernetes Secrets or workload identity for production credentials.
+5. **Keep deployments immutable and least-privileged.** Preserve digest-pinned
+   images, non-root/read-only container settings, and renderer validation.
+6. **Keep dependencies deliberate.** Production dependencies belong in
+   `requirements.txt`; testing-only dependencies belong in
+   `requirements-dev.txt`. Do not add either casually.
+
+## Working rule
+
+When `AGENTS.md` and `CLAUDE.md` differ, update them to match, but follow the
+more complete guidance in `CLAUDE.md` for the current task.

CHANGELOG.md

@@ -0,0 +1,186 @@
+# Changelog
+
+Notable changes to `s3gc`, newest first.
+
+This file records **what changed and why**, with enough context that someone
+picking the repository up later — human or agent — can tell a deliberate design
+decision from an accident. Defects found by running the tool against real
+clusters carry their evidence, because the reasoning is usually the expensive
+part to reconstruct.
+
+Format loosely follows [Keep a Changelog](https://keepachangelog.com/).
+Customer names, cluster identifiers and credentials never appear here (see
+`CLAUDE.md`, rule 3); findings are described in terms of the behaviour they
+expose.
+
+## [Unreleased]
+
+Changes below are on `feature/kubernetes-job-runner` and not yet released.
+
+### Fixed
+
+- **`--age` silently collected nothing for anything older than a day.** The
+  filter used `timedelta.seconds`, the sub-day remainder (0..86399), so computed
+  age never exceeded 23 h: a 30 d 5 h old object reported **5**. Harmless at the
+  default `age=0`, which is why it went unnoticed — but `--age 24`, the natural
+  choice by analogy with `--useage 24`, would produce an empty auxiliary table
+  and a dry-run reporting a clean bucket. Now uses `total_seconds()`.
+
+- **`--usecollected` against a missing or empty auxiliary table exited 0**,
+  which is indistinguishable from success. That is exactly what a load-balanced
+  ClickHouse Service produces, because the auxiliary table is a *node-local*
+  `ReplacingMergeTree`: collect writes it on one replica and a later phase looks
+  for it on another. It now fails loudly and explains the replica-pinning
+  requirement.
+
+- **GCS endpoints now fall back to per-object deletion automatically.** Google
+  Cloud Storage's S3-compatible API has no batch `DeleteObjects`, so
+  `remove_objects()` fails there. Detected from the endpoint, with a warning
+  that the per-object path is markedly slower.
+
+### Added
+
+- **S3 authentication modes — `--s3auth=static|aws|iam`**, merging
+  [PR #2](https://github.com/Altinity/s3gc/pull/2) by **@realyota**, which added
+  `--s3auth=aws`, `--s3profile` and `--s3-session-token` so credentials can come from
+  the boto3 chain (AWS SSO profiles) or as explicit temporary credentials.
+
+  That PR replaced the workload-identity path outright. Both are kept instead,
+  because they are not interchangeable: `iam` uses MinIO's own provider, needs no
+  `boto3`, and is what the validated Kubernetes deployments use — it hands MinIO the
+  *provider* rather than frozen keys so credentials refresh across a long collect or
+  delete. `aws` resolves through boto3 and suits a workstation with SSO.
+
+  | Mode | Credentials | boto3 |
+  |---|---|---|
+  | `static` (default) | explicit keys, optional session token | no |
+  | `aws` | boto3 chain / `--s3profile` | **yes** |
+  | `iam` | MinIO workload identity (IRSA/IMDS/ECS) | no |
+
+  `--s3profile` implies `aws`. Contradictory combinations are rejected rather
+  than silently resolved, so nobody ends up authenticating with an identity they
+  did not ask for.
+
+- **Operator-facing S3 listing errors**, also from PR #2: a failed listing now names
+  the required permission (`s3:ListBucket` on the bucket ARN, **even for
+  `--dry-run`**) and prints the `aws sts get-caller-identity` /
+  `aws s3api list-objects-v2` commands to verify the same credentials. `UserVisibleError`
+  reports such failures without a traceback.
+
+- **Wider log-secret filtering** (PR #2): `s3accesskey` and `s3sessiontoken` are now
+  redacted alongside `chpass` and `s3secretkey`.
+
+- **Kubernetes wiring for the new auth**, which PR #2 did not include: `S3AUTH` and
+  `S3PROFILE` are exposed by `job.yaml.tmpl`, required and validated by `render.py`
+  (`S3PROFILE` without `S3AUTH=aws` is an error), and default to `iam` in
+  `example.env`. Without this the flags were unreachable from a Job. The session token
+  stays out of the non-secret env file — it belongs in the credentials Secret.
+
+- **Warning when `--samples` disagrees with the auxiliary table's
+  `PARTITION BY`.** The table is created as `PARTITION BY CRC32(objpath) %
+  <samples>` at collect time, so a different value during the use phase loses
+  partition pruning. Measured on production-scale data: ~2 min per sample when
+  matched against ~26 min when not.
+
+- **Cumulative deletion total alongside the per-attempt one.** The closing
+  `N objects … are removed` line counts only the process that printed it, which
+  understated one resumed run by 16.61 TiB. It now says "in this attempt" and
+  logs the auxiliary table's cumulative tombstone count.
+
+- **30 regression tests** covering the boolean matrix (per flag, per spelling,
+  plus bare-CLI compatibility), the age filter, the fail-loud path, the samples
+  warning, the GCS fallback, and renderer pull-secret handling.
+
+### Changed
+
+- **Images are published publicly to `ghcr.io/altinity/s3gc`** instead of a
+  private Docker Hub repository, matching `altinity-mcp` and
+  `altinity-sql-browser`. CI authenticates with the automatic `GITHUB_TOKEN`,
+  so there is no registry secret to manage or rotate.
+
+  *Why it matters operationally:* a private image forces whoever runs a Job to
+  copy a registry credential into the target namespace as an `imagePullSecret`
+  and remember to delete it afterwards. A public image removes that step
+  entirely.
+
+- **`boto3` is a pinned runtime dependency** (`boto3==1.43.65`), required by
+  `--s3auth=aws`. It is imported lazily, so the other modes never load it. PR #2 added
+  it unpinned; the pinned set is kept per `CLAUDE.md` rule 5 — unpinned `jsonargparse`
+  resolves to 4.50.x and fails at import.
+
+- **`IMAGE_PULL_SECRET` is now optional.** `render.py` omits the
+  `imagePullSecrets` block when the value is empty, rather than emitting a
+  meaningless `- name: ""`. Set it only for a private mirror.
+
+- **Multi-architecture builds are mandatory, not advisory.** ClickHouse node
+  pools are frequently arm64 — one observed pool was 5× arm64 and 1× amd64,
+  where an amd64-only image can only ever schedule on a sixth of the capacity.
+  CI builds `linux/amd64,linux/arm64` in a single step.
+
+- **CI validates the rendered manifest without a Kubernetes cluster.** Strict,
+  digest-pinned Kubeconform schema validation replaces client-side `kubectl`,
+  which attempts OpenAPI discovery against a nonexistent API server on GitHub
+  runners.
+
+- **CI explicitly excludes `dev_cluster` tests.** The pull-request suite stays
+  offline even after opt-in environment-dependent coverage is added.
+
+- Removed the deprecated S3 IAM selector. `S3AUTH=static|aws|iam` is now the
+  only supported authentication interface.
+
+### Documentation
+
+- `CHHOST` must be a **per-replica** Service, never the load-balanced one, and
+  every phase of a cleanup must use the same host.
+- S3 authentication guidance now keeps static, AWS SSO/profile, and workload
+  identity modes together; the Kubernetes guide calls out dedicated ClickHouse
+  user provisioning and only the required table and system-table grants.
+- Contributor guidance now requires test-driven, focused coverage for every
+  feature, defect, and material operational behaviour change.
+- Sharding recipe for `--collectonly`, which has no resume: a crash re-lists
+  from the beginning, which is expensive on multi-million-object buckets.
+  Re-running a shard is safe because the auxiliary table is a
+  `ReplacingMergeTree` keyed on `objpath`.
+- Per-cluster values that cause silent failure when wrong: `S3PATH` may
+  legitimately be empty (bucket-root layouts); the disk is not always named
+  `s3` (GCS-backed clusters commonly use `gcs`, which changes both the
+  anti-join scope and the auxiliary table name); a `*_cache` disk is a
+  filesystem cache over the same blobs, not a second reference scope.
+- Buildx builder containers cache `/etc/resolv.conf` at creation, so a builder
+  left running across a network change fails with
+  `lookup registry-1.docker.io: i/o timeout` while the host resolves fine.
+
+### Migration notes
+
+- **Pull secrets are registry-scoped.** Moving from Docker Hub to GHCR silently
+  invalidates an existing `imagePullSecret` even though its *name* still looks
+  right: a secret holding `index.docker.io` credentials does not apply to
+  `ghcr.io`, so the kubelet falls back to an anonymous token and the pod sits in
+  `ImagePullBackOff` with `failed to fetch anonymous token: 401 Unauthorized`.
+  With the public image the correct action is to **remove** the secret reference
+  (leave `IMAGE_PULL_SECRET` empty), not to repoint it.
+
+- **The GHCR package must be made public once**, in the organisation's package
+  settings. It cannot be done through the REST API — the visibility endpoint
+  returns 404 and the standard token lacks `write:packages`. Until it is flipped,
+  every pull still needs credentials and the benefit above is not realised.
+
+### Verified
+
+All three phases were exercised end to end against a development ClickHouse cluster using the
+image built from this branch, pulled anonymously from the public registry with no
+`imagePullSecret`: `collect` (188 objects), `dry-run` (exactly the 16 seeded orphan
+fixtures), `delete` (cluster preflight, per-batch checkpoints, cumulative total) and a
+verifying `dry-run` reporting zero. `auth=iam` in the log confirms workload identity still
+resolves after the credential-resolution rewrite, and the referenced tables were untouched.
+Unit tests cover the `static`/`aws` modes; the GCS per-object fallback is still only
+unit-tested, pending a real GCS endpoint.
+
+## v0.2 — 2025-01-31
+
+- Added an option to avoid batch deletion for services such as GCS.
+
+## v0.1 — 2024-06-12
+
+- Added object last-modified timestamps to the auxiliary table.
+- Added the object age option.