ci: rebuild only Docker variants whose bases changed and make images reproducible - #2545
ci: rebuild only Docker variants whose bases changed and make images reproducible#2545xavierleune wants to merge 1 commit into
Conversation
…reproducible Scheduled rebuilds used a single fingerprint covering the 15 base images of all variants: any single base update (e.g. php:8.3-zts-trixie or golang:1.26-trixie) rebuilt and re-pushed all 24 images, moving every generic tag (php8.4, 1-php8.4, ...) roughly every other day. Comparing two such rebuilds of the same release showed byte-identical content (same dpkg versions, same frankenphp binary) but different digests, caused only by build artifacts: timestamp() label, layer mtimes and embedded provenance. - Compute one fingerprint per variant (its php-base + golang-base digests) and only rebuild/push the variants whose fingerprint changed. The label is read from the versioned builder tag; the legacy all-variants fingerprint is still accepted so the transition (old bake file in the latest release tag) keeps the current behavior until the next release. - Store the fingerprint label on builder images only, so runner digests depend solely on their content. - Make images reproducible: SOURCE_DATE_EPOCH and created/revision labels pinned to the built commit, rewrite-timestamp=true on push, BuildKit provenance disabled (its timestamps made every digest unique; the signed GitHub attestations pushed by actions/attest are kept), and removal of files with non-deterministic content (apt logs, ldconfig aux-cache) from Debian runner layers. Two independent no-cache builds of the 8.4-trixie runner now produce the same digest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
I like the idea, but we need to double check that this won't break anything. It's a tricky part of the CI! |
|
yep that's super tricky. LMK if I can help |
|
That's why I use my packages in ubi 10 docker containers 🤣. Rob referred to that time as Deskation, but I'm far too impatient. I'll review later, but it's a good idea. |
| # Remove files with non-deterministic content (embedded timestamps and mtimes) to keep image digests reproducible | ||
| rm -rf /var/lib/apt/lists/* /var/log/apt /var/log/dpkg.log /var/log/alternatives.log /var/cache/ldconfig/aux-cache |
There was a problem hiding this comment.
I assume you've made sure that this actually leads to reproducible image digests for base files. Otherwise everything looks correct.
There was a problem hiding this comment.
yes I did and it worked pretty well👌
Multiple builds will lead to the same build hash, so that combined to the build gated per version that really changes, it should work pretty well to reduce full rebuilds
3 participants
Problem
The nightly scheduled rebuild uses a single fingerprint covering the base images of all variants (
php:{8.2,8.3,8.4,8.5}-zts-{trixie,bookworm,alpine}+golang:1.26-*). When any single one of them moves, all 24 images are rebuilt and re-pushed, so every generic tag (php8.4,1-php8.4,latest, …) gets a new digest roughly every other day (13 full rebuilds in the last 30 scheduled runs).Comparing two such rebuilds of the same release (July 16 vs July 18,
1-php8.4, both triggered by base images unrelated to the 8.4 variant) shows byte-identical content — same dpkg package versions, samefrankenphpbinary sha256, sameinstall-php-extensions, same watcher libs — yet different digests. The churn came only from build artifacts:org.opencontainers.image.created = timestamp()(guarantees a new config digest on every build),WORKDIR /applayer differed),org.opencontainers.image.revisionpointing atmain's HEAD instead of the built release tag.This causes spurious pulls/deploys for users who track digests (watchtower, renovate, digest pinning, CI caches).
Changes
Per-variant fingerprint
docker-compute-fingerprints.shnow computes one fingerprint per variant (the digests of itsphp-base+golang-basecontexts) and outputs arebuild_variantslist; the build/push matrix only includes variants whose bases actually changed.docker-bake.hclis the pre-change one, so behavior stays exactly as today until the next release ships the new bake file, then per-variant skipping kicks in automatically.main()function: it replaces itself on disk viagit checkoutwhile running, and bash reads scripts lazily.preparejob (the per-variant check reads 12 labels via skopeo).Reproducible images
SOURCE_DATE_EPOCHand thecreatedlabel are pinned to the built commit's date;revisionnow points at the release tag's commit on scheduled runs.rewrite-timestamp=trueon the push output clamps layer file mtimes.*.attest=type=provenance,disabled=true): its embedded timestamps made every digest unique. The signed GitHub attestations pushed byactions/attest(push-to-registry: true) are unchanged./var/log/dpkg.log,/var/log/apt,/var/log/alternatives.log, ldconfig'saux-cachewhich embeds mtimes).With this, a rebuild of unchanged inputs converges to the same digest, so tags only move when content actually changes.
Verification
--no-cachebuilds ofrunner-php-8-4-23-trixie(linux/arm64, BuildKit v0.31.2) produce the same manifest digest (sha256:516a940f…) — the Go build itself is already deterministic.ff8d5d61…), so the transition is seamless.skip=true; simulating aphp:8.3-zts-trixiedigest change post-transition rebuilds onlyphp-8-3-32-trixie.shellcheck,actionlintanddocker buildx bake --print(default variables, local usage) all pass.Behavioral changes worth maintainers' attention
org.opencontainers.image.created(and the OCI configcreatedfield) now reflect the built commit's date instead of the build date.dev.frankenphp.base.fingerprintlabel (builder images keep it).Out of scope (accepted): builder images (Go build cache) and Alpine (apk db not double-build-tested) are not guaranteed reproducible; with per-variant skipping they are only rebuilt when their own bases change anyway.
🤖 Generated with Claude Code