Skip to content

fix: Docker cache#2314

Merged
sergerad merged 4 commits into
nextfrom
sergerad-fix-docker-cache
Jul 6, 2026
Merged

fix: Docker cache#2314
sergerad merged 4 commits into
nextfrom
sergerad-fix-docker-cache

Conversation

@sergerad

@sergerad sergerad commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

We have Docker builds failing due to mismatched caching https://github.com/0xMiden/node/actions/runs/28658546904/job/84993414912?pr=2312.

bin/ntx-builder/src/server/get_network_note_status.rs had been updated (in a newer commit) to implement handle() with 4 params, matching a newer version of the GetNetworkNoteStatus trait in miden-node-proto. But the build linked that fresh ntx-builder source against a stale, previously-cached miden-node-proto .rlib — compiled from an older commit where the trait still only declared 2 params. Two different commits' code got mixed into the same build.

The cache mounts have no id= at all (--mount=type=cache,sharing=locked,target=/app/target), so there was a single global, unkeyed /app/target mount shared by every arch and every binary. Cargo decides whether to recompile a path crate by comparing source file mtimes against its recorded fingerprint mtimes. BuildKit's COPY . . stamps every file with the same normalized timestamp regardless of actual content — so from Cargo's point of view, miden-node-proto's source didn't look "newer" than the fingerprint left behind by whichever earlier build populated that shared mount, even though the trait definition had genuinely changed since then. Cargo concluded "unchanged, reuse the cached .rlib" and just relinked it, instead of recompiling.

Docker's timestamp normalization defeats Cargo's mtime-based staleness check on a shared build-cache mount. The solution is to restore each file's real last-commit mtime before compiling. This makes sure a file whose content actually changed since the cached build always looks newer to Cargo, so the crate gets recompiled instead of silently reusing an incompatible cached artifact.

Changes

  • Restore real file mtimes before compiling — BuildKit normalizes every COPY'd file to the same timestamp, which breaks Cargo's mtime-based fingerprinting and let it silently reuse stale .rlibs from a different commit sharing the same /app/target cache mount. A new workflow step generates docker-file-mtimes.tsv from git log, and the Dockerfile touches each file back to its real last-commit time before the final cargo build.
  • Key cache mounts per architecture — cargo-registry, cargo-git, and app-target cache mounts now include ${TARGETARCH} in their id, so amd64 and arm64 builds never share compiled objects, registry sources, or git checkouts.
  • Scope cargo chef prepare/cook to the target binary — both now take --bin ${BIN}, matching the final cargo build --bin ${BIN}. Previously they resolved features for the whole workspace, which never matched what a single binary needs (e.g. miden-protocol, miden-tx enable different, non-overlapping features per binary), forcing shared deps to recompile on every build regardless of caching.
  • Key the /app/target mount per binary too — app-target-${BIN}-${TARGETARCH} gives each of the 5 matrix builds (node, validator, remote-prover, network-monitor, ntx-builder) its own target dir, so they stop invalidating each other's cached artifacts on a shared, lock-guarded mount. Registry/git-db mounts stay arch-only since raw sources aren't feature-dependent and are worth sharing.
  • Fetch full git history in CI (fetch-depth: 0) so the mtime step can look up each tracked file's actual last-commit timestamp, not just files touched by the tip commit.

Note: the per-binary cache mount id change means the first CI run after this PR merges will cold-build every image (as seen in run 28758866086) — expected one-time cost before the new caches are warm. Subsequent run showed correct cache usage.

Changelog

changelog = "none"
reason    = "Internal change only."

@sergerad sergerad changed the title Add per-file mtime fix: Docker cache Jul 5, 2026

@kkovaacs kkovaacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

Hope this indeed solves the stale cache issues we've seen from time to time...

@sergerad sergerad merged commit 31d8b22 into next Jul 6, 2026
25 checks passed
@sergerad sergerad deleted the sergerad-fix-docker-cache branch July 6, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants