feat(proxies): cargo/crates pull-through cache, and fix the Go module proxy

[luthermonson]

The Go module proxy has never served a request

It binds to the CNI bridge gateway (10.88.0.1:8082), but that address does not exist when the daemon starts. The bridge is created lazily by CNI on the first job container, and both cleanup() and cleanStaleBridge() delete it outright (pkg/networking/network_linux.go). So Listen fails with EADDRNOTAVAIL on every boot, the error is logged as a warning and swallowed, and GOPROXY is never injected into any container.

Start() runs MkdirAll before the failing Listen, which is exactly why cache/gomod exists on every node, empty, at 4 KB. That empty directory was the clue.

Three further defects would have kept it useless even with the bind fixed:

1. EnvVars() advertised the bound address rather than the configured one — meaningless inside a container after any fallback.
2. GOPROXY=…,direct is not fail-open: the comma separator only falls through on 404/410, so a wedged proxy hard-failed builds. Now |direct.
3. cleanup was dead code (if !cleanup { cleanup = true }) — the cache was wiped on every shutdown regardless of configuration.

Re-checked against current main after the L2Bridge work landed: Manager.GatewayIP() now returns the host adapter address on the Windows L2Bridge path, where a direct bind succeeds and the fallback never fires. On Linux the CNI bridge is still created with the first job container, so the wildcard fallback in proxies.Listen is still what makes the proxy work at all. The fix holds.

Changes

• pkg/proxies/listen.go (new) — Listen() with a wildcard fallback, so the gateway address becomes reachable as soon as CNI brings the bridge up. A genuine error (port in use) is still returned rather than masked.
• pkg/proxies/server.go (new) — the HTTP server lifecycle shared by every cache proxy. See "Shutdown" below.
• pkg/proxies/go — advertise the configured address; |direct so any proxy error falls through to upstream; honour cleanup = false; use the shared server.
• pkg/proxies/cargo (new) — pull-through cache for the sparse index, .crate tarballs and rustup dist. The index is revalidated on a TTL with conditional GETs (If-None-Match / If-Modified-Since); tarballs are immutable and cached permanently. config.json is rewritten so dl points at the proxy while api still reaches the real registry, so cargo publish/search keep working. Concurrent misses are collapsed by a per-path mutex.
• pkg/runtime — support bind-mounting proxy-generated files into job containers.

Why a mounted config file rather than environment variables

Cargo ignores CARGO_SOURCE_* and CARGO_REGISTRIES_* for source replacement. This was verified empirically against a dead endpoint rather than assumed: with the env vars set, cargo reported "Updating crates.io index" and went straight to the real registry; the same settings in a config file produced "Updating ephemerd index" and the expected connection error.

So the proxy generates a .cargo/config.toml and mounts it read-only at the container root, relying on Cargo's ancestor-directory config search. That applies to any checkout path, needs no knowledge of the workdir or the image's CARGO_HOME (left writable), and a repository's own .cargo/config.toml still wins. No workflow changes required. rustup takes RUSTUP_DIST_SERVER, where an env var does work.

Had this been built on the env-var assumption, the proxy would have been silently bypassed — the same class of failure as the Go proxy above.

The mount destination is /.cargo on Linux and C:\.cargo on Windows, both of which are the filesystem root Cargo's ancestor search terminates at. Linux jobs on Windows and macOS hosts run inside the Linux VM sidecar, which runs its own ephemerd and generates its own config, so the host GOOS is the right default for the container OS.

---

Shutdown: the test hang

TestCrate_ConcurrentMissesFetchUpstreamOnce failed intermittently in CI, always at exactly 5.01s, with:

--- FAIL: TestCrate_ConcurrentMissesFetchUpstreamOnce (5.01s)
    cargoproxy_test.go:136: Stop: shutting down cargo proxy: context deadline exceeded

Instrumenting Stop with a ConnState hook and a full goroutine dump caught it: no handler was running, and exactly one connection was stuck in ConnState "new".

That is Go's http.Transport dialing speculatively. Under a burst of parallel requests it opens more connections than it ends up using; the spares land in the client's idle pool having never sent a byte. Server-side they are "new" and stay that way. http.Server.Shutdown refuses to close a "new" connection until it has been in that state for over five seconds (net/http's issue-22682 heuristic), so Shutdown polls for the full five seconds and the caller's five-second context expires first. Nothing leaked and nothing hung forever — the shutdown simply could not succeed. Both proxies had the same Stop, so both were affected.

pkg/proxies/server.go now owns the server lifecycle for every cache proxy:

• Unread connections are closed up front. A connection that has not sent a request has nothing to drain, so shutdown finishes in microseconds instead of waiting out a grace period meant for a different problem.
• BaseContext ties every request context to the proxy. This fixes the failure the net/http heuristic exists for: when the grace window does close on a genuinely stuck handler, cancelling the proxy context unblocks its in-flight upstream fetch before the connections are forced shut. Without it, Shutdown neither cancels request contexts nor gives up on its own.
• The accept goroutine is joined, so nothing outlives Stop.
• Shutdown always returns, and returns an error only if the forced close itself failed. Having to force is logged, not returned: the server is stopped either way, and a slow drain is not a failure to stop.

Upstream fetches now run under the proxy's context rather than the requesting job's. A fetch fills a shared cache that other requests for the same path are queued behind, so one cargo hanging up — or its container being killed — must not abort the download everybody else is waiting for. It stays bounded by the client timeout, and Stop can still cancel it.

The regression tests are deterministic rather than hopeful: the unread connection is dialed directly instead of hoping the transport races the right way (TestShutdown_NotDelayedByUnreadConnections), and TestStop_CancelsInFlightUpstreamFetch pins that a request wedged on a dead upstream costs the grace window, not the 120 s client timeout.

---

Fail-open: Cargo has no |direct

This was the design blocker, and it is the reason the proxy is shaped the way it is.

GOPROXY="<url>|direct" falls through to the origin on any proxy error, so the Go proxy can only ever slow a build down. Cargo has no equivalent. Once [source.crates-io] replace-with points at the proxy there is no second source, no fallback and no retry-elsewhere — a dead proxy is a red build for every Rust job on the node. A cache must never be a hard dependency of the job path.

Three layers, cheapest first:

1. Not started → not injected

A proxy that fails to Start is never added to the cache-proxy list, so neither the mount nor the env var reaches any container. Jobs behave as if ephemerd had no cache. Covers boot-time failures (port in use, unwritable cache dir).

2. Running → always answers

No route returns 5xx. Previously the index and config.json routes returned 502 when upstream was unreachable with nothing cached, on the reasoning that "cargo treats this as fatal either way" — which was only true because we had told cargo to replace crates-io with us. They now 307 to the real origin, like the crate and rustup routes already did. config.json matters most: it is the first thing Cargo asks for, and redirecting it hands the job the genuine document whose dl points at the crates.io CDN, so the rest of that build simply bypasses the proxy.

So "the proxy accepts connections" is the only thing a build depends on. Upstream outages, an unreadable cache, a full disk and an unparseable config.json all degrade to an uncached direct fetch. Genuine 404s still pass through as 404s — "no such crate" is not an outage. Malformed request paths are still refused with 400 rather than turned into a redirect built out of attacker-supplied bytes; real Cargo never emits them.

3. Running but wedged → withdraws itself

Layer 2 assumes requests still reach a handler. A listener can wedge while the daemon lives on — that has happened on this fleet, in the self-upgrade path — and layer 2 cannot help, because the job never reaches a handler.

So a watchdog probes the proxy's own listener over real TCP every 15 s and, after two consecutive failures, rewrites the mounted config.toml to an inert one with no [source] table at all. A directory is bind-mounted, so containers already running see the swap: their next cargo invocation goes straight to crates.io. It is restored automatically once probes succeed again. Both transitions are logged (cargo cache withdrawn from jobs / cargo cache re-enabled for jobs).

The probe deliberately targets the bound address, not the advertised one. On Linux the bridge gateway does not exist until CNI creates it with the first job container, so probing the advertised address would report every freshly booted daemon as dead — precisely when it is fine. A wildcard binding is probed on loopback.

This subsumes the "health-gate at injection time" option: the gate lives in the one place that needs no cross-package plumbing, and unlike an injection-time check it also covers a proxy that dies after the job started.

Residual failure mode

Stated honestly, in the package comment and in docs/getting-started/configuration.md:

A cargo build that has already read the active config and is mid-resolve when the listener wedges will fail — nothing can retract a config file Cargo has already parsed. The exposure is a single Cargo invocation, bounded by the watchdog interval plus Cargo's own retries (net.retry = 3 in the generated config). Eliminating it would mean not using source replacement, which would mean not caching crates at all. If ephemerd's whole process freezes the watchdog freezes with it — but then nothing is scheduling jobs either.

needsHostAccess

needsHostAccess() gained cfg.CargoProxy.Enabled. Main's L2Bridge ACL ladder blocks the host from containers unless something ephemerd serves needs reaching, and it only knew about dind and the Go proxy. On a Windows L2Bridge node with only the cargo proxy enabled, every Rust job would have failed on an unreachable proxy with no fallback.

Config

[cargo_proxy]: enabled (false), port (8083), upstream (https://index.crates.io), rustup_upstream (https://static.rust-lang.org), index_ttl (10m), cleanup (false).

cleanup defaults false, deliberately unlike [module_proxy] — wiping a pull-through cache on every restart is the bug diagnosed above. Registered in managedCaches() as LiveSafe: true; the generated container config lives outside the cache root so cache clear cargo cannot pull a mounted file out from under a running job.

Merge surface

Rebased onto current main (L2Bridge, native-macOS removal, dind host-port scoping) — merged clean, no conflicts. Two other proxies (npm/pip/pub) and a registry mirror are in flight on other branches and touch the same two files, so the footprint there is deliberately tiny:

• cmd/ephemerd/main.go: the [cargo_proxy] start block, the port added to gatewayPorts, cacheProxyMounts collected alongside cacheProxyEnvVars, and one term added to needsHostAccess.
• pkg/config/config.go: one CargoProxyConfig struct plus its CleanupEnabled(), and ModuleProxyConfig.Cleanup changed to *bool (which is what fixes the dead cleanup knob).

Everything else lives in pkg/proxies/.

Testing

go build ./..., GOOS=linux go build ./... and GOOS=windows go build ./... clean; go vet ./... clean; golangci-lint run reports 0 issues.

go test ./pkg/proxies/... -race passes, including 15 consecutive runs of the previously flaky package — the shutdown regression reproduced in roughly one run in three before the fix and zero times after. ~60 tests against an httptest fake upstream, no network access.

Not yet run on a node. The Go-proxy fix in particular is worth confirming live, since the symptom (an empty 4 KB directory) is easy to mistake for "working but unused".

Follow-ups found, not fixed

• Manager.GatewayIP() hardcodes 10.88.0.1 when Subnet is empty, while Config.subnet() → pickSubnet() may auto-select a different 10.x/16 on conflict. On such a host the two disagree.
• The GatewayPorts ACCEPT rules land in EPHEMERD-FORWARD, jumped from FORWARD, but container→gateway traffic hits INPUT — so those rules are currently no-ops. Harmless today (reachability comes from the default INPUT policy), but the carve-out is not doing what it appears to.

sccache

docs/arch/sccache-evaluation.md. Recommends building it, sequenced after the disk-pressure GC, with a hard size cap wired into that GC from day one. Local-disk backend, partitioned per repo, and scoped to cargo build rather than docker build so it stays disjoint from the BuildKit layer cache instead of storing the same output twice.