diff --git a/README.md b/README.md index 3fbb1ab..13642a7 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,8 @@ Redirect Git traffic through cachew: insteadOf = https://github.com/ ``` -Restore a repository from a snapshot (with automatic delta bundle to reach HEAD): +Restore a repository from a snapshot (with automatic delta bundle to reach HEAD); see +[docs/git-restore.md](docs/git-restore.md) for how the restore flow and parallel snapshot downloads work: ```sh cachew git restore https://github.com/org/repo ./repo @@ -109,7 +110,9 @@ Multiple backends can be configured simultaneously — they are automatically co are ordered from lowest/nearest to highest/authoritative. Reads check each tier in order and backfill lower tiers on a hit. Writes go to all tiers in parallel. Replica invalidations evict only non-authoritative tiers; the final cache block is authoritative. Tiered caches use the metadata backend to track authoritative ETags and invalidate stale lower-tier -copies before falling through to the authoritative tier. +copies before falling through to the authoritative tier. See [docs/tiering.md](docs/tiering.md) for a full explanation +of the tiering semantics, and [docs/architecture.md](docs/architecture.md) for how requests flow through strategies to +the cache. ### Memory diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..06fe4ef --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,62 @@ +# Architecture: the request path + +How an HTTP request flows through Cachew, from client to upstream. + +``` +┌────────┐ ┌──────┐ ┌──────────────┐ ┌──────────────────┐ ┌───────┐ +│ Client │──▶│ OPA │──▶│ Interceptors │──▶│ Strategy (route) │──▶│ Cache │ +└────────┘ └──────┘ └──────────────┘ └────────┬─────────┘ └───────┘ + │ miss + ▼ + ┌──────────┐ + │ Upstream │ + └──────────┘ +``` + +- **OPA** authorizes every request (`internal/opa`). +- **Strategies** are protocol-aware handlers (Git, GitHub releases, Go + modules, Hermit, Artifactory, host, proxy). Each is registered against the + mux at config load (`internal/config/config.go`); strategies implementing + `strategy.Interceptor` wrap the mux instead, so they can inspect the raw + request line. +- Each strategy receives a **namespaced view** of the cache + (`cache.Namespace`), so strategies never collide on keys. +- Most strategies use the shared handler (`internal/strategy/handler`), which + implements the cache-or-fetch loop: look up the key, serve from cache on a + hit, and on a miss stream the upstream response to the client and the cache + simultaneously. Strategies are not limited to the Cache — e.g. the Git + strategy also maintains bare clones and serves packs directly. +- The **api-v1 strategy** (`internal/strategy/apiv1.go`) exposes the composed + cache itself over HTTP (`/api/v1/object/{namespace}/{key}`, …). This is the + API that `client/` and the `Remote` cache implementation speak, and it is + what makes instance-to-instance tiering possible. + +Strategies see a single `Cache`; whether it is one backend or several tiers +composed together is invisible to them. That composition is described in +[tiering.md](tiering.md). + +## Ranged and parallel downloads + +`client.ParallelGet` (`client/parallel_get.go`) downloads a large object as +many concurrent ETag-pinned byte-range requests instead of one stream. Its +main consumer is `cachew git restore`'s snapshot download — see +[git-restore.md](git-restore.md) for the full semantics. The same machinery is +reused inside the S3 backend (`internal/cache/s3_parallel_get.go`): a single +S3 stream is limited to a fraction of the available bandwidth, so whole-object +and large ranged reads fan out into parallel sub-range requests against the +pinned object revision. + +Ranged reads interact with cache tiering — a partial body must never be +backfilled into a lower tier as if it were the whole object; see +[tiering.md](tiering.md). + +## The Cache contract + +Strategies and cache backends all program against the same interface +(`internal/cache/api.go`), whose key guarantees are: + +- Expired objects are never returned. +- Objects are invisible until completely written and closed. +- `Delete` is atomic; missing objects invalidate successfully. +- Conditional options (`If-None-Match`, `If-Match`, `If-Range`, `Range`) are + evaluated against the stored ETag with RFC 9110 semantics. diff --git a/docs/git-restore.md b/docs/git-restore.md new file mode 100644 index 0000000..f069a64 --- /dev/null +++ b/docs/git-restore.md @@ -0,0 +1,53 @@ +# Git restore: snapshots, bundles, and parallel downloads + +`cachew git restore ` recreates a working tree from the +server's cached artifacts instead of running `git clone`. The flow +(`cmd/cachew/git.go`): + +1. **Snapshot download** — fetch `/git/{repo}/snapshot.tar.zst`, a periodic + `.tar.zst` archive of the working tree, and pipe it straight into + extraction so decompression overlaps the transfer. When the target + directory doesn't exist yet, extraction goes into a staging directory + renamed into place on success, so a failed download never leaves a + half-written checkout. Restoring over an existing checkout extracts + directly into it (a rename can't replace a non-empty directory), so a + failure there can leave partially extracted files behind. +2. **Delta bundle** — the snapshot response advertises a bundle URL + (`X-Cachew-Bundle-Url`) covering commits from the snapshot's commit to the + mirror's current HEAD. The client fetches and applies it to catch up + without talking to the upstream. A bundle failure is a warning, not an + error — the restore continues with the snapshot state. +3. **Freshen** (only with `--ref`/`--commit`) — ask the server to ensure the + required refs/commits exist on its mirror, then `git pull --ff-only` from + origin so the working tree catches up. Skipped entirely when the + snapshot+bundle already contain everything requested. + +## Parallel snapshot download + +The snapshot is downloaded as many concurrent byte-range requests +(`client.ParallelGet`, driven by `--download-concurrency` and +`--download-chunk-size-mb`): + +- **Discovery**: the first chunk is requested with a `Range` header; the + response reveals the object's total size and ETag, plus the snapshot + metadata headers (commit, bundle URL). The git strategy advertises these on + both full (200) and ranged (206) responses, so the client learns everything + it needs from this one response (`internal/strategy/git/snapshot.go`). +- **Pinning**: every subsequent chunk carries `If-Match` with the discovery + ETag, so an object rewritten mid-download (e.g. by the periodic snapshotter) + is rejected rather than spliced together from two revisions. +- **Streaming**: chunks complete out of order and are reassembled into a + sequential stream with bounded buffering, feeding `tar` extraction as bytes + arrive. + +## Fallbacks and failure modes + +Single-stream fallback happens at discovery time only: a server that ignores +the range, an object with no ETag to pin to, an object that fits in the first +chunk, or `--download-concurrency=1` all degrade to one full read. + +After discovery, a mid-download rewrite is **fatal**: pinned chunks fail their +`If-Match` precondition and the download errors rather than delivering a +corrupt archive. The client does not retry single-stream and does not fall +back to `git clone` — retrying the restore (or cloning) is the caller's +decision. Bundle failures, by contrast, are non-fatal as described above. diff --git a/docs/tiering.md b/docs/tiering.md new file mode 100644 index 0000000..ca48395 --- /dev/null +++ b/docs/tiering.md @@ -0,0 +1,91 @@ +# How cache tiering works + +How multiple cache backends are composed into a single tiered cache with +fallback, backfill, and invalidation semantics. For how a request reaches the +cache in the first place, see [architecture.md](architecture.md). Source of +truth: `internal/cache/tiered.go` and `internal/cache/api.go`. + +Cache backends are object stores with per-object TTLs and metadata. Three are +configurable: `memory`, `disk`, and `s3`. Configuring more than one `cache` +block composes them into a `Tiered` cache automatically +(`cache.MaybeNewTiered`); with a single block the cache is wrapped so that +`Invalidate` is a no-op, since the only tier is authoritative. + +**Order matters.** Cache blocks are ordered nearest-first. The **final tier is +authoritative** — typically shared storage like S3 — and everything before it +is treated as a local copy that can be re-fetched. + +``` +cache memory { } # tier 0: nearest, fastest +cache disk { } # tier 1 +cache s3 { ... } # tier 2: authoritative +``` + +``` + read: probe in order, first hit wins + ────────────────────────────────────▶ +┌────────┐ ┌──────┐ ┌────────────────┐ +│ memory │ │ disk │ │ s3 (authorit.) │ +└────────┘ └──────┘ └────────────────┘ + ◀──────────────────────────────────── + backfill tier 0 on a deeper hit + + write: all tiers in parallel +``` + +## Reads + +`Open`/`Stat` probe tiers in order and return the first definitive answer. +When a deeper tier hits, the returned reader transparently **backfills tier +0** as the caller reads (`backfillReadCloser`), so the next read is served +locally. Backfill is asynchronous and safe: only a stream consumed to EOF +commits the tier-0 entry; a partial read or mid-stream error discards it. + +Conditional requests complicate "definitive". A tier holding a *different +version* than the request's validators name (failed `If-Match`, `If-Range` +miss) is not a definitive miss — deeper tiers are consulted for the named +version. Only when no tier holds it does the first tier's outcome stand. A +tier that errored while being probed takes precedence, so outages are not +misreported as missing versions. + +Ranged reads return partial bodies, which must never be backfilled as whole +objects. Instead a bounded background **healer** re-fetches the full object +from the serving tier and refreshes tier 0 out of band, so a divergent tier 0 +still converges even when clients only ever issue ranged requests — as the +parallel snapshot downloader does (see [architecture.md](architecture.md)). + +## Writes + +`Create` writes to **all tiers in parallel** through a single writer; the +first error aborts every in-flight write. Entries only become readable once +completely written and closed — a cancelled context discards the object in +every tier. + +## ETags and stale-tier invalidation + +When a write **replaces** an existing object in the authoritative tier with +different content, the tiered cache records the new authoritative ETag in the +metadata store (the `metadata` block; see [metadatadb-s3.md](metadatadb-s3.md)). +On reads, a tier whose ETag doesn't match the recorded authoritative ETag is +**invalidated and skipped**, falling through to the next tier. This is how +replicas still holding the previous version converge onto the shared tier +after a rewrite. Keys that have never been rewritten have no recorded ETag and +are served from whichever tier hits first. + +## Delete vs Invalidate + +- `Delete` removes the object from **every** tier. +- `Invalidate` evicts stale copies from the **non-authoritative** tiers only; + the final tier is left intact by construction. This is what replicas use to + drop local copies without destroying shared state. + +## Instance-to-instance tiering + +Cachew is designed to run as a local instance (workstation/CI) backed by a +shared remote instance. The pieces: + +- The remote instance serves its cache via the **api-v1 strategy** (see + [architecture.md](architecture.md)). +- `client/` is a standalone Go client for that API, and + `internal/cache/remote.go` adapts it to the `Cache` interface — a remote + Cachew instance as a cache tier.