feat(server): redirect blob GETs to presigned object-store URLs#6
Merged
Conversation
v2GetBlob proxies every blob byte through the epoch process: it reads from the object store and io.Copy's the body back to the client, paying TLS on both sides for multi-GiB VM disk/memory blobs. A single stream tops out at ~20 MB/s even with 3 CPUs, so a 10.5 GB snapshot pull takes ~9 min, and concurrent transfers saturate the pod. When EPOCH_BLOB_REDIRECT is set, v2GetBlob now responds with a 307 to a presigned object-store URL (minio-go PresignedGetObject), so blob bytes flow client<->storage directly and epoch leaves the data path entirely. GCS honors the SigV4-signed URL and serves Range requests natively. Measured on a same-region GCE VM pulling the 5.7 GB memory-ranges blob: path single-stream 10.5 GB pull proxy (1 CPU, before) 3.4 MB/s ~51 min proxy (3 CPU) 20 MB/s ~9 min presigned direct-to-GCS 275 MB/s ~38 s ~13x over the proxy with zero client changes — any redirect-following OCI client (registryclient/vk-cocoon, oras, crane, docker) benefits transparently, and CopyBlobExact still verifies the digest end to end. Intra-blob Range parallelism (measured 1.28 GB/s at 8 connections) would need a client-side downloader and is out of scope here. The flag defaults off. On any server-side failure (existence check or presign) the handler falls back to streaming. Note there is no fallback once the 307 is sent: every client hitting /v2/blobs must have egress to the object-store host when the flag is enabled.
Turn on EPOCH_BLOB_REDIRECT so pulls bypass the proxy. Bump resources from 1 CPU / 512Mi to 3 CPU / 4Gi: pushes still stream through epoch (this PR only redirects GETs) and the redirect fallback path also proxies, so the pod still needs headroom for multi-GiB transfers.
Pin the merged cocoon-common main (cross-node migration API surface, PR #3) via pseudo-version until v0.2.2 is tagged.
CMGS
force-pushed
the
feat/blob-redirect-presigned-url
branch
3 times, most recently
from
e7c6072 to
efbd7a8
Compare
- clamp EPOCH_BLOB_REDIRECT_TTL to [1s, 7d]: an out-of-range value fails every presign and silently degrades to proxying, so fall back to the default below 1s (the minio presign floor) and clamp above 7d - read the redirect env via commonk8s.EnvBool / EnvDuration, dropping the hand-rolled strconv/ParseDuration to match the package convention - mark the blobRedirectTTL config field; note why redirectBlob HEADs before presigning; document the EPOCH_BLOB_REDIRECT_TTL knob
CMGS
force-pushed
the
feat/blob-redirect-presigned-url
branch
from
efbd7a8 to
4ebffbd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
v2GetBlobproxies every blob byte through the epoch pod: it reads from the object store (GCS) andio.Copys the body back to the client, paying TLS on both sides for multi-GiB VM disk/memory blobs. Consequences:connection refused).Change
When
EPOCH_BLOB_REDIRECTis set,v2GetBlobresponds with a 307 to a presigned object-store URL (minio-goPresignedGetObject) instead of proxying. Blob bytes then flow client ↔ storage directly and epoch leaves the data path entirely. GCS honors the SigV4-signed URL and servesRangerequests natively.registryclient/vk-cocoon,oras,crane,docker— benefits transparently.CopyBlobExactstill verifies the digest end to end. Go stripsAuthorizationon the cross-host redirect, so the registry token is never leaked to GCS./v2/.../blobs/{digest}GET only./dl/(server-side reassembly of multi-part cloud images) is untouched — it can't be a simple redirect and still proxies.Benefit
Single-blob throughput from a same-region GCE VM:
~13× over the (already bumped) proxy, zero client changes. 8-connection intra-blob Range parallelism measured 1.28 GB/s — but that needs a client-side ranged downloader and is intentionally out of scope here.
Verified live (deployed to cocoonstack-us, image
redirect-stream-20260618)GET /v2/.../blobs/<digest>now returns307→…storage.googleapis.comin ~30 ms (server out of the byte path); following it serves200/206straight from GCS.cocoonsnapshot pull (win11, ~9.6 GiB) ran with every layer GET as a307; end-to-end ~2 min, of which the GCS download is only ~35 s — the remainder iscocoon snapshot importwriting to local disk (network is no longer the bottleneck).Safety / rollout
EPOCH_BLOB_REDIRECTunset). The deploy manifest turns it on./v2/blobsmust reach the object-store host (storage.googleapis.com). Verified reachable from US + SG cocoonset nodes before enabling. Confirm node egress before flipping the flag in a new environment.Test
resolveBlobRedirectTTLunit test (default / parse / invalid / non-positive).206for ranged GETs; deployed and confirmed307→GCS on cocoonstack-us.