fix: retry transient source-read and segment-copy failures in UploadPartCopy - #133
Merged
Conversation
…artCopy A single truncated frame GET (backend drops a long-lived connection mid-body: ClientPayloadError 'received 8331264 of 8388636 bytes') or a 200-with-embedded-InternalError UploadPartCopy failed the whole client part. botocore's retry layer only covers a request up to the response headers, so body-read failures had no retry at all. During the 2026-07-16 Scylla backup this failed 12/13 hosts' upload jobs. - read_source_bytes(): full-body ranged GET with retry on transient transport/backend errors, used by frame reads and single-object reads - _stream_raw_source_with_resume(): raw copy-source streaming resumes with a ranged GET at the exact byte offset already yielded - passthrough copy_segment: retry upload_part_copy on retryable errors (same part number + range, safe to re-issue) - new env knobs: S3PROXY_SOURCE_READ_ATTEMPTS (4), S3PROXY_SOURCE_READ_BACKOFF (0.5s)
ServerSideHannes
added a commit
that referenced
this pull request
Jul 28, 2026
RGW can accept a part, commit to a 200, stream it, and only then put the
failure in the body ("InternalError: The server did not respond in time.",
status code: 200). Three retry layers all miss that shape because each decides
from the HTTP status line, which already read 200:
- rclone's LowLevelRetries gates on status 429/500/503 (backend/s3/s3.go)
- rclone's string fallback matches only transport phrases ("use of closed
network connection", "unexpected EOF reading trailer"), not InternalError
- botocore's max_attempts in S3Client sees the same 200
#133 added this retry to UploadPartCopy and #138 to CompleteMultipartUpload,
but plain UploadPart was left bare -- and it is the only one of the three a
Scylla backup uses (267 PUT, 0 UploadPartCopy measured against the backup
bucket). At 50MB rclone chunks a 6.2GB SSTable is ~762 internal PUTs, so one
unretried transient failure loses the whole multi-GB file.
2026-07-28 prod: 10 of 13 racks failed, ~2.4TiB never uploaded, every failure
on main.companies (47 of 48 large files).
Reuses the existing base.SOURCE_READ_ATTEMPTS machinery from #133, so no new
tunables. Safe at both call sites: the full ciphertext is still in memory when
upload_part is called (the del happens after), so a retry re-PUTs identical
bytes to the same internal part number.
ServerSideHannes
added a commit
that referenced
this pull request
Jul 28, 2026
RGW can accept a part, commit to a 200, stream it, and only then put the
failure in the body ("InternalError: The server did not respond in time.",
status code: 200). Three retry layers all miss that shape because each decides
from the HTTP status line, which already read 200:
- rclone's LowLevelRetries gates on status 429/500/503 (backend/s3/s3.go)
- rclone's string fallback matches only transport phrases ("use of closed
network connection", "unexpected EOF reading trailer"), not InternalError
- botocore's max_attempts in S3Client sees the same 200
#133 added this retry to UploadPartCopy and #138 to CompleteMultipartUpload,
but plain UploadPart was left bare -- and it is the only one of the three a
Scylla backup uses (267 PUT, 0 UploadPartCopy measured against the backup
bucket). At 50MB rclone chunks a 6.2GB SSTable is ~762 internal PUTs, so one
unretried transient failure loses the whole multi-GB file.
2026-07-28 prod: 10 of 13 racks failed, ~2.4TiB never uploaded, every failure
on main.companies (47 of 48 large files).
Reuses the existing base.SOURCE_READ_ATTEMPTS machinery from #133, so no new
tunables. Safe at both call sites: the full ciphertext is still in memory when
upload_part is called (the del happens after), so a retry re-PUTs identical
bytes to the same internal part number.
ServerSideHannes
added a commit
that referenced
this pull request
Jul 29, 2026
273 of 288 fatal copy failures in the 2026-07-29 capture were HTTP 504
GatewayTimeout on UploadPartCopy; another 17 on PutObject and 7 on GetObject.
The segment loop in copy.py already retries via is_retryable_source_error, but
504/GatewayTimeout was absent from _RETRYABLE_S3_ERROR_CODES -- which had 500,
502 and 503 but not 504 -- so it hit `raise` on first occurrence. That is why
only 4 UPLOAD_PART_COPY_SEGMENT_RETRY events were logged against 297 x 504.
Reproduced against the real object: 572 UploadPartCopy calls at 8-way
concurrency yielded 1 x 504 (0.17%), latency p50 1.14s / p90 3.31s / max 61.86s.
Latency is heavy-tailed, so a 504 is transient congestion when Hetzner's own
server-side copy exceeds an internal deadline -- not a permanent condition.
botocore does retry 504 (its _retry.json has a gateway_timeout policy) but all
its attempts fire inside the same congestion window and exhaust ("reached max
retries: 3" x297). Retrying here with SOURCE_READ_BACKOFF_SEC exponential
backoff gives the backend time to clear. UploadPartCopy is idempotent for a
given PartNumber+range, so the retry is safe -- the same argument #133 used.
At 0.17% per copy a 4.7GB file (572 copies) has a ~63% chance of hitting a 504,
and 10 nodes doing this concurrently approaches certainty, which is why
main.companies failed on 10 of 13 racks while smaller tables passed.
Also fixes MockS3Response.__aenter__ fidelity: aiobotocore's StreamingBody
returns the wrapped ClientResponse (no size-aware read()), so the mock returning
`self` let an `async with body: body.read(n)` TypeError pass the whole suite
while failing in production. Reintroducing that bug now fails 24 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During the 2026-07-16 Scylla backup, 12/13 hosts' upload jobs failed. Root cause chain:
ClientPayloadError: received 8331264 of 8388636 bytes(deterministic offset, ~4 min in), or an UploadPartCopy streams a 200 then embedsInternalError: The server did not respond in time.Fix
read_source_bytes()inhandlers/base.py: ranged GET + full body read with retry (4 attempts, exponential backoff) on transient transport/backend errors. Used by_iter_multipart_plaintextframe reads and_download_encrypted_single._stream_raw_source_with_resume()inmultipart/copy.py: the raw copy-source stream resumes with a ranged GET at the exact byte offset already yielded — bytes are never re-fetched or duplicated.copy_segment: retriesupload_part_copyon retryable errors (same part number + range → idempotent).is_retryable_source_error) covers aiohttp payload/connection errors, botocore timeout/streaming errors, and S3 error codes InternalError/SlowDown/RequestTimeout/ServiceUnavailable/BadGateway. NoSuchKey/AccessDenied and other client errors are never retried.S3PROXY_SOURCE_READ_ATTEMPTS(default 4),S3PROXY_SOURCE_READ_BACKOFF(default 0.5s).Tests
tests/unit/test_source_read_retry.py(8 tests): truncated-body retry & give-up, non-retryable passthrough, error classification, stream resume at exact offset (whole-object and ranged), non-retryable propagation, and an end-to-end passthrough copy that survives one 500'd segment copy + one truncated frame GET and round-trips the plaintext. Full unit suite: 585 passed.🤖 Generated with Claude Code