Skip to content

perf(files_external): opt-in same-endpoint S3 MultipartCopy - #62673

Open
TobyTheHutt wants to merge 3 commits into
nextcloud:masterfrom
TobyTheHutt:perf/files_external-s3-server-side-copy
Open

perf(files_external): opt-in same-endpoint S3 MultipartCopy#62673
TobyTheHutt wants to merge 3 commits into
nextcloud:masterfrom
TobyTheHutt:perf/files_external-s3-server-side-copy

Conversation

@TobyTheHutt

Copy link
Copy Markdown

Fixes #62645.

Assisted-by: ClaudeCode:claude-opus-4-7

Summary

Add opt-in server-side MultipartCopy dispatch on the AmazonS3 external-storage backend. Cross-mount MOVE and COPY between two mounts on the same S3 endpoint stop streaming object bytes through the PHP host. Behaviour on upgrade remains identical to master until an admin enables the flag.

Enable:

occ config:app:set files_external amazons3_server_side_copy --value=true --type=boolean

Behavioural matrix on upgrade

Configuration Flag Behaviour
Any existing S3 mount false (default) Identical to pre-upgrade. No fast-path attempts. No new log entries. No extra S3 API calls.
Same-account mounts with s3:GetObject on source true Cross-mount MOVE and COPY complete without materialising bytes on the Nextcloud host. Wall-clock scales with S3-to-S3 bandwidth.
Cross-account without source-bucket s3:GetObject true Fast path attempts. S3Exception(403) per top-level op. One WARNING per op. Extra S3 round-trips. Streamed fallback completes. Failure limit caps log flood at four per request per endpoint.
Provider without UploadPartCopy (Storj, GCS XML) true Same shape as the previous row. Fallback completes.

Required IAM on the destination-mount credential

Source bucket:

  • s3:GetObject
  • s3:GetObjectVersion for versioned sources
  • kms:Decrypt for SSE-KMS sources

Destination bucket:

  • s3:PutObject
  • s3:AbortMultipartUpload
  • s3:ListBucketMultipartUploads
  • kms:GenerateDataKey for SSE-KMS destinations

Guards that force the streamed parent path

  • Feature flag off (default).
  • Source wrapped in Encryption.
  • Terminal source after wrapper unwrap is not AmazonS3.
  • Endpoint identity mismatch on hostname, port, TLS mode, region, or access-key ID.
  • Caller passes $preserveMtime === true.
  • Per-endpoint failure limit tripped in the current request.

Checklist

  • Code is properly formatted (php-cs-fixer clean via pre-commit).
  • Sign-off message added to all commits.
  • Tests: cross-mount unit and integration coverage in apps/files_external/tests/Storage/. Gated by run_cross_mount => true in config.amazons3.php, provisioned by the updated env/start-amazons3-ceph.sh.
  • Screenshots. Not applicable. Server-only change, no UI.
  • Documentation. Not applicable. Single opt-in AppConfig key. Description surfaces via occ config:app:list files_external.
  • Backports. Not applicable. New feature targeting master.
  • Labels. To be added by a maintainer.
  • Milestone. To be added by a maintainer.

AI (if applicable)

  • The content of this PR was partly or fully generated using AI. Design, implementation, adversarial code review, and local MinIO SDK-level validation performed with Claude Opus 4.7 via Claude Code CLI. Every iteration was reviewed by the human contributor.

Fixes nextcloud#62645.

Assisted-by: ClaudeCode:claude-opus-4-7
Signed-off-by: Tobias Harnickell <tobias.harnickell@bedag.ch>
@TobyTheHutt
TobyTheHutt requested a review from a team as a code owner July 29, 2026 14:53
@TobyTheHutt
TobyTheHutt requested review from leftybournes, nfebe, salmart-dev and sorbaugh and removed request for a team July 29, 2026 14:53
Assisted-by: ClaudeCode:claude-opus-4-7
Signed-off-by: Tobias Harnickell <tobias.harnickell@bedag.ch>
@CarlSchwan
CarlSchwan requested a review from icewind1991 July 30, 2026 09:56
@CarlSchwan

Copy link
Copy Markdown
Member

It's a lot of new code 😬

@TobyTheHutt

Copy link
Copy Markdown
Author

I know, sorry for that. Most of the diff is the S3 multipart protocol (Initiate, UploadPartCopy × N, Complete) plus its invariants (SSE-C rewrap, orphan-MPU cleanup, wrapper transparency). I had a hard time to trim it further without dropping invariants and risking silent data or key leaks.

The tests account for about half the diff. If it helps the review, I could split them off into a spearate PR.

…blocks

Assisted-by: ClaudeCode:claude-opus-4-7
Signed-off-by: Tobias Harnickell <tobias.harnickell@bedag.ch>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

files_external amazons3: cross-mount MOVE should use server-side MultipartCopy when peer is same S3 endpoint

2 participants