Specifying a limit for total data transfer

[chamilton-ccn]

Why?

LTO-9 tape can store up to 45TB of data (compressed). When using rsync to copy multiple terabytes of data to LTFS-formatted tape, it is sometimes advantageous to limit the amount of data copied in any single transfer to allow the drive and tape to cool down before resuming. Doing so can significantly increase the lifespan of the drive. This can be achieved by creating separate backup "jobs" and inserting a pause between each, but that can become unmaintainable as the amount of data requring backup increases. The ability to specify a maximum amount of data to be transferred in total is an efficient way to achieve this.

While the primary motivation for this feature is to limit the total amount of data copied to a tape in a single run, to avoid having the drive running for days on end, its implementation is general, not specific to LTO/LTFS/etc. This feature can be used in any scenario where someone might need to transfer data up to a specific limit.

---

Parameters

• --data-transfer-limit Specifies a maximum amount of data to transfer; whole-file only! Delta-sync transfer size is not taken into consideration. If the next file to be transferred would exceed the specified limit, rsync will exit with code 26. Example: If you have 20 x 10GiB files to copy, and you specify a limit of 90GiB (--data-transfer-limit=90GiB), only 9 files will be copied (90GiB).
• --delay-transfer-limit-check Tells rsync to check the transfer size after transferring files. Example: If you have 20 x 10GiB files to copy, and you specify a limit of 95GiB (--data-transfer-limit=95GiB), 10 files will be copied (100GiB). This option allows rsync to continue transferring data so long as the limit has not yet been exceeded.

---

Testing

See:

• testsuite/transfer-limit_test.py
• testsuite/delay-transfer-limit-check_test.py

This modification is also currently being used in production.

[steadytao]

This is a useful feature request but I think it needs a careful design pass before it is merge-ready. It adds global transfer-limit state, a new multiplex message, new exit-code behaviour and file-list/generator control-flow changes so the important questions are protocol compatibility, sender/receiver agreement and what exactly counts as transferred data.

A few things I would want clarified before deeper review:

• Is MSG_DATA_LIMIT_REACHED safe with older protocol peers or does it need protocol-version/capability gating?
• Should the limit count source file length, actual bytes written, matched-data bytes or only whole-file transfer candidates?
• What should happen with hard-linked files where a later path may not transfer data but still creates an entry?
• Should delete phases still run after the transfer limit is reached?
• The //CEH3 markers and generated configure churn should probably be cleaned up before review.

I would suggest adding a short protocol/semantics note to the PR description and possibly splitting the generated-file refresh from the functional change.

[chamilton-ccn]

@steadytao Thank you! I'm looking into all of this and I'll reply back when I have a more thorough response.

The //CEH3 markers and generated configure churn should probably be cleaned up before review.

Yeah -- I did remove those in a later commit; sorry about that! I'll squash all of this in a subsequent commit.

Thanks again!