write_chunkindex_to_repo: reduce memory needs

[ThomasWaldmann]

Fixes #9886.

A full index rewrite used to build one sorted Python list of all keys (about 90 bytes per key, e.g. about 9 GB for a 100M chunks repo). As the keys are uniformly distributed hash digests, this partitions them by their leading key bits into 2 ** prefix_bits similarly sized, disjoint sets and selects / sorts / writes one partition's keys at a time, so the extra memory needed is bounded by one partition (about one fragment, at most 32 MB worth of keys), independent of the index size.

• Selecting a partition is a cheap, C-level filtering scan of the in-memory hash table — needs items(): support iterating over a key-prefix partition borghash#50, so this requires borghash ~= 0.2.0 (CI will stay red on the dependency until 0.2.0 is released).
• prefix_bits is chosen from the selected-entry count, aiming about 5% below CHUNKINDEX_FRAGMENT_ENTRIES_MAX. That headroom matters because rounding the partition count up to a power of two would otherwise be able to put the expected partition size right at MAX. With hash digests as keys the partition sizes are tightly concentrated around their expectation, so every partition lands well below MAX and becomes exactly one fragment: at 5M entries and MAX=400k, the 16 partitions measure 311,896 … 313,950 entries, i.e. >100 sigma of margin.
• For ≤ MAX entries — every ordinary incremental backup — prefix_bits = 0, so there is a single partition, no filtering scans, and behavior is exactly as before.
• Determinism/convergence is preserved: partition membership and prefix_bits only depend on the selected entries, so identical entry sets still produce identical fragments; and as the prefix compares the keys' leading bits, ascending prefixes yield the same globally sorted key sequence as one all-keys sort did.
• ChunkIndex gets a new_count property (count of F_NEW entries, maintained incrementally; computed lazily for tables loaded from a file), so the incremental path can pick prefix_bits without a counting pass. Bonus: a nothing-new incremental write now skips the full table scan entirely.

Measured (5M uniformly distributed entries, full rewrite, macOS vmmap physical footprint peak): peak extra memory of the write 480 MB → 85 MB, runtime unchanged (21.2 s vs 21.4 s incl. serialization + store).

New tests: real partitioning with uniformly distributed keys (bounded fragments, disjoint + complete + contiguous ranges of the sorted key space, rebuild round-trip, incremental and full), convergence with shuffled insertion order, new_count maintenance (insert/overwrite/delete/clear_new/clear/failed insert/file round-trip) and prefix-filtered iteritems.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.70%. Comparing base (eae8f6d) to head (4f861a3).
⚠️ Report is 114 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #10078   +/-   ##
=======================================
  Coverage   86.69%   86.70%           
=======================================
  Files          98       98           
  Lines       17166    17174    +8     
  Branches     2607     2609    +2     
=======================================
+ Hits        14882    14890    +8     
  Misses       1586     1586           
  Partials      698      698           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.