Implement retention namespace generation transitions

[flyingrobots]

Goal

Implement the accepted keep.segment-store/v2 retention and one-way migration campaign tracked by #19 without weakening Keep's exact-byte, fail-closed storage law.

The affected invariant is that retention authority must be generation-checked, canonical, deterministic, bounded, and durable before any reader or writer may treat a version-2 state as admitted. This branch preserves every version-1 segment, catalog, and publication-head byte.

Plan

• Freeze the version-2 format, golden corpus, typed coordinates, and canonical retention and migration records.
• Implement deterministic closure verification, generation transition planning, publication preparation, ordered storage ports, and receipts.
• Implement bounded migration inventory, migration intent and receipt construction, verification-first execution, and filesystem migration authority observation.
• Complete production filesystem retention publication and exact idempotent retry behavior.
• Complete restart recovery, immutable reader fencing, and model-based namespace transition evidence.
• Complete migration filesystem integration, partial-prefix recovery, corruption coverage, and the process-death matrix.

Current status

This is an early Draft PR. The format and storage-independent laws are substantially implemented, but the requirements ledger still marks the production filesystem, restart, reader-fence, model, and crash-injection slices as in progress or planned. Version 1 therefore remains the only admitted production store.

The current branch contains 50 commits spanning the #19 campaign. The latest repair separates repository-task root identity observation from the strict production ext4 admission probes. This restores the hosted process-death crash matrix without weakening production admission, and crash-matrix I/O diagnostics now preserve their exact source.

Failure modes

• Stale or mismatched generations refuse before mutation.
• Missing, corrupt, excessive, changed, substituted, or noncanonical closure and migration evidence has typed refusal boundaries.
• Short and trailing restart artifacts refuse at their exact transfer boundary.
• Storage failures stop at the named publication or migration phase; incomplete production filesystem and recovery slices remain explicit nonclaims.

Validation

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets --all-features -- -D warnings
• cargo test --workspace --all-features
• cargo xtask durability-crash-matrix
• cargo xtask source-structure-check
• cargo check --workspace --all-targets --all-features --target x86_64-unknown-linux-gnu
• Focused catalog restart streaming laws: 7 passed
• Focused recovery-stage materialization laws: 3 passed

Compatibility, recovery, performance, and security

• Format/API compatibility: version 2 preserves admitted version-1 immutable bytes; migration is one-way and does not yet authorize a production version-2 writer.
• Recovery: the durable formats and transition phases are specified, while filesystem crash recovery remains open and is not claimed by this Draft.
• Benchmark impact: no performance claim is made. The restart slice adds deterministic 64 MiB virtual-input harnesses proving bounded callback and writer state rather than publishing throughput numbers.
• Security: capability-relative, no-follow filesystem admission and repeated evidence revalidation are preserved. This work does not claim secure deletion or semantic/causal authority.

Alternatives rejected

• Inferring retention or migration completeness from path existence.
• Re-encoding version-1 immutable data in place.
• Treating storage-independent ports and golden bytes as production durability evidence.
• Whole-artifact allocation where bounded transfer is sufficient.

Open questions

Peer review is most useful on the remaining filesystem publication/restart ordering and the reader-fence aperture before those irreversible protocol slices are implemented.

Closes #19 only when every unchecked Plan item and the colocated requirements ledger are complete.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 6d800a78-8bba-47e7-8754-05c6925e4012

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}