test(odb): establish dynamic store correctness and performance baselines - #2853
Draft
Sebastian Thiel (Byron) wants to merge 18 commits into
Draft
test(odb): establish dynamic store correctness and performance baselines#2853Sebastian Thiel (Byron) wants to merge 18 commits into
Sebastian Thiel (Byron) wants to merge 18 commits into
Conversation
Provide a gix-testtools-generated SHA-1/SHA-256 ODB catalogue and a directly included mutation helper for publishing and removing pack components, rewriting MIDX files, changing alternates, and creating maintenance-like intermediate states. This gives correctness tests, benchmarks, and fuzzing one reproducible vocabulary for the upcoming refresh, loading-race, and dynamic-slot work.
Exercise multiple handles in deterministic single-threaded sequences while packs, MIDX files, and alternates change, and cover malformed-index recovery plus slot over-commitment. These checks establish the correctness baseline needed before removing yields, fixing load and refresh races, or changing slot allocation.
Measure cold and warm packed lookups, repeated misses, prefix disambiguation, startup scanning, and concurrent access at and beyond available parallelism, with Refresh::Never comparisons. This makes unnecessary refreshes, fixed-slot over-commitment, and future dynamic-growth changes measurable against the same fixture used by correctness tests.
Sebastian Thiel (Byron)
changed the title
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
from
July 30, 2026 07:32
d127a44 to
591c867
Compare
Add a deterministic single-threaded opcode VM that interleaves operations across multiple handles with component-wise pack publication and removal, MIDX replacement, alternates, malformed indices, and slot pressure. Seeded maintenance sequences cover interrupted and reordered transitions so the upcoming correctness changes can be fuzzed without baking in a particular failure.
A handle that encountered a malformed pack index marked the file missing and exhausted its load cursor. Replacing that index atomically under the same path left the existing handle unable to see its objects even after a strict refresh, while a fresh store succeeded. Keep failed indices suppressed while their recorded modification time is unchanged, but replace the slot state and reset index-loading progress once the file changes. This retries new contents without repeatedly opening the same known-bad file and preserves the immutable fast path for successfully loaded indices.
With a single slot, replacing a standalone index with a MIDX could reuse the same numerical slot while changing its contents. Consolidation assumed a generation change implied a different slot vector, panicked, and still left the reused slot scheduled for deletion. Treat generation as part of index identity and remove a reused destination from the disposal list. This reflects what generation tracking is for: stale handles must reject the old contents even when the slot number stays the same, while the newly assigned slot must remain reachable.
Return a stable outcome from every VM opcode so a corpus artifact describes not only the requested operations, but also whether they applied, were rejected, were skipped, or produced a particular query result. Treat missing or malformed MIDX inputs as the one modeled mutation rejection and fail on other fixture mutation errors instead of silently changing the effective scenario. Add opt-in per-op tracing so replays expose the exact decoded sequence and outcomes without slowing normal fuzzing.
Treat opening the controlled fixture as infallible and make header/object or iterator/find disagreement fail when the ODB is valid. During component-wise maintenance, report the expected stale-index outcome explicitly instead. Classify lookup failures by subsystem so a replay distinguishes refresh pressure, loose traversal or decoding, and packed-object failures without depending on platform-specific error strings.
Let live handles write deterministic loose blobs, query them through find, header, and prefix APIs, and remove them from disk before repeating those queries. This covers the ordinary mixed loose/packed ODB state that precedes and overlaps Git maintenance instead of limiting the VM to prebuilt packs. Keep the model single-threaded and track only the generated object IDs; the filesystem remains the source of truth for visibility across independent and cloned handles.
Drive prefix disambiguation, ordinary lookup, candidate collection, a guaranteed missing full ID, and the fixture catalogue’s known cross-pack collision as distinct deterministic modes. Validate the candidate set against each lookup result so the VM checks more than the outer status. Exercise both object iteration orderings and add a seed that makes every mode reachable with two colliding packs installed, preventing corpus luck from deciding whether these API paths receive coverage.
A one-slot store with pack A loaded could see pack B and an A-only MIDX, overwrite the slot for A while consolidating, and only then report that B required another slot. The failed refresh left the handle unable to read A even though it had been usable before the refresh. Reject disk states whose final index count exceeds slot capacity before assigning any file to a slot. Capacity failure is therefore atomic: it still reports InsufficientSlots, while the last successfully loaded state and its objects remain usable.
Let the VM remember a packed object through Exists, which loads its index without opening pack data. If a later operation reports a refresh failure while that object remains reachable in a valid ODB, require the same handle to read it successfully. This makes failed-refresh atomicity an immediate, deterministic oracle instead of relying on a later iterator or random lookup to expose partially replaced slot state.
Cloned handles share slot state while keeping independent snapshots, but the VM checkpoint previously refreshed and compared only the addressed handle. A refresh through one handle could therefore disturb a peer without an immediate deterministic check. Checkpoint every strict handle with the same store identity against one fresh reference, and extend failed-refresh preservation to those peers. Readable MIDX, maintenance-removal, and alternate seeds keep these single-threaded interleavings in the corpus.
The VM copied a source handle observation into a clone even though gix_odb::Handle::clone() collects a new handle-local snapshot. After the source read a pack and the index was removed, a clone could refresh without ever observing that object itself, causing the failed-refresh oracle to report a false preservation failure. Start cloned handles without an observed object. They must establish their own observation through Find, Header, or Exists before the preservation oracle applies, matching the state tracked by the real handle.
The VM treated prevent_pack_unload() and RefreshMode::Never as one checkpoint skip, leaving their distinct contracts untested. Maintenance could invalidate a stored pack location, or a never-refresh query could rescan disk, without producing a deterministic failure. Track real pack locations acquired by stable handles and verify them after every interleaved operation. For never-refresh handle operations, allow only the store initialisation scan and assert through store metrics that no later disk reconciliation occurs. Mixed-policy seeds exercise both contracts through pack removal, MIDX replacement, and peer-driven refreshes.
…stion A never-refresh handle with too few slots could fail its lazy initial scan, remain uninitialized, and scan the same pack directory again on every subsequent miss. The deterministic VM reproduced this with two complete indices and a one-slot store. Publish the loose-database portion of the initial snapshot before reporting slot exhaustion. This records that initialization was attempted without partially assigning pack slots: never-refresh handles stop rescanning, while strict handles can still reconcile after the on-disk state fits. The regression test covers both outcomes through shared handles.
A loaded index can outlive its pack while maintenance removes or replaces files. This leaves pack loading, shared missing-state propagation, and later recovery distinct from the already-covered case where pack data was mapped before removal. Add a readable VM sequence and an ordinary scenario test that load only the index, remove the pack, query it through two never-refresh handles, and restore it through a strict shared handle. The assertions preserve deterministic failure and recovery semantics without adding another opcode or production counter.
A handle can retain a loaded MIDX while maintenance removes both its packs and the MIDX. A later iterator may still yield those snapshot IDs even though object lookup correctly reports them missing. The VM treated this as a failure once the filesystem returned to a valid empty ODB. Classify an unreadable iterated ID as stale when it is no longer reachable from the generated disk state, while retaining the assertion for every still-reachable ID. A readable MIDX-removal seed keeps this distinction reproducible without weakening the correctness oracle.
Sebastian Thiel (Byron)
force-pushed
the
odb-parallelism
branch
from
July 30, 2026 10:00
591c867 to
d090f99
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tasks
Codex inserts after this line
Purpose
This PR establishes the correctness and performance baseline for upcoming work on the multi-handle, multi-threaded dynamic object database. It intentionally changes test infrastructure rather than production ODB behavior: the aim is to make filesystem transitions, refresh behavior, slot pressure, and handle interactions reproducible before changing the implementation.
The infrastructure is intended to support this sequence of follow-up work:
Refresh::Never.What this adds
Shared ODB scenarios
A
gix-testtoolsfixture script builds a deterministic catalog of valid SHA-1 and SHA-256 packs and indices. A utility module undergix-odb/tests/copies and mutates that catalog for tests, benchmarks, and fuzzing without introducing another support crate.The shared vocabulary can:
This keeps all three kinds of coverage grounded in the same well-known repositories and object IDs.
Deterministic correctness coverage
The integration tests exercise multiple live handles in deliberately interleaved, single-threaded sequences while:
The existing threaded packed-object smoke test also now always uses at least two workers and verifies that every enumerated object is actually found.
Benchmarks
Criterion benchmarks provide baselines for:
Refresh::Never;These should make regressions and improvements in refresh suppression, prefix lookup, startup behavior, and slot over-commitment visible.
Deterministic fuzz VM
A single-threaded bytecode VM interleaves operations across several handles with disk mutations. It covers lookup, header access, iteration, prefix queries, refresh modes, pack-component transitions, MIDX transitions, alternates, malformed indices, and slot pressure.
The seed corpus includes standalone packs, MIDX use, alternates, malformed recovery, slot pressure, pack replacement, MIDX expiration, and interrupted maintenance. Maintenance states can expose markers or new components before old packs are removed, remove packs while a MIDX still references them, and abandon or reverse transitions.
The VM checks returned object hashes, header/data agreement, fresh-store equivalence at valid checkpoints, and existing handle metrics. Store statistics can be used or extended further where they help characterize later correctness or performance changes.
Verification
cargo test -p gix-odb --all-features --lib --test odbcargo clippy -p gix-odb --all-features --tests --benches -- -D warnings