fix: retry raced Lance branch enumeration

[azimafroozeh]

What & why

Lance enumerates named branches by listing _refs/branches/ and then reading every listed ref. If a branch is deleted between those reads, the otherwise read-only enumeration can fail with NotFound.

This change retries only that read-only enumeration, with a fixed limit of three total attempts. It does not turn a storage error into permission to replay the enclosing operation, and it does not introduce a new HTTP or OpenAPI error contract. Every production Dataset::list_branches() call now goes through the same helper.

Backing issue / RFC

• Fixes an accepted issue: Closes bug: operations racing branch_delete fail with an unclassified "Not found" branch-ref error #515
• Is an RFC PR, or implements a merged RFC
• Trivial fast-lane

The retry boundary is aligned with RFC-038: the engine retries the known effect-free read locally instead of exposing a generic whole-operation retry signal.

Checklist

• Change is focused (one logical change)
• Tests added/updated for behavior changes
• Public docs updated if user-facing surface changed, or N/A — no public surface change
• Reviewed against docs/dev/invariants.md

Local verification

• cargo fmt --all --check — passed
• scripts/check-agents-md.sh — passed
• cargo test -p omnigraph-engine --locked --lib branch_control::tests -- --nocapture — 10 passed
• cargo test -p omnigraph-engine --locked --test forbidden_apis — 18 passed
• cargo clippy -p omnigraph-engine --all-targets --locked -- -D warnings -W clippy::dbg_macro — passed

Notes for reviewers

The regression test uses Lance's file-object-store:// provider and an object-store wrapper to physically delete a ref after it has been listed but before Lance reads it. It proves that the next attempt performs a fresh listing and succeeds. A second test holds the ref in a repeated NotFound state and proves the exact three-attempt bound and final substrate error.

There is intentionally no 409 mapping or instruction for callers to retry an effectful operation. Repeated churn escapes through the ordinary storage-error path for the owning operation to classify.

Greptile Summary

The PR centralizes Lance branch enumeration behind a bounded, read-only retry that handles refs deleted between listing and reading.

• Retries NotFound enumeration failures up to three total attempts.
• Routes manifest, recovery, indexing, and native table-branch enumeration through the helper.
• Adds race and exhaustion regression tests plus a structural guard preventing raw production calls outside the helper.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph/src/branch_control.rs	Adds the bounded branch-enumeration retry boundary and focused tests for a vanished ref and exact retry exhaustion.
crates/omnigraph/src/db/manifest.rs	Routes manifest branch listing, descendant discovery, and delete preflight through the centralized helper.
crates/omnigraph/src/db/manifest/recovery.rs	Routes recovery-time branch observations through the retry helper while preserving existing gates and identity checks.
crates/omnigraph/src/storage_layer.rs	Renames the sealed storage method to distinguish native Lance branches from graph-level branches.
crates/omnigraph/src/table_store.rs	Implements native branch enumeration through the centralized retry helper.
crates/omnigraph/tests/forbidden_apis.rs	Updates renamed gateway inventories and structurally enforces a single raw Dataset::list_branches production call site.

Sequence Diagram

sequenceDiagram
    participant Caller
    participant Helper as list_branch_contents
    participant Lance
    participant Store as Object store
    Caller->>Helper: enumerate branches
    loop At most three attempts
        Helper->>Lance: Dataset::list_branches()
        Lance->>Store: list _refs/branches/
        Store-->>Lance: listed refs
        Lance->>Store: read each ref
        alt Ref disappears after listing
            Store-->>Lance: NotFound
            Lance-->>Helper: NotFound
            Helper->>Helper: yield and retry
        else Enumeration succeeds
            Store-->>Lance: branch contents
            Lance-->>Helper: branch map
            Helper-->>Caller: branch map
        else Other error or final NotFound
            Lance-->>Helper: substrate error
            Helper-->>Caller: OmniError::Lance
        end
    end

Loading

_{Reviews (3): Last reviewed commit: "test(branch): guard enumeration retry bo..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Branching and merge
• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[azimafroozeh]

Closes #515.