feat(storage): classify typed storage failures

[ragnorc]

Summary

Implements the accepted RFC-038 from #523.

• replaces exhaustive OmniError::Lance(String) with exhaustive OmniError::Storage(StorageFailure)
• exposes the closed StorageFailureKind taxonomy without adding generic retry/replay semantics
• classifies object-store, local storage I/O, Lance 10, and all Lance namespace error codes from typed evidence only
• preserves the complete historical display text for genuine adapter and Lance storage failures
• corrects Arrow/internal, DataFusion/user, and persisted Blob-integrity category boundaries
• keeps commit conflict replay decisions at their existing operation-local proof boundaries
• keeps generic storage failures at HTTP 500 and leaves OpenAPI unchanged

This is the clean replacement for #491. It starts from current main, preserves the crash-safe initialization work and current Blob/manifest implementations, and does not port the obsolete Blob code or conflict blocks from that PR.

Implementation notes

• source traversal is capped at eight typed links; opaque or over-depth sources classify as Unknown
• is_transient() means exactly kind == Transient
• named Lance conversion helpers force each engine call site to choose storage, domain, or internal semantics
• ExternalBlobPolicy and ExternalBlobSource remain separate user-supplied dependency boundaries
• source guards reject blanket From<lance::Error>, From<ArrowError>, and From<DataFusionError> implementations
• user error documentation and unreleased v0.10 notes document the Rust API break and intentional category corrections

Validation

• cargo test -p omnigraph-storage --locked
• focused engine error, staged table-store, lifecycle, Lance surface-guard, source-guard, server mapping, and OpenAPI suites
• RUST_MIN_STACK=16777216 cargo test --workspace --locked --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints
• default and failpoint-superset workspace Clippy with -D warnings -W clippy::dbg_macro
• cargo test -p omnigraph-server --locked --features aws
• cargo fmt --all --check
• git diff --check
• no OpenAPI file diff

The stack override is required for the full macOS debug suite: without it, branch_merge_with_blob_columns_preserves_blob_data overflows the default test-thread stack on both the accepted main baseline and this branch. The isolated test and the complete workspace suite pass with the larger stack.

Supersedes #491.

---

Note

High Risk
Touches the universal storage and Lance error path across engine, manifest publish/recovery, and HTTP/CLI surfaces; misclassification could change retry behavior or user-visible error categories despite message-preservation goals.

Overview
Implements RFC-038 by replacing the string-only OmniError::Lance variant with OmniError::Storage(StorageFailure), where each failure carries a closed StorageFailureKind (transient, configuration, not found, precondition, permanent, unknown) plus a complete diagnostic message that stays the same for operators and HTTP 500 bodies.

omnigraph-storage adds bounded typed traversal (max eight source links) over std::io::Error and object_store::Error, maps adapter failures to StorageError::Backend / structured Io, and drops the dedicated CreateIfAbsentUnsupported variant in favor of classified I/O messages.

OmniError gains explicit converters (storage, storage_context, storage_namespace, datafusion, datafusion_internal, arrow_internal) so Lance, namespace, object-store, and I/O evidence classify without blanket From impls. RetryableCommitConflict stays separate from generic Lance precondition conflicts; manifest publish retry vocabulary remains row-level CAS only.

Call sites across manifest, recovery, table ops, merge/mutation/export, and blob paths switch from ad-hoc Lance strings to the helpers; many former “Lance” cases now correctly surface as manifest internal, blob integrity, or DataFusion user errors. Server and CLI map Storage via failure.to_string() (no storage: double-prefix); blob tests still hide physical URIs on internal paths.

Adds broad unit coverage for I/O/object-store/Lance/namespace matrices, CLI message preservation, and API error mapping tests.

^{Reviewed by Cursor Bugbot for commit 1f8a84a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

Implements RFC-038’s typed storage-failure model across the storage adapter, core engine, server, and CLI.

• Replaces string-based Lance errors with StorageFailure and a closed failure-kind taxonomy.
• Adds bounded typed-source classification for object-store, filesystem, Lance, and namespace errors.
• Migrates engine call sites to explicit storage, domain, and internal conversion helpers while preserving operation-local conflict handling.
• Keeps generic storage failures mapped to HTTP 500 and updates tests and documentation for the Rust API change.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph-storage/src/lib.rs	Introduces the typed storage taxonomy, bounded source-chain classification, and classified adapter error variants while retaining typed success sentinels for absence and precondition outcomes.
crates/omnigraph/src/error.rs	Replaces string Lance errors with typed storage failures and centralizes explicit Lance, DataFusion, Arrow, namespace, and context conversions.
crates/omnigraph/src/table_store.rs	Migrates staged commits and engine-owned scans to explicit conversion modes while retaining the exact effect-free conflict boundary.
crates/omnigraph/src/db/manifest/publisher.rs	Updates manifest publishing to preserve operation-local conflict and recovery semantics under the typed storage model.
crates/omnigraph-server/src/lib.rs	Maps typed storage failures to HTTP 500 without duplicating the complete storage diagnostic.
docs/rfcs/0038-typed-storage-failures.md	Documents the accepted taxonomy, evidence boundaries, and prohibition on inferring generic retry safety.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    E[Typed backend or engine error] --> C{Classification boundary}
    C -->|object_store or std::io| S[StorageFailureKind]
    C -->|Lance storage or namespace evidence| S
    C -->|DataFusion user error| D[DataFusion category]
    C -->|Arrow or engine invariant| I[Internal domain category]
    S --> O[OmniError::Storage]
    D --> Q[OmniError::DataFusion]
    I --> M[Manifest or Blob internal error]
    O --> H[HTTP 500 with complete diagnostic]
    Q --> B[HTTP 400 query error]
    M --> H

Loading

_{Reviews (3): Last reviewed commit: "fix(storage): preserve internal failure ..." | Re-trigger Greptile}

Context used (3)

• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access
• Knowledge Base — OmniGraph write path
• Knowledge Base — Query execution engine

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a8a4a57. Configure here.}