feat(optimize): surface vector index layout so coverage answers can-it-prune

[memmmmike]

Resolves #486.

Coverage currently answers "does an index exist over these rows" but not "can it prune" — the exact blindspot from the #432 incident, where pending_indexes: [] reported a healthy index while every nearest() flat-read ~10 GB.

What this adds (reporting only; the repair remains #485):

• TableOptimizeStats.vector_index_layouts: per built vector index — name, partition summary, indexed rows, worst rows-per-partition across delta segments, and a degenerate flag.
• The flag is computed per segment: Lance searches delta segments independently, so a well-partitioned later segment cannot repair a mono-partition earlier one. It fires on rows-per-partition (> 4096, generous slack over ~sqrt(N) healthy sizing), which catches both the mono-partition case and the incident's own 8-partitions-over-852k shape — partition count alone would miss the latter.
• CLI: layouts print under each table; degenerate ones loudly, with the worst segment's rows/partition. --json carries the full struct.
• Statistics missing the expected fields are a loud error, not a silent skip — a silent skip would recreate the exact failure mode this issue is about. lance_surface_guards now pins num_indexed_rows and num_indexed_rows_per_delta (aligned with indices[]) beside the existing num_partitions pin, so a Lance upgrade that changes the shape fails the guard instead of degrading the stats.

Scope checked: skipped_for_drift intentionally reports no layout (that HEAD is not manifest-accepted); TableOptimizeStats is not serialized by the server, so no OpenAPI drift.

Tests: classifier unit tests including the incident shape and the bad-segment-behind-good-segment case; an end-to-end maintenance test asserting layouts surface through optimize() on a real vector index; the guard pins. Maintenance suite 34/34. CLI smoke on a local graph verified both text and --json output, and a no-vector-column graph stays clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01E4S9Xpe9G7pLEv9JaoXUAb

Greptile Summary

The PR exposes physical vector-index layouts through optimize statistics and CLI output so operators can distinguish index coverage from effective pruning.

• Parses Lance vector-index statistics into per-segment partition and row counts.
• Classifies layouts using the worst rows-per-partition segment and surfaces a degeneracy flag.
• Adds text and JSON reporting plus unit, integration, and Lance-surface guard coverage.

Confidence Score: 4/5

The PR appears safe to merge after the non-blocking optimize documentation mismatch is corrected.

The changed snapshot flow refreshes layout statistics after index effects, and no current runtime or compatibility defect was established; the remaining issue is that the documented optimize output omits the newly emitted layout fields.

Files Needing Attention: crates/omnigraph-cli/src/main.rs and docs/user/operations/maintenance.md

Important Files Changed

Filename	Overview
crates/omnigraph/src/table_store.rs	Parses Lance vector-index statistics into aligned per-segment partition and row counts, failing loudly on missing or malformed fields.
crates/omnigraph/src/db/omnigraph/table_ops.rs	Defines the public layout report and computes worst-segment rows per partition and degeneracy.
crates/omnigraph/src/db/omnigraph/optimize.rs	Carries vector-index layouts through no-work and effectful optimize result paths using the appropriate dataset snapshot.
crates/omnigraph-cli/src/main.rs	Adds human and JSON layout reporting, but the corresponding user-facing maintenance documentation was not updated.
crates/omnigraph/tests/maintenance.rs	Adds an end-to-end assertion that optimize surfaces the layout of a real vector index.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Lance index_statistics] --> B[TableStore parser]
    B --> C[Per-segment partitions and rows]
    C --> D[VectorIndexLayout classifier]
    D --> E[TableOptimizeStats]
    E --> F[CLI text output]
    E --> G[CLI JSON output]

Loading

_{Reviews (1): Last reviewed commit: "feat(optimize): surface vector index lay..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Context used - CLAUDE.md (source)

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[greptile-apps]

Document vector layout output

optimize --json now emits vector_index_layouts, and the human output reports healthy and degenerate layouts, but the maintenance documentation still enumerates the old field set. Update docs/user/operations/maintenance.md in this PR so operators and JSON consumers can discover the new layout, partition, and degeneracy fields.

Context Used: CLAUDE.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!