perf(changes): derive prunable commit intervals in O(delta)

[ragnorc]

What & why

The per-commit change enumerator derived one commit's entity changes by a full ordered-by-id merge of BOTH pinned table versions for every changed interval — O(table extent), not O(delta). A one-row update on a large table read the whole table on both sides. This was honestly pinned as a GROWING cost tripwire; RFC-030 §4.2/§4.3 specified the sanctioned fix (row-version candidate pruning + a transaction no-delete proof) and §14 recorded it as deferred. This PR ships it: the common insert/update/no-delete commit is now derived in O(delta).

The design

Per changed interval, changes::candidate_scan proves whether the commit's effect is row-set-preserving — every Lance transaction in the interval is Append or a RewriteRows merge Update (an exhaustive, wildcard-free Operation match, so a new Lance variant compile-errors into review). When proven:

• Child scan is O(delta): scoped by Scanner::with_fragments to exactly the fragments the parent lacks (the child-minus-parent manifest diff — pure metadata, no data reads), with the _row_last_updated_at_version ∈ (begin, end] window dropping carried-over rows a fragment rewrite pulled along.
• Parent before-images by BTREE: each candidate is classified against a batched id IN (chunk) index probe of the parent, reusing the same typed rows_equal / emitted_image the full merge uses (so it inherits the recent Blob descriptor-collision fix).
• No delete pass needed: one transaction per commit + the D2 rule + no delete-capable merge arm ⇒ a prunable interval has zero logical deletes.

Any unproven op (delete, overwrite, restore, compaction, a branch/lineage change, a non-advancing/oversized interval, a missing/cleaned transaction) falls back to the exact ordered merge, which classifies deletes correctly. The EmitSource seam yields the same id-ordered Emit stream as the old next_emit, so the streaming / ContinuationKey / budgeting contract is unchanged.

Correctness floor (guarded)

The classifier accepts every Operation::Update as non-deleting because OmniGraph's merge_insert never sets a by-source delete arm. A forbidden_apis.rs source guard locks that (WhenNotMatchedBySource / when_not_matched_by_source must be absent from engine source), so introducing one forces the classifier to be re-gated first.

Backing RFC

• Implements the deferred §4.2/§4.3 optimization of the accepted RFC-030 (recorded as shipped in §14).

Local verification

• Correctness (identical to the full merge): the entire changes (38) and point_in_time (15) suites pass with pruning enabled — the candidate path produces the same results as the exact merge.
• Flatness: changes_cost.rs::changes_page_opens_and_data_reads_are_bounded_by_delta flipped from assert_grows to assert_flat (data reads 10→9 across the extent sweep vs the old 11→23), with a companion changes_page_unproven_op_scan_term_grows_with_table_extent keeping the fallback honestly pinned as growing. The fixture reconciles the id BTREE (the production steady state, where the parent probe is an index lookup).
• §9 L0 guard: commit_changes_falls_back_for_overwrite_that_removes_an_id — an overwrite that drops one id and changes another still surfaces the delete (pruning would miss it).
• Guards: classifier unit tests; the write-path no-delete-arm guard; forbidden_apis (18, updated .dataset() registrations for the new emitter), lance_surface_guards (32).
• cargo fmt --all --check and both clippy graphs (-D warnings) clean; full failpoint-superset workspace test.

Notes for reviewers

• Base is change-feed (feat(changes): commit entity diffs and a durable change feed #519) — this builds on the CDC enumerator + the Blob descriptor-collision fix. Retarget to main once feat(changes): commit entity diffs and a durable change feed #519 merges.
• The flat-cost claim is for a reconciled id BTREE. On a never-optimized graph the parent probe full-scans (correct, slower); the feed's realistic steady state is periodically optimized.
• Deferred: an inductive per-write row-set-preserving transaction-property certificate (a write-path change) — only needed if a delete-capable merge arm is ever introduced.

---

Note

Medium Risk
Touches CDC correctness for the change feed, but unproven intervals always use the exact merge and new pruning requires durable transaction proofs plus integration/cost guards.

Overview
RFC-030 §4.2/§4.3 candidate pruning replaces always doing a full ordered parent/child merge for every changed table interval. New changes::candidate_scan classifies each interval; when every Lance transaction in (begin, end] is Append or a proven non-deleting RewriteRows Update, enumeration scans only new child fragments (manifest diff), filters by _row_last_updated_at_version, and classifies rows with batched id IN (chunk) parent probes—same Emit stream and pagination as before via EmitSource.

Safety gates: RewriteRows updates prune only with omnigraph.no_by_source_delete (stamped at the keyed merge_insert chokepoint) or insert_absence; marker-less external merges and removing ops (delete, overwrite, etc.) fall back to the exact merge. Fragment row-version metadata must be loadable or the interval falls back. OrderedRows::open_scan adds fragment scope and composed filters for this path.

Tests/docs: cost tripwire flips to flat for pruned upserts and stays growing for deletes/overwrites; overwrite delete visibility; forbidden_apis blocks by-source delete merge arms; RFC-030 §14 updated as shipped.

^{Reviewed by Cursor Bugbot for commit 5c87662. Bugbot is set up for automated code reviews on this repo. Configure here.}

Greptile Summary

The PR adds an O(delta) change-feed derivation path for proven insert/update-only transactions while retaining the exact full-table merge as the fallback.

• Scans only child fragments introduced by the interval and filters candidates using row-version metadata.
• Probes parent rows in bounded id chunks to classify inserts and updates and preserve before-images.
• Persists a no-by-source-delete transaction marker for keyed writes and adds correctness, source-guard, and cost-scaling coverage.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible follow-up scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
crates/omnigraph/src/changes/candidate_scan.rs	Introduces transaction qualification, changed-fragment selection, parent-image probing, and the pruned ordered emitter with conservative fallback gates.
crates/omnigraph/src/changes/enumerate.rs	Selects the pruned or full-merge emitter per interval while preserving the existing image, budget, and continuation pipeline.
crates/omnigraph/src/changes/row_compare.rs	Extends ordered scans with structured predicates and optional fragment scoping used by candidate scans and parent probes.
crates/omnigraph/src/table_store.rs	Adds and persists the keyed-write provenance marker consumed by the pruning classifier.
crates/omnigraph/tests/changes.rs	Adds a regression proving overwrite operations retain full-merge delete detection.
crates/omnigraph/tests/changes_cost.rs	Pins the qualified update path as flat in table extent and preserves a growing-cost tripwire for fallback operations.
crates/omnigraph/tests/forbidden_apis.rs	Guards the invariant that engine keyed merges contain no delete-capable by-source arm.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Changed table interval] --> B{Identity, branch, versions, metadata, and transactions proven safe?}
    B -- No --> C[Exact ordered parent-child merge]
    B -- Yes --> D[Compute child-minus-parent fragments]
    D --> E[Scan rows updated within version window]
    E --> F[Probe parent by batched IDs]
    F --> G[Classify insert or update]
    C --> H[Ordered Emit stream]
    G --> H
    H --> I[Existing budgeting and continuation logic]

Loading

_{Reviews (7): Last reviewed commit: "docs(changes): audit the persisted marke..." | Re-trigger Greptile}

Context used:

• Knowledge Base — OmniGraph core storage: manifest, table store, and Lance access
• Knowledge Base — OmniGraph write path

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a592fe228

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ragnorc]

Rebased onto the updated change-feed.

The base branch replaced the rows_equal scope hint (BlobComparisonScope) with an unconditional identity qualifier: managed-Blob descriptors are now keyed by the owning data file's immutable path (a globally-unique per-file UUID), which is authoritative across lineage without a same-branch/cross-branch distinction. The candidate-scan parent probe therefore calls the scope-free rows_equal directly, and the interim "adapt … scope" commit (which existed only to pass SameLineage to the old signature) is dropped as obsolete. build_change_feed_cut is already registered in the surface guard on the base.

No behavior change to the pruning logic. Verified against the rebased base: engine builds, and changes (42), changes_cost (6, including the pruned-FLAT / fallback-GROWING gates), and forbidden_apis (18) all pass.