feat(scalar-index): push scan limit into index search

[gstamatakis95]

Closes #6949

Problem

A scalar index query that matches many rows can be slow. For example, a large range query against a B-tree index that hits most or all of the data. When the scan has a LIMIT, the index still builds the full set of matching rows before the limit is applied later in the plan.

This change passes the limit down into the index search. A B-tree can then stop early once it has found enough matches.

What changed

Index layer (lance-index)

• Added a new trait method ScalarIndex::search_limited(query, metrics, limit). It has a default that ignores the limit and calls search. Because of the default, existing index types and call sites do not need any changes.
• BTreeIndex implements the new method. It searches matching pages in order and stops as soon as it has gathered at least limit real matches. The pages it has not reached yet are never read. Null page handling is skipped when a limit is set.
• ScalarIndexExpr::evaluate_limited passes the limit only to a single index lookup. For AND, OR, and NOT it does not pass the limit, because those need the full result of each side.

Exec and planner (lance)

• MaterializeIndexExec (legacy path) and ScalarIndexExec (default path) now accept a limit through with_limit(...).
• Scanner::index_search_limit() computes the value to push, which is limit + offset. It is wired into both scalar_indexed_scan and new_filtered_read.
• LogicalScalarIndex passes the limit to each of its segments.

Correctness

Pushing the limit is only a speed optimization. A GlobalLimitExec still applies the exact limit and offset at the top of the plan, so the index only needs to return at least limit + offset rows.

The pushdown is turned off in any case where returning the first N matches is not the same as returning any N matches:

• An ORDER BY, vector search, or full text search reorders rows before the limit.
• An aggregate is present, so the limit applies after aggregation.
• A refine filter or an inexact (recheck) result could drop matched rows later in the plan.
• Any relevant fragment has deletions. Deleted rows are removed after the index search, so stopping early could leave fewer than limit live rows.

Tests

• test_search_limited_short_circuits (B-tree unit test). Confirms the search returns at least limit rows but reads fewer than all pages.
• test_limit_pushed_into_scalar_index (scanner, end to end). Checks the exact row count and that every returned row matches the filter. A second phase adds deletions and confirms results stay correct.
• Re-ran the B-tree, scalar index exec, inexact plan, secondary index (legacy and modern storage), all test_limit tests, and scalar_logical tests. All pass. cargo clippy --tests -- -D warnings and cargo fmt are clean.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[codecov]

Codecov Report

❌ Patch coverage is 97.11538% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/expression.rs	86.66%	0 Missing and 2 partials ⚠️
rust/lance/src/dataset/scanner.rs	97.97%	2 Missing ⚠️
rust/lance-index/src/scalar/btree.rs	98.63%	0 Missing and 1 partial ⚠️
rust/lance/src/io/exec/scalar_index.rs	93.75%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!