Apply KNN distance constraints in every index backend

[realgetOff]

Fixes #308. Diagnosis and reproduction numbers are in this comment.

The bug

vec0BestIndex accepts constraints on the distance column and unconditionally sets omit = 1:

pIdxInfo->aConstraintUsage[i].argvIndex = argvIndex++;
pIdxInfo->aConstraintUsage[i].omit = 1;

That tells SQLite the vtab enforces the constraint itself, so SQLite drops the WHERE term from the query plan and never re-checks it. But the constraint was only ever read back out of idxStr in vec0Filter_knn_chunks_iter (FLAT). rescore_knn, vec0Filter_knn_diskann and ivf_query_knn never looked at it, so on those columns the predicate silently vanished and violating rows came back with no error.

It's a regression rather than a design gap: #166 landed the distance constraints on 2026-02-13, and the ANN backends (#276, #277, #278) merged on 2026-03-31 without wiring them up.

Measured on 200-300 random float[8] rows, k = 20, threshold at the median observed distance:

backend	before	after
flat	0 violating rows	0 violating rows
rescore (bit)	20 / 20 violating	0
rescore (int8)	20 / 20 violating	0
diskann	9-11 / 20 violating	0
ivf	9-11 / 20 violating	0

DiskANN is the one that bites in practice, since it's on by default (SQLITE_VEC_ENABLE_DISKANN 1) while IVF is opt-in. AND distance < -1.0 also returned a full page of rows instead of none.

The change

One helper, vec0_distance_constraints_satisfied(), becomes the single point of truth for the predicate, so omit = 1 has exactly one place it can be honored and a future backend has an obvious thing to call:

• FLAT keeps its existing pre-filter semantics. Its four nested switch cases collapse into a single loop over the chunk bitmap that calls the helper — net -44 lines, and all 204 existing snapshots stay byte-identical.
• rescore filters the rescored float distances before the top-k truncation. The coarse quantized distances from phase 1 aren't comparable to a user-supplied threshold, so the filter can't be pushed down into the quantized scan; applying it after truncation would mean a lower-bound constraint chops off the head of the sorted results.
• DiskANN / IVF compact their final result sets.

It also fixes a latent bug on the rescore path that the new filter would otherwise have exposed: when every candidate is filtered out, result_k becomes 0, sqlite3_malloc(0) returns NULL, and the existing if (!out_rowids || !out_distances) check would have reported that as SQLITE_NOMEM.

Caveat I'd rather state than hide

On DiskANN and IVF, a lower-bound constraint (distance >, >=) can return fewer than k rows, because the filter runs over the final result set rather than the candidate pool. Matching FLAT's semantics there means reaching into diskann_search / ivf_query_knn to widen the search, which felt like scope creep for a correctness fix. The tests encode this distinction explicitly (INDEX_DEFS_FILLING_K) rather than glossing over it — happy to go further if you'd prefer uniform semantics.

Verification

New parametrized tests in tests/test-knn-distance-constraints.py cover all four operators across flat / rescore-bit / rescore-int8 / diskann, plus ivf behind a build-flag skipif mirroring tests/conftest.py.

• 17 of the new assertions fail on main (22 with IVF enabled), all pass with this change
• Full suite: 262 passed, 0 failed, 204 snapshots unchanged
• Clean under ASan + UBSan across all four backends and both empty-result edge cases

Conflict check: this shouldn't collide with #313 (IVF internals in sqlite-vec-ivf.c; I only touch the dispatch site in sqlite-vec.c) or #311 (DiskANN quantization, not filtering).

One unrelated snag for the record: vendor/ isn't in the repo or a submodule, so make loadable and make cli don't work from a fresh clone — that's #291, and it's the only thing that stood between me and a one-command build.