Add distributed index segment build APIs

[u70b3]

Summary

Implements PR 1 from #55: the foundational E1 + E2 primitives for distributed index builds.

• adds snapshot-owned, single-use scalar and vector segment builders
• builds over explicit fragment subsets without committing a dataset manifest
• supports caller-assigned segment UUIDs where Lance core permits them
• trains reusable shared IVF centroids and residual PQ codebooks
• injects shared models through the Arrow C Data Interface
• serializes and parses protobuf IndexMetadata with C getters
• adds move-only C++ RAII wrappers for builders, models, metadata, and returned bytes

Design notes

• Builds on Lance 9.1.0-beta.3 (e934cc2c), the core baseline used by RFC: C-level primitives for distributed index builds (fragment-scoped uncommitted builds, segment metadata, progress, cancellation) #55; the pin upgrade and its error-constructor compatibility migration landed separately in chore(deps): bump lance to 9.1.0-beta.3 #58.
• Uses a fixed-width mode field (AUTO, LOCAL_TRAIN, PRECOMPUTED) instead of a boolean so zero-initialized C options have unambiguous defaults.
• Model inputs are synchronously borrowed and restored, allowing one trained IVF/PQ model to build multiple disjoint segments.
• Trainer outputs carry metric, dimension, PQ parameters, and IVF identity provenance; builders reject mismatched models.
• Arrow schema/array trees are validated before import, including malformed UTF-8, child/buffer structure, slices, NULL values, and overflow-prone dimensions.
• Segment metadata bytes are allocated with malloc and released with lance_free_bytes.

Impact

C and C++ workers can now train a shared model once, build physical index segments over disjoint fragment assignments, and ship protobuf metadata to a coordinator without changing the dataset manifest. Commit-existing-segments remains the next PR in the tracker.

Validation

• cargo check --all-targets --locked
• cargo clippy --all-targets --locked -- -D warnings
• cargo test --locked (268 C API tests, 3 model tests, 1 helper test)
• strict C11 and C++17 compilation with Clang/GCC and -Werror
• cargo test --locked --test compile_and_run_test -- --ignored --test-threads=1
  • real C and C++ dynamic-link runs
  • IVF training -> residual PQ training -> reuse the same models for two disjoint fragment segments

Tracks #55.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The worker-side uncommitted-build design is a sound fit for the core primitives, but shared PQ models must preserve the metric's quantization contract. For DOT distance, train the PQ codebook on raw vectors while retaining the IVF model only for identity/provenance; residual training remains appropriate for L2 and cosine.

[lance-gatekeeper]

DOT PQ is trained on residuals here even though Lance's DOT encoder/search path does not use residuals. Passing Some(&ivf) unconditionally makes build_pq_model_in_fragments subtract the nearest centroid; the resulting schema is marked DOT, then the segment builder accepts the codebook and quantizes raw DOT vectors with it, which can silently produce a mismatched index and degrade recall.

Please pass the IVF model only for L2/Cosine (matching core build_ivf_model_and_pq and PQBuildParams::use_residual) and pass None for DOT. The centroids can still be required for shared-model identity/provenance.

Reproducer

I ran cargo test --test gate_dot_pq -- --nocapture against this head with a focused integration test that builds 128 positive Float32 vectors, trains DOT IVF/PQ through these C entry points, and compares that codebook with build_pq_model_in_fragments(..., DistanceType::Dot, ..., None, None) on the same dataset. The regression assertion requires the C codebook to remain on the raw-vector scale: assert!(c_mean_abs >= core_mean_abs * 0.5). It fails with C mean_abs=23.066938 versus core mean_abs=101.07422; the C output is residual-centered.

[lance-gatekeeper]

Fixed in 3f04af3: DOT PQ training now uses raw vectors, while L2 and cosine retain IVF residual training; the focused regression test verifies the C codebook against the canonical core DOT trainer.\n\n

[lance-gatekeeper]

Fixed in 3f04af3: DOT PQ training now uses raw vectors, while L2 and cosine retain IVF residual training; the focused regression test verifies the C codebook against the canonical core DOT trainer.

[lance-gatekeeper]

✅ Gate recommendation: approve.

The DOT PQ path now matches the pinned core metric contract: DOT trains on raw vectors, while L2 and cosine retain IVF residual training. The focused regression test covers the prior failure, and the worker-side uncommitted-build design remains sound.

[jja725]

Could we ensure precomputed DOT PQ builds use a DOT quantizer here? Whole-dataset builds take core’s normal path, which creates the supplied codebook with DistanceType::L2. That leaves the stored codes L2-encoded while metadata/search use DOT, silently hurting recall.

[u70b3]

Fixed in 3d82c70. lance-c now fails fast for supplied DOT PQ codebooks when the fragment selection covers the full dataset, preventing the L2-encoded mismatch. A TODO and C API note document that complete support requires an upstream Lance fix.

[jja725]

Change LGTM, could you rebase to resolve conflicts

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The new supplied-DOT safeguard is inverted: full-dataset V3 builds already preserve the required L2 PQ-assignment contract, while strict fragment subsets take the distributed path that rewraps the codebook as DOT. Preserve full-dataset use and fail closed only for strict-subset supplied DOT PQ until the upstream distributed builder uses L2.

[lance-gatekeeper]

This guard is inverted: it rejects the safe full-dataset path and leaves the mismatched distributed path enabled. In pinned Lance e934cc2, implicit or explicit full coverage normalizes to build_vector_index, whose quantizer builder deliberately calls Q::build(..., DistanceType::L2, ...). A strict subset with supplied IVF instead enters build_distributed_vector_index; make_global_pq rewraps the same L2-trained codebook with DistanceType::Dot, and ProductQuantizer then uses DOT when assigning codes. That can silently change stored codes and reduce recall.

Please keep full-dataset supplied DOT PQ enabled and reject effective strict subsets in AUTO/PRECOMPUTED for both IVF-PQ variants until upstream make_global_pq reconstructs with L2. The header contract and regression expectations need the same inversion.

Reproducer

With this head checked out, I ran a bounded assignment reproducer matching Lance’s L2/DOT argmin:

node -e 'const v=[1.1,0], c=[[1,0],[10,0]]; const l2=c.map(x=>(v[0]-x[0])**2+(v[1]-x[1])**2); const dot=c.map(x=>1-v[0]*x[0]-v[1]*x[1]); const pick=a=>a.indexOf(Math.min(...a)); console.log({l2,dot,l2_assignment:pick(l2),dot_assignment:pick(dot)})'

It reports l2_assignment: 0 and dot_assignment: 1. The same vector/codebook therefore receives a different code solely because distributed reconstruction switches the quantizer metric. The focused repository test also passes while asserting that the affected one-of-two-fragments case executes, so it does not detect the mismatch.

[u70b3]

Fixed in 8c15d19 — the guard is inverted as suggested: full-coverage selections (implicit NULL or an explicit list of every fragment) now pass, and only an effective strict subset with a supplied DOT PQ model set is rejected, in AUTO and PRECOMPUTED for both IVF-PQ variants. The subset check compares fragment-id sets (mirroring effective_vector_fragments) rather than counts, and the header contract and regression test are inverted to match; the test now executes an implicit full-coverage DOT PQ build end to end to cover the safe path.

I verified each link of the analysis against pinned e934cc2 before making the change: create.rs:845 normalization, builder.rs:517 hard-coded L2 in load_or_build_quantizer, vector.rs:642 make_global_pq rewrapping with DOT, and pq/storage.rs:350 persisting the quantizer's distance type into the index metadata. The analysis holds — thank you for the careful catch.

[lance-gatekeeper]

Fixed in 8c15d19: the guard now preserves implicit and explicit full-dataset coverage and rejects supplied DOT PQ only for effective strict subsets in AUTO and PRECOMPUTED across both IVF-PQ variants; the header contract and regression coverage match that verified behavior.

[u70b3]

@jja725 heads-up on 8c15d19: after re-reading the pinned Lance source, I believe lance-gatekeeper's latest analysis is correct and my previous fix (873e0c6) had the guard inverted.

What I verified in e934cc2:

• Routing (rust/lance/src/index/create.rs:845): an implicit or explicit full-coverage selection normalizes to None and takes build_vector_index; only an effective strict subset with precomputed IVF enters build_distributed_vector_index.
• Full-dataset path is safe: load_or_build_quantizer builds the PQ quantizer with a hard-coded DistanceType::L2 (rust/lance/src/index/vector/builder.rs:517), and PQIndexStorage persists that distance type into the index metadata (rust/lance-index/src/vector/pq/storage.rs:350), so readers compute PQ distances with L2 as well. Encoding and search stay consistent — the "L2-encoded codes vs DOT search" mismatch I was guarding against does not actually happen on this path.
• Distributed path is the broken one: make_global_pq (rust/lance/src/index/vector.rs:642) only maps Cosine→L2, so a supplied DOT codebook is rewrapped with DistanceType::Dot and code assignment silently switches metric — that is where the real mismatch lives.

So 8c15d19 flips the fail-fast: full-coverage DOT PQ builds are enabled again, and effective strict subsets with a supplied DOT PQ model set are rejected (AUTO/PRECOMPUTED, IVF-PQ and IVF-HNSW-PQ) until upstream make_global_pq reconstructs with L2. The subset check compares fragment-id sets rather than counts, the header contract and regression test are inverted accordingly, and the test now runs an implicit full-coverage DOT PQ build end to end.

This reverses the direction of your earlier suggestion, so I'd rather not proceed on the bot's analysis alone — could you take another look and make the final call? If we agree on the direction, I'm happy to file the upstream Lance issue for make_global_pq (rewrap supplied DOT codebooks with L2) so the subset restriction can eventually be removed.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Both prior Gate findings are addressed. DOT PQ training remains on raw vectors, while supplied DOT PQ segment builds preserve the safe full-dataset path and reject only effective strict subsets across AUTO/PRECOMPUTED and both IVF-PQ variants. The C contract and focused regression coverage match the pinned Lance routing and quantizer semantics.