[Vector Upsert 4/5] Apply the query's visible-document set before vector candidate generation - #19300
Open
xiangfu0 wants to merge 4 commits into
Open
Conversation
This was referenced Aug 19, 2026
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #19300 +/- ##
============================================
- Coverage 67.12% 57.88% -9.24%
+ Complexity 1424 7 -1417
============================================
Files 3462 2663 -799
Lines 220677 160702 -59975
Branches 35255 26548 -8707
============================================
- Hits 148136 93028 -55108
+ Misses 60708 59760 -948
+ Partials 11833 7914 -3919
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
xiangfu0/upsert-vector-4-upsert-candidate-scope
branch
from
August 19, 2026 01:38
ca0c7f0 to
7ee68b3
Compare
xiangfu0
changed the title
xiangfu0
added
bug
…r scan Extract a shared computeExactMatches helper that unifies the exact top-K and threshold scan paths and can restrict scoring to an allowed-document bitmap. Introduce VectorCandidateScope, an immutable set of document IDs a vector predicate is allowed to consider as candidates. Vector top-K is not monotonic, so unlike an ordinary predicate -- which is correct to intersect with the result afterwards -- a document set that defines what the query may see has to be applied before candidate generation. The type carries that contract and says nothing about where the restriction came from; callers decide that. ExactVectorScanFilterOperator accepts one via a new constructor overload, and no production caller passes one yet. Also let the operator record its own exact-scan search and fallback metrics, which were previously invisible to VectorSearchMetrics, and report the applied candidate-filter cardinality in explain output. Add filtered-reader coverage for HnswVectorIndexReader.
… param cleanup Only record a vector search metric when a search actually executed, and guarantee backend search parameters are cleared even when configuration fails partway or the explain-context refresh throws. Previously a failure inside configureBackendParams still recorded a search that never ran, and an exception from refreshExplainContext could skip clearBackendParams entirely.
…ata filters FilterPlanNode treated any non-leaf AND sibling as a metadata filter when wiring pre-filter bitmaps for filter-aware ANN. A nested subtree containing a VECTOR_SIMILARITY predicate (e.g. AND(vector, AND(vector2, metadata)) or AND(vector, NOT(vector2))) was eagerly materialized into a bitmap, executing top-K candidate generation out of its boolean context and pushing its results into the sibling vector search as a pre-filter. Track the retained child FilterContexts alongside their operators so wirePreFilterForVectorOperators can pair them, and only treat subtrees with no vector predicate as metadata. Adaptive pre-filter selection for plain metadata siblings is unchanged.
…tor candidate generation FilterPlanNode constructed and executed VECTOR_SIMILARITY before adding SegmentContext.getDocIdsSnapshot() as an outer AND. Obsolete physical versions of upserted rows could therefore occupy per-segment ANN top-K slots and be removed only afterward, producing fewer than K rows or omitting nearer current rows. Pass the snapshot to vector predicates as a required VectorCandidateScope so it constrains candidate generation, while retaining the outer bitmap AND as defense in depth. FilterPlanNode remains the only place that knows the scope comes from the segment's queryable-document snapshot; the operators only see 'documents this predicate may consider'. Choose the execution path at plan time, where both the reader capability and forward index availability are known: - a filter-aware reader receives the scope and does filtered ANN; - a reader that cannot restrict its search is bypassed for ExactVectorScanFilterOperator over the allowed documents, which reports it through fallbackReason; - an empty scope needs no candidate generation and no reader capability at all, so it short-circuits to an empty operator; - when neither path is available the query fails clearly. Because the choice is made once, VectorSimilarityFilterOperator no longer needs a runtime exact-scan branch, and it captures the reader capability once instead of re-reading it per execution, so a plan built on one answer can never execute against another. Required scopes stay separate from optimizer-selected metadata filters and are intersected with them before candidate generation. Non-upsert queries keep the adaptive metadata behavior, and non-vector plans do not copy the snapshot.
xiangfu0
force-pushed
the
xiangfu0/upsert-vector-4-upsert-candidate-scope
branch
from
August 20, 2026 09:09
7ee68b3 to
064abd0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 4/5 of the split of #19287 (Fix FULL-upsert vector candidate generation).
This is the core fix. Stacked on #19299.
Summary
FilterPlanNodeconstructed and executedVECTOR_SIMILARITYbefore addingSegmentContext.getDocIdsSnapshot()as an outer AND. Obsolete physical versions ofupserted rows could therefore occupy per-segment ANN top-K slots and be removed only
afterward, producing fewer than K rows or omitting nearer current rows.
Pass the snapshot to vector predicates as a required
VectorCandidateScopeso itconstrains candidate generation, while retaining the outer bitmap AND as defense in
depth.
FilterPlanNoderemains the only place that knows the scope comes from thesegment's queryable-document snapshot; the operators only ever see "documents this
predicate may consider". That snapshot is already more general than upsert — it is the
queryable-document set, which also covers delete tombstones and
skipUpsertDelete— sono upsert vocabulary leaks into
pinot-core's filter operators.The execution path is chosen at plan time, where both the reader capability and
forward index availability are known:
ExactVectorScanFilterOperatorover the allowed documents, reported throughfallbackReason;short-circuits to an empty operator;
Because the choice is made once,
VectorSimilarityFilterOperatorneeds no runtimeexact-scan branch, and it captures the reader capability once instead of re-reading it
per execution, so a plan built on one answer can never execute against another. Required
scopes stay separate from optimizer-selected metadata filters and are intersected with
them before candidate generation. Non-upsert queries keep the adaptive metadata
behavior, non-vector plans do not copy the snapshot, and the new explain attributes are
emitted only for queries that actually carry a scope.
Performance and compatibility
ANN approximation semantics are unchanged. Segments whose reader cannot restrict its
search use a correctness-first exact scan whose cost is proportional to the
allowed-document count times vector dimension; consuming segments of upsert tables take
this path today because
MutableVectorIndexis not filter-aware. Making the mutableindex filter-aware would retire that fallback and can be done separately.
Validation
(
FilterPlanNodeTest,VectorSimilarityFilterOperatorTest,FilterAwareVectorSearchTest,ExactVectorScanFilterOperatorTest,VectorRadiusFilterOperatorTest,VectorSearchStrategyTest).Stack