Skip to content

[Vector Upsert 2/5] Harden vector search metric recording and backend param cleanup - #19298

Open
xiangfu0 wants to merge 2 commits into
apache:masterfrom
xiangfu0:xiangfu0/upsert-vector-2-backend-param-cleanup
Open

[Vector Upsert 2/5] Harden vector search metric recording and backend param cleanup#19298
xiangfu0 wants to merge 2 commits into
apache:masterfrom
xiangfu0:xiangfu0/upsert-vector-2-backend-param-cleanup

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Part 2/5 of the split of #19287 (Fix FULL-upsert vector candidate generation).

Summary

Two robustness fixes in VectorSimilarityFilterOperator.executeSearch:

  • Only record a vector search metric when a search actually executed. Previously a
    failure inside configureBackendParams still recorded a search that never ran.
  • Guarantee backend search parameters are cleared even when configuration fails partway
    through the setters or the explain-context refresh throws, by claiming cleanup before
    the first setter and clearing in a nested finally. Backend search parameters
    (nprobe, efSearch) are ThreadLocal overrides on shared per-segment readers, so a
    skipped cleanup leaks a stale override onto a pooled query thread where it silently
    applies to a later query's search.

Part 4/5 relies on this hardening for its early-return exact-scan path.

Validation

  • VectorSimilarityFilterOperatorTest: 23 tests passed, including the new
    testBackendParamsAreClearedWhenConfigurationFailsPartway.

Stacked on #19297 — this PR shows its commits until that merges; review only the last commit here.

Stack

  1. [Vector Upsert 1/5] Support allowed-document filtering in exact vector scan #19297 — exact-scan allowed-document filtering
  2. [Vector Upsert 2/5] Harden vector search metric recording and backend param cleanup #19298 — backend param cleanup hardening
  3. [Vector Upsert 3/5] Never materialize nested vector subtrees as metadata filters #19299 — nested vector metadata guard
  4. [Vector Upsert 4/5] Apply the query's visible-document set before vector candidate generation #19300 — apply the visible-document set before candidate generation (core fix)
  5. [Vector Upsert 5/5] Add FULL-upsert vector integration coverage #19301 — integration coverage

@codecov-commenter

codecov-commenter commented Aug 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.89474% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.21%. Comparing base (e4888b0) to head (5796c1c).

Files with missing lines Patch % Lines
...operator/filter/ExactVectorScanFilterOperator.java 83.05% 6 Missing and 4 partials ⚠️
...not/core/operator/filter/VectorCandidateScope.java 75.00% 2 Missing ⚠️
...perator/filter/VectorSimilarityFilterOperator.java 88.88% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19298      +/-   ##
============================================
+ Coverage     67.20%   67.21%   +0.01%     
  Complexity     1424     1424              
============================================
  Files          3462     3463       +1     
  Lines        220361   220394      +33     
  Branches      35147    35158      +11     
============================================
+ Hits         148087   148134      +47     
+ Misses        60467    60448      -19     
- Partials      11807    11812       +5     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-25 67.21% <82.89%> (+0.01%) ⬆️
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 67.21% <82.89%> (+0.01%) ⬆️
unittests 67.21% <82.89%> (+0.01%) ⬆️
unittests1 57.84% <82.89%> (+<0.01%) ⬆️
unittests2 39.22% <0.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/upsert-vector-2-backend-param-cleanup branch 2 times, most recently from 8abc05f to 5796c1c Compare August 19, 2026 01:39
@xiangfu0 xiangfu0 added bug Something is not working as expected vector Related to vector similarity search query Related to query processing metrics Related to metrics emission and collection data-integrity Related to correctness of data or query results labels Aug 19, 2026
…r scan

Extract a shared computeExactMatches helper that unifies the exact top-K and
threshold scan paths and can restrict scoring to an allowed-document bitmap.

Introduce VectorCandidateScope, an immutable set of document IDs a vector
predicate is allowed to consider as candidates. Vector top-K is not monotonic,
so unlike an ordinary predicate -- which is correct to intersect with the result
afterwards -- a document set that defines what the query may see has to be
applied before candidate generation. The type carries that contract and says
nothing about where the restriction came from; callers decide that.
ExactVectorScanFilterOperator accepts one via a new constructor overload, and no
production caller passes one yet.

Also let the operator record its own exact-scan search and fallback metrics,
which were previously invisible to VectorSearchMetrics, and report the applied
candidate-filter cardinality in explain output.

Add filtered-reader coverage for HnswVectorIndexReader.
… param cleanup

Only record a vector search metric when a search actually executed, and
guarantee backend search parameters are cleared even when configuration fails
partway or the explain-context refresh throws. Previously a failure inside
configureBackendParams still recorded a search that never ran, and an exception
from refreshExplainContext could skip clearBackendParams entirely.
@xiangfu0
xiangfu0 force-pushed the xiangfu0/upsert-vector-2-backend-param-cleanup branch from 5796c1c to 0e2f053 Compare August 20, 2026 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something is not working as expected data-integrity Related to correctness of data or query results metrics Related to metrics emission and collection query Related to query processing vector Related to vector similarity search

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants