Skip to content

Allow parallel optimizations in Qdrant after uploading#208

Merged
KShivendu merged 1 commit into
masterfrom
qdrant-parallel
Jan 2, 2025
Merged

Allow parallel optimizations in Qdrant after uploading#208
KShivendu merged 1 commit into
masterfrom
qdrant-parallel

Conversation

@timvisee

@timvisee timvisee commented Oct 2, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

The max_optimization_threads parameter is currently configured incorrectly after uploading, and this does not allow parallel optimizations.

This PR sets it to a high value so that we do use parallel optimizations as it'll likely result in better results. Note that setting such high limit is fine, because in practice we're still limited by CPU budget. We cannot set it back to null through this API.

On my machine with 24 CPUs I now see 3 parallel optimizations, rather than just 1. The indexing time is more than 2 times quicker.

Am I correct that the benchmark machine we used for the public results does not have more than 8 CPU cores? If we did have more cores, we should probably redo the benchmarks.

@filipecosta90

filipecosta90 commented Oct 3, 2024

Copy link
Copy Markdown
Contributor

@timvisee have you noticed: #163 that solved it, plus included backoff to avoid any errors?

@KShivendu

KShivendu commented Jan 2, 2025

Copy link
Copy Markdown
Member

@timvisee I'm merging this

@KShivendu KShivendu merged commit a26483b into master Jan 2, 2025
@KShivendu KShivendu deleted the qdrant-parallel branch January 2, 2025 18:31
fcostaoliveira added a commit to redis-performance/vector-db-benchmark that referenced this pull request Jul 10, 2026
@fcostaoliveira @weaviate-git-bot
@pre-commit-ci @ankane @tellet-q @KShivendu @filipecosta90 @joein @generall @ekorman @igniting @SebanDan @eltociear @timvisee @LukasWestholt @xzfc @claude @paulorsousa
feat(milvus): match_any filter (keyword/int IN-list) (#82)
9fdf1a2 
* Updated Weaviate Docker image url (auto PR by bot) (#109)

* updated the Weaviate Docker image location (automated bot update)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* updated the Weaviate Docker image location (automated bot update)

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* pgvector improvements (#98)

* pgvector improvements

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updated Postgres parameters

* Use versioned Docker image

* Updated pgvector to 0.6.2

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit suggestions (#47)

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.4.0...v4.5.0)
- [github.com/psf/black: 22.12.0 → 24.3.0](https://github.com/psf/black/compare/22.12.0...24.3.0)
- [github.com/PyCQA/isort: 5.12.0 → 5.13.2](https://github.com/PyCQA/isort/compare/5.12.0...5.13.2)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* refactoring: Standardize format of search params in engine configs (#122)

* refactoring: Standardize format of search params in engine configs

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* address review and rename params to config

* address review

* fix: upgrade milvusdb image version

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* refactor: Nested search params in ES config (#120)

* refactor: Nested search params in ES config

Co-authored-by: filipe oliveira <filipecosta.90@gmail.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Remove extra config vars

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Remove extra comment

* feat: Add keyword, text, and float index types in ES

---------

Co-authored-by: filipe oliveira <filipecosta.90@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* refactor: Fix and simplify benchmark processing notebook (#125)

* feat: Fix process benchmarks notebook

* refactor: Fix and simplify benchmark processing notebook

* fix: Backward comp for processing script

* feat: Add params

* feat: Add sparse vectors benchmark support for Qdrant (#114)

* feat: Add sparse vectors benchmark support in Qdrant

* fix: Self review

* feat: Add sparse dataset for CI benchmarks

* feat: Introduce SparseVector class

* feat: Disallow sparse vector dataset being run with non sparse vector engine configs

* feat: use different engine config to run sparse vector benchmarks

* fix: use different engine config to run sparse vector benchmarks

* feat: Optimize CI benchmarks workflow

* feat: Add 1M sparse dataset

* fix: remove scipy, read csr matrix manually (#117)

* fix: remove scipy, read csr matrix manually

* fix: Dataset query reader should have sparse_vector=None by default

* refactor: Changes based on feedback

* refactoring: refactor sparse vector support (#118)

* refactoring: refactor sparse vector support

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* feat: Use pydantic construct

* refactor: Update all engines to use Query and Record dataclasses (#116)

* refactor: Update all engines to use Query and Record dataclasses

* feat: Add ruff in pre-commit hooks

* fix: Type mismatches

* fix: Redis search client types and var names

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Type issues detected by linter

* fix: iter_batches func type

* refactor: knn_conditions should be class level constant

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: Type issue

* fix: Allow python 3.8 since scipy is now removed

* fix: Add missing redis-m-16-ef-128 config

* fix: redis container port

* fix linter

---------

Co-authored-by: George <george.panchuk@qdrant.tech>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>

* fix: Remove mc (#127)

* fix: mc setup env vars are not set

* fix: Remove mc

* fix: Manual benchmarks (#128)

* fix: Manual benchmarks

* fix: Remove gcs secrets

* feat: Add mmap support for reading sparse vectors to avoid OOM error in CI (#129)

* fix: Manual benchmarks

* fix: Remove gcs secrets

* feat: Use mmap to read sparse vectors

* fix: Format

* fix: Make unused var private

* fix: Avoid reading all mmaped sparse vectors into memory (#130)

* fix: Use smaller sparse dataset for faster iteration (#132)

* feat: Force docker image removal (#131)

* feat: Use both RssAnon and VmRSS in CI benchmarks (#133)

* feat: Use RssAnon instead of VmRSS

* feat: Show both RSSAnon and VMRss

* Automate running benchmarks for all engines (#134)

* ci: Run *-default benchmarks for all engines

* Update poetry.lock

* feat: Add H&M filter dataset to CI benchmarks (#140)

* feat: Add H&M filter dataset to CI benchmarks

* fix: Remove extra space

* fix: Add ServerAliveInterval and ServerAliveCountMax (#141)

* add runner resources monitoring

* Revert "add runner resources monitoring"

This reverts commit 7e5663e2cff34ecd73c115a763af471884991d42.

* add ServerAliveInterval and ServerAliveCountMax

---------

Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>

* feat: Add BQ to CI benchmarks (#148)

* feat: Add BQ to CI benchmarks

* feat: Allow manual benchmarks to specify engine config

* feat: Add DBpedia OpenAI embedding dataset with 100k vectors (#150)

* Fix github triggers (#154)

Fix github triggers

* Weaviate version 1.25.1 (#143)

* Updated weaviate image to 1.25.1

* Updated weaviate image to 1.25.1

* feat: Add 15m timeout for CI benchmarks (#157)

* feat: Add 15m timeout for CI benchmarks

* feat: Add Slack alert for CI benchmarks (#158)

* feat: Add Slack alert for CI benchmarks

* Update continuous-benchmark.yaml

* fix: Use quotes on func name to declare they arent commands

* feat: Separate failure reasons

* fix: YAML error

* fix: Rename nodes in cluster mode (#161)

* up qdrant version

* more configs

* fix: Typo in func name (#164)

* add convertor

* timeout at 30 mins

* set ports instead of network_mode (#145)

* set ports instead of network_mode

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* set ports for weaviate

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Use efficient filtering for opensearch (#167)

Current filtering query does post-filtering which is not ideal and
resulted in low precision.

Results with post filtering:
```
{
  "params": {
    "dataset": "arxiv-titles-384-angular-filters",
    "experiment": "opensearch-default",
    "engine": "opensearch",
    "parallel": 1,
    "config": {
      "knn.algo_param.ef_search": 128
    }
  },
  "results": {
    "total_time": 708.6168032020068,
    "mean_time": 0.07045893384491791,
    "mean_precisions": 0.11399200000000001,
    "std_time": 0.06840096039381999,
    "min_time": 0.008397486002650112,
    "max_time": 3.3753458530118223,
    "rps": 14.111999538838594,
    "p95_time": 0.18164870390755816,
    "p99_time": 0.20864198897208555
  }
}
```

Results with new efficient filtering:
```
{
  "params": {
    "dataset": "arxiv-titles-384-angular-filters",
    "experiment": "opensearch-default",
    "engine": "opensearch",
    "parallel": 1,
    "config": {
      "knn.algo_param.ef_search": 128
    }
  },
  "results": {
    "total_time": 394.4290532110026,
    "mean_time": 0.03913764159695711,
    "mean_precisions": 0.610144,
    "std_time": 0.05352479065894972,
    "min_time": 0.0009066620114026591,
    "max_time": 2.1307434440095676,
    "rps": 25.35310195481576,
    "p95_time": 0.1274049270534305,
    "p99_time": 0.2078282342318563
  }
}
```

* Fix opensearch query parser (#172)

* Fix opensearch query parser

Also specify schema for id in arxiv-titles-384-angular filters dataset.
This is needed as default mapping created by opensearch takes the type
as float, but in the dataset we also have string.
This leads to 99% precision on the test dataset with opensearch.

```
{
  "params": {
    "dataset": "arxiv-titles-384-angular-filters",
    "experiment": "opensearch-default",
    "engine": "opensearch",
    "parallel": 10,
    "config": {
      "knn.algo_param.ef_search": 128
    }
  },
  "results": {
    "total_time": 391.3466711850051,
    "mean_time": 0.37053632343088827,
    "mean_precisions": 0.98962,
    "std_time": 0.3485163122349799,
    "min_time": 0.04950378900684882,
    "max_time": 3.548553360000369,
    "rps": 25.552791773390613,
    "p95_time": 1.0473700279486362,
    "p99_time": 1.4396829416653787
  }
}
```

Fixes #171

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* [pre-commit.ci] pre-commit suggestions (#169)

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.5.0 → v4.6.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.5.0...v4.6.0)
- [github.com/psf/black: 24.3.0 → 24.4.2](https://github.com/psf/black/compare/24.3.0...24.4.2)
- [github.com/astral-sh/ruff-pre-commit: v0.3.5 → v0.5.0](https://github.com/astral-sh/ruff-pre-commit/compare/v0.3.5...v0.5.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Updated milvus from 2.3.1 to 2.4.1  (#144)

* updated milvus from 2.3.1 to 2.4.1

* Fixed json logging numeric/text issue on docker composes of milvus

* feat: Add debug logs (#166)

* feat: Add debug logs

* feat: Add file and func name in logs

* fix: Remove funcname

* fix after upgrading milvus (#175)

* fix: Unbound variable error (#178)

* Only copy search and upload in first step (#179)

Memory will be collected by qdrant_collect_stats.sh

* Fail CI if any benches fail (#180)

* Fail CI if any benches fail

* More types of traps

* remove intentional exit

* fix CI (#181)

* Allow CI to force clear previously running resources if required (#182)

* Allow CI to force clear previously running resources if required

* Remove running vector-db-benchmark instances

* Bump qdrant versions to 1.11.0 (#184)

* Update qdrant-client version to 1.11

* Run all engines if poetry changed

* Bump qdrant docker image version

* Add continuous benchmark for tenants (#183)

* Split Ci benchmarks into 2 jobs (#186)

* Increase CONTAINER_MEM_LIMIT to 160mb (#191)

* fix: pgvector and_subfilter (#193)

* fix: pgvector and_subfilter

* fix: exact match filter raise error instead of return value

* docs: update process-benchmarks.ipynb (#185)

benchmar -> benchmark

* Disable workflow for investigation (#197)

* Enable workflow again

* Disable cron (#201)

* Enable cron (#203)

* Improve volumes and logging (#202)

* Add logging to search requests

* Use default volumes instead of bind

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add benchmark on collection load time (#204)

* Recover collection from a snapshot
* Push init_time_ms values into postgres
* Collect telemetry separately
* Add dedicated benchmark strategy
* Use new benchmark-server-3
* Run in parallel

* Report dataset's info in slack (#211)

* Add job name into slack message

* Add reason into slack message

* Debug

* Revert "Debug"

This reverts commit 27f0b5c0d16a12f4b0d4d6af39927c69e904a023.

* Debug

* Debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug

* Debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Debug

* Debug

* Debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Format the message

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Update text (#212)

* Add benchmark on parallel upload and search (#215)

* Add parallel upload&search workflow

* Introduce new step: search without upsert

* Update tools/upload_parallel_results_postgres.sh

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Update tools/run_ci.sh

Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Fix indent

* Explicit mode check

* Provide CONTAINER_MEM_LIMIT explicitly

* Store parallel results in separate folder

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>

* Allow parallel optimizations in Qdrant after uploading (#208)

* Only run runLoadTimeBenchmark once per day

* Add comment

* Split into 2 and introduce concurrency groups

* Update name

* Add volume to persist datasets

* Add volume to persist datasets

* Debug

* Update datasets.json during benches

* Revert debug

* Add a workflow to remove datasets volume

* Revert debug

* Rename file

* Add ServerAliveInterval and ServerAliveCountMax to rsync and more (#220)

* [pre-commit.ci] pre-commit suggestions (#210)

* [pre-commit.ci] pre-commit suggestions

updates:
- [github.com/pre-commit/pre-commit-hooks: v4.6.0 → v5.0.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.6.0...v5.0.0)
- [github.com/psf/black: 24.4.2 → 24.10.0](https://github.com/psf/black/compare/24.4.2...24.10.0)
- [github.com/astral-sh/ruff-pre-commit: v0.5.0 → v0.8.6](https://github.com/astral-sh/ruff-pre-commit/compare/v0.5.0...v0.8.6)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* fix: weaviate-client version constraint (#199) (#200)

* ci/fix-poetry-install (#221)

* Fix package-mode

* Update deps

* Fix 403 (#222)

* Fix job's name in notification (#224)

* Update qdrant-single-node-bq-rps.json

* Update qdrant-single-node-bq-rps.json

* Update qdrant-single-node-bq-rps.json

* Add Compare Versions Workflow (#225)

* Update deps

* Add workflow: check image existence and trigger image build

* [pre-commit.ci] auto fixes from pre-commit.com hooks, for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Complete compare versions' workflow (#226)

* Compare versions

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Use GITHUB_STEP_SUMMARY

* Add doc strings

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Cancel manual workflow early (#227)

* Cancel run if prepareImage fails

* Use query_points (#230)

* Use query_points

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix sparse vector name in query (#232)

* Fix sparce vector name

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Run ci with payload (#231)

* Add with_payload param

* Update poetry

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Use find instead of ls to search files (#234)

* Run benchmarks with feature flags for dev (#237)

* Run benchmarks with QRANT__FEATURE_FLAGS__ALL=true for dev only

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Fix fetching upload results (#239)

* Ensure search in current dir only (#240)

* Ensure search in current dir only

* Fix

* Introduce workflow input `feature_flags_all` for manual benchmarks (#241)

* Fix descriptions

* Apply suggestions from code review

* Use boolean input type in workflow input for toggling all feature flags (#242)

* Enable indexing while uploading

* Reduce ci code duplication (#245)

* Add gh action to send slack message

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Add CPU stats into monitoring (#244)

* Add comment line

* Store cpu and telemetry cpu values for default bench

* Improve README docs and examples (#217)

* Improve GH outputs (#246)

* add changes for running cohere wiki benchmark

* rescore with prefetch option

* upd deps

* upd params

* upd params

* upd test config

* feat(milvus): support match_any filter (keyword/int IN-list)

Adds the `match_any` condition to the Milvus expression builder: a
`{"<field>":{"match":{"any":[...]}}}` condition now emits `field in [...]`,
the OR-of-values semantics that mirror qdrant's Condition::matches(field, Vec).
Strings are quoted/escaped, numbers inlined; bool/null/nested items are skipped
so an invalid expression is never produced. Empty (or all-skipped) list is a
clean no-op. Exact-match/range behaviour unchanged.

Tests: unit assertions for string `in`, int `in`, empty-list no-op, exact-value
regression; integration `test_binary_milvus_match_any` drives the real binary
against a live Milvus container via the shared fixture and asserts recall >= 0.9
vs ground truth filtered to the matching docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(milvus): empty match_any matches nothing, not every row

Empty (or all-skipped) match_any returned None; when it was the sole clause
parse_milvus_conditions returned None -> no filter -> every row returned (the
inverse of the filter). Emit `field in []` (a valid match-nothing expression)
so an empty IN-set matches no rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(parity): include index-build wait in total_time; count failed queries (#91)

Two critical parity/trustworthiness fixes surfaced by a 15-agent adversarial
review against the v0 Python oracle (qdrant/vector-db-benchmark).

1. Index-build wait now counted in upload total_time (all applicable engines).
   6 of 8 engines computed total_time = read_time + upload_time *before* the
   async index-readiness wait (RediSearch percent_indexed, Qdrant GREEN, ES/OS
   force-merge+refresh, Milvus create_index+load), so the headline indexing-time
   column excluded the dominant ingest cost — only mongodb included it, making
   every other engine look artificially fast and breaking cross-engine
   comparison. Each now times the wait separately and adds it (mirrors
   mongodb_engine, matches v0 which times through post_upload()):
     redis, valkey, qdrant (KNN + sparse), elasticsearch, opensearch, milvus.
   pgvector is unaffected (builds its HNSW index in configure() before upload,
   so the cost is already inside upload_time). vectorsets/turbopuffer have no
   local index-build wait.

2. Failed/requested/succeeded query accounting.
   RPS and the latency percentiles are computed over successful queries only,
   while total_time (the RPS denominator) spans the whole run and failures were
   merely eprintln'd — a run that sheds timeouts (e.g. a saturated client) wrote
   to disk as clean, tail-flattered numbers with no trace of the drops.
   - SearchResults gains requested_queries + failed_queries; compute_search_stats
     takes requested_queries and derives failed = requested - successes.
   - save_search_results emits requested_queries / succeeded_queries /
     failed_queries; the runner prints a loud WARNING when any query failed.
   - turbopuffer (inline stats) corrected: num_queries now = successes (was
     mislabeled as num_to_run) and reports requested/failed consistently.

Follow-ups noted for later ranks: weaviate has NO post-upload readiness wait at
all (may search a partially-indexed store) — a behavioral gap beyond timing.

make check + make test (49) + stats_tests (3) + integration-test (15) all pass;
verified end-to-end on redis/random-100 (Index time in total_time; new query
counts in the result JSON).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(coverage): detect & report client CPU / concurrency saturation (#92)

* fix(parity): include index-build wait in total_time; count failed queries

Two critical parity/trustworthiness fixes surfaced by a 15-agent adversarial
review against the v0 Python oracle (qdrant/vector-db-benchmark).

1. Index-build wait now counted in upload total_time (all applicable engines).
   6 of 8 engines computed total_time = read_time + upload_time *before* the
   async index-readiness wait (RediSearch percent_indexed, Qdrant GREEN, ES/OS
   force-merge+refresh, Milvus create_index+load), so the headline indexing-time
   column excluded the dominant ingest cost — only mongodb included it, making
   every other engine look artificially fast and breaking cross-engine
   comparison. Each now times the wait separately and adds it (mirrors
   mongodb_engine, matches v0 which times through post_upload()):
     redis, valkey, qdrant (KNN + sparse), elasticsearch, opensearch, milvus.
   pgvector is unaffected (builds its HNSW index in configure() before upload,
   so the cost is already inside upload_time). vectorsets/turbopuffer have no
   local index-build wait.

2. Failed/requested/succeeded query accounting.
   RPS and the latency percentiles are computed over successful queries only,
   while total_time (the RPS denominator) spans the whole run and failures were
   merely eprintln'd — a run that sheds timeouts (e.g. a saturated client) wrote
   to disk as clean, tail-flattered numbers with no trace of the drops.
   - SearchResults gains requested_queries + failed_queries; compute_search_stats
     takes requested_queries and derives failed = requested - successes.
   - save_search_results emits requested_queries / succeeded_queries /
     failed_queries; the runner prints a loud WARNING when any query failed.
   - turbopuffer (inline stats) corrected: num_queries now = successes (was
     mislabeled as num_to_run) and reports requested/failed consistently.

Follow-ups noted for later ranks: weaviate has NO post-upload readiness wait at
all (may search a partially-indexed store) — a behavioral gap beyond timing.

make check + make test (49) + stats_tests (3) + integration-test (15) all pass;
verified end-to-end on redis/random-100 (Index time in total_time; new query
counts in the result JSON).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(coverage): detect and report client CPU / concurrency saturation

Adds the missing signal the review flagged as goal #3: the tool reported DB
throughput/latency with zero awareness of whether the *client* was the
bottleneck. When the benchmark client saturates (more worker threads than
cores, or CPU-bound on serialization/TLS), QPS plateaus and measured latency
captures run-queue wait — and it was published as clean server-side numbers.

New `proc_cpu` module (Linux, no new deps):
- Samples /proc/self/stat (process ticks) and /proc/stat (all-core jiffies)
  around the timed search window. The tick unit cancels in the ratio, so
  client_cpu_cores_used and system_cpu_pct need neither libc/sysconf nor a
  hardcoded USER_HZ. Degrades to None (never a false positive) off Linux.
- compute() derives: available_cores (available_parallelism), oversubscribed
  (parallel > cores), client_cpu_cores_used, system_cpu_pct, client_saturated
  (>0.85 core/core, or oversubscribed, or system >0.90), saturation_reason.

Wiring:
- SearchResults gains the six coverage fields (Default, so all engine builders
  are unaffected; the 4 mixed-search builders now use ..Default::default() so
  future field additions never break them again). Zero per-engine churn — the
  runner samples CPU around engine.search()/search_mixed() and annotates the
  returned struct in one place (experiment.rs).
- experiment.rs prints a loud per-run WARNING when client_saturated; result
  JSON emits parallel/available_cores/oversubscribed/client_cpu_cores_used/
  system_cpu_pct/client_saturated/saturation_reason.
- summary.rs: analyze_precision_performance now PREFERS non-saturated points
  for each precision bucket (falls back to a saturated one only when it is the
  sole candidate), so the headline "best QPS" is never a client-bound point.
  Adds a scaling-collapse detector (per ef: raising parallel gains <10% QPS
  while p95 rises) and prints a CONCURRENCY/CPU SATURATION WARNINGS block;
  saturation fields flow into the summary JSON.

make check + all 83 bin tests (incl. 5 proc_cpu) pass. Verified live on
redis/random-100 at parallel=32 on a 14-core box: per-run + summary warnings
fire and the JSON carries oversubscribed=true, client_cpu_cores_used≈1.4,
client_saturated=true.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(parity): numpy-linear percentiles matching v0 (#94)

* fix(parity): include index-build wait in total_time; count failed queries

Two critical parity/trustworthiness fixes surfaced by a 15-agent adversarial
review against the v0 Python oracle (qdrant/vector-db-benchmark).

1. Index-build wait now counted in upload total_time (all applicable engines).
   6 of 8 engines computed total_time = read_time + upload_time *before* the
   async index-readiness wait (RediSearch percent_indexed, Qdrant GREEN, ES/OS
   force-merge+refresh, Milvus create_index+load), so the headline indexing-time
   column excluded the dominant ingest cost — only mongodb included it, making
   every other engine look artificially fast and breaking cross-engine
   comparison. Each now times the wait separately and adds it (mirrors
   mongodb_engine, matches v0 which times through post_upload()):
     redis, valkey, qdrant (KNN + sparse), elasticsearch, opensearch, milvus.
   pgvector is unaffected (builds its HNSW index in configure() before upload,
   so the cost is already inside upload_time). vectorsets/turbopuffer have no
   local index-build wait.

2. Failed/requested/succeeded query accounting.
   RPS and the latency percentiles are computed over successful queries only,
   while total_time (the RPS denominator) spans the whole run and failures were
   merely eprintln'd — a run that sheds timeouts (e.g. a saturated client) wrote
   to disk as clean, tail-flattered numbers with no trace of the drops.
   - SearchResults gains requested_queries + failed_queries; compute_search_stats
     takes requested_queries and derives failed = requested - successes.
   - save_search_results emits requested_queries / succeeded_queries /
     failed_queries; the runner prints a loud WARNING when any query failed.
   - turbopuffer (inline stats) corrected: num_queries now = successes (was
     mislabeled as num_to_run) and reports requested/failed consistently.

Follow-ups noted for later ranks: weaviate has NO post-upload readiness wait at
all (may search a partially-indexed store) — a behavioral gap beyond timing.

make check + make test (49) + stats_tests (3) + integration-test (15) all pass;
verified end-to-end on redis/random-100 (Index time in total_time; new query
counts in the result JSON).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(coverage): detect and report client CPU / concurrency saturation

Adds the missing signal the review flagged as goal #3: the tool reported DB
throughput/latency with zero awareness of whether the *client* was the
bottleneck. When the benchmark client saturates (more worker threads than
cores, or CPU-bound on serialization/TLS), QPS plateaus and measured latency
captures run-queue wait — and it was published as clean server-side numbers.

New `proc_cpu` module (Linux, no new deps):
- Samples /proc/self/stat (process ticks) and /proc/stat (all-core jiffies)
  around the timed search window. The tick unit cancels in the ratio, so
  client_cpu_cores_used and system_cpu_pct need neither libc/sysconf nor a
  hardcoded USER_HZ. Degrades to None (never a false positive) off Linux.
- compute() derives: available_cores (available_parallelism), oversubscribed
  (parallel > cores), client_cpu_cores_used, system_cpu_pct, client_saturated
  (>0.85 core/core, or oversubscribed, or system >0.90), saturation_reason.

Wiring:
- SearchResults gains the six coverage fields (Default, so all engine builders
  are unaffected; the 4 mixed-search builders now use ..Default::default() so
  future field additions never break them again). Zero per-engine churn — the
  runner samples CPU around engine.search()/search_mixed() and annotates the
  returned struct in one place (experiment.rs).
- experiment.rs prints a loud per-run WARNING when client_saturated; result
  JSON emits parallel/available_cores/oversubscribed/client_cpu_cores_used/
  system_cpu_pct/client_saturated/saturation_reason.
- summary.rs: analyze_precision_performance now PREFERS non-saturated points
  for each precision bucket (falls back to a saturated one only when it is the
  sole candidate), so the headline "best QPS" is never a client-bound point.
  Adds a scaling-collapse detector (per ef: raising parallel gains <10% QPS
  while p95 rises) and prints a CONCURRENCY/CPU SATURATION WARNINGS block;
  saturation fields flow into the summary JSON.

make check + all 83 bin tests (incl. 5 proc_cpu) pass. Verified live on
redis/random-100 at parallel=32 on a 14-core box: per-run + summary warnings
fire and the JSON carries oversubscribed=true, client_cpu_cores_used≈1.4,
client_saturated=true.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(parity): numpy-linear percentiles (p50/p95/p99) matching v0

compute_search_stats used nearest-rank percentiles (`sorted[floor(N*q)]`),
which biased every percentile upward versus v0's `np.percentile` (linear
interpolation) and made `p99 == max` for any N<=100 — at N=100,
`floor(0.99*100)=99` always returned the single largest sample.

Adds `percentile_linear(sorted, q)` (position `q*(N-1)`, interpolated) and
routes both compute_search_stats and turbopuffer's inline stats through it, so
percentiles match v0 and are consistent across engines. New tests pin the numpy
values and the N=100 p99<max behavior.

make check + all bin tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(parity): repeat each search config, report warm best-of (v0 REPETITIONS) (#95)

* feat(parity): repeat each search config and report warm best-of (v0 REPETITIONS)

The runner measured each search config exactly once — a single, often-cold draw
(cold OS page cache / index warm-up) with high run-to-run variance, not
comparable to v0 which runs each config REPETITIONS times (default 3) and its
best-QPS-per-precision-bucket selection implicitly discards the cold first run.

Adds `--repetitions N` (default 3, also read from the REPETITIONS env var to
mirror v0). run_single_experiment now runs each measured search `N` times and
reports/saves the best-RPS run; `--repetitions 1` restores single-shot. Prints
each rep's QPS and marks the chosen line "(best of reps)". Calibration is
unaffected (runs once, before the timed reps).

Enables clap's `env` feature for the REPETITIONS env binding.

make check + all 83 bin tests pass. Verified live on redis/random-100: 3 reps
run, best-of reported, one result file saved, precision 1.0, exit 0 (integration
tests assert exit 0 + precision, both hold).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(redis): pin --repetitions 1 in mixed-benchmark call-count test

test_binary_redis_mixed_benchmark asserts the exact per-run FT.SEARCH (10) and
HSET (102) counts; the new default of 3 warm repetitions made it 30/106. Pin
--repetitions 1 so it measures a single pass, which is what it validates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* perf(redis-family): per-thread sample buffers in KNN search hot loop (#96)

* perf(redis): per-thread sample buffers in KNN search hot loop (WIP rank 4)

Replace the 5 shared Arc<Mutex<Vec<f64>>> per-query pushes with thread-local
buffers merged on join, and switch the work counter to Relaxed. Removes 5
contended locks per query from the timed loop; metrics are order-independent
so results are unchanged. redis filter-only + mixed loops and the other engines
follow in this branch.

Verified: builds; all 15 redis integration tests pass (--test-threads=1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* perf(valkey,vectorsets): per-thread sample buffers in KNN search hot loop

Same conversion as redis: replace the 5 shared Arc<Mutex<Vec<f64>>> per-query
pushes with thread-local buffers merged on join, work counter → Relaxed.
Removes 5 contended locks/query from the timed loop; metrics unchanged.

Verified: build + clippy clean; 15/15 redis+vectorsets integration tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* perf(engines): per-thread sample buffers in KNN search hot loop (remaining 7) (#97)

Completes the rank-4 overhead fix (after #96 did redis/valkey/vectorsets):
qdrant, elasticsearch, opensearch, pgvector, milvus, mongodb, weaviate.

Same conversion in each engine's primary parallel search() KNN loop: replace
the 5 shared Arc<Mutex<Vec<f64>>> per-query pushes with thread-local buffers
returned from each scoped worker and concatenated on join; work-dispatch
counter SeqCst → Relaxed. The async engines (qdrant/es/os) return the tuple
from their existing per-worker-runtime closure. Metrics are order-independent
(percentiles sort) so results are unchanged — pure overhead removal.

Verified: make check clean (0 warnings), 85 unit tests pass. Per-engine
integration validated by CI.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(weaviate): apply search-time ef via full-class PUT (fetch-merge-put) (#79)

setup_search sent a partial `PUT /v1/schema/{class}` body containing only
`{"vectorIndexConfig":{"ef":N}}`. Weaviate requires the complete class object on
that endpoint, so it read the class name as "" and rejected every call with
422 ("class name is immutable: attempted change from \"<class>\" to \"\""). The
error was swallowed as a warning, so the query-time ef sweep never applied and
every search ran at the class's default dynamic ef (-1) — flattening the
recall/QPS curve (constant precision across ef=64/128/256/512).

Fix: GET the current class, merge the requested ef into its vectorIndexConfig,
and PUT the full object back (immutable efConstruction/maxConnections are sent
back unchanged, so they don't trip the immutability check). Failures now return
Err instead of a swallowed warning, so a broken ef sweep aborts the run rather
than silently producing misleading results.

Co-authored-by: fcostaoliveira <filipe@redis.com>

* feat(weaviate): search over gRPC (default) instead of GraphQL (#90)

* fix(weaviate): apply search-time ef via full-class PUT (fetch-merge-put)

setup_search sent a partial `PUT /v1/schema/{class}` body containing only
`{"vectorIndexConfig":{"ef":N}}`. Weaviate requires the complete class object on
that endpoint, so it read the class name as "" and rejected every call with
422 ("class name is immutable: attempted change from \"<class>\" to \"\""). The
error was swallowed as a warning, so the query-time ef sweep never applied and
every search ran at the class's default dynamic ef (-1) — flattening the
recall/QPS curve (constant precision across ef=64/128/256/512).

Fix: GET the current class, merge the requested ef into its vectorIndexConfig,
and PUT the full object back (immutable efConstruction/maxConnections are sent
back unchanged, so they don't trip the immutability check). Failures now return
Err instead of a swallowed warning, so a broken ef sweep aborts the run rather
than silently producing misleading results.

* feat(weaviate): search over gRPC (default) instead of GraphQL

The GraphQL query path (POST /v1/graphql with the query vector serialized into
the GraphQL document as a text array) is Weaviate's slow path and caps benchmark
throughput ~2x below the gRPC path used by the official clients — independent of
Weaviate version and hardware. It also worsens with vector dimensionality.

Add a gRPC search path (vendored weaviate.v1 protos -> tonic/prost client checked
in as engine/weaviate_grpc.rs; no protoc/build.rs at build time):
- near_vector_search_grpc sends the query vector as packed little-endian f32
  (vector_bytes), requests uuid+distance metadata, maps the object UUID back to
  the integer index via the existing uuid_to_int (inverse of upload id_to_uuid).
- search() drives async RPCs on one shared multi-thread tokio runtime; each worker
  thread enters the runtime context and gets its own lazily-connected channel.
- gRPC is the default; GraphQL is retained as fallback when a metadata filter is
  present (gRPC filter translation not implemented) or WEAVIATE_USE_GRAPHQL is set.
- The class-level ef (set via the REST schema update from the prior fix) still
  governs recall; gRPC only changes the query transport.

Verified locally against Weaviate 1.38.2: recall parity with GraphQL (1.0000 on
random-100, identical single-client QPS), no panics. Stacked on the ef-fix branch.

* fix(weaviate): async task fan-out for gRPC search (scales under concurrency)

The first gRPC impl fanned out 100 OS threads each calling handle.block_on on a
shared runtime. That doesn't scale: per-query block_on overhead capped even
single-client throughput, and under load with larger (high-dim) payloads it
serialized so 100 clients delivered fewer QPS than 1 (negative scaling on
gist-960 / dbpedia-1536).

Replace with idiomatic async concurrency: rt.block_on spawns  tokio
tasks, each with its own connection, awaiting searches off a shared atomic work
queue. GraphQL fallback keeps the blocking-thread model. near_vector_search_grpc
is now async. Verified locally (recall parity 1.0000; single-client throughput
~4x the block_on version).

---------

Co-authored-by: fcostaoliveira <filipe@redis.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* docs: sync engine protocols/versions with the code (full sweep) (#98)

The README engine table, integration-test version list, and the duplicate
tables in AGENTS.md/DOCKER_README.md had drifted from the actual client crates
(Cargo.toml) and test server images (tests/docker-compose.test.yml).

Corrections (verified against source):
- Weaviate: was "reqwest (REST API) / HTTP-REST + GraphQL" — now gRPC (search,
  default; tonic 0.12 / prost 0.13) + REST (schema/upload), with a note on the
  GraphQL fallback (filtered queries or WEAVIATE_USE_GRAPHQL) and WEAVIATE_GRPC_PORT.
  Added weaviate_grpc.rs to the project-structure listing.
- Qdrant client: qdrant-client 1.13 → 1.17.
- Integration-test server versions: Elasticsearch 8.10.2→9.4.3, OpenSearch
  2.19.2→3.7.0, PgVector PG16→PG18, Qdrant v1.13.4→v1.18.2, Weaviate
  1.28.9→1.38.2, Milvus 2.5.6→v2.6.19, MongoDB 8.0.4→8.0.17, Redis default 8.8.0.
- Redis quick-start example image: redis:8.6.0 → redis:8.8.0 (matches the tested
  image); same fix in AGENTS.md/DOCKER_README.md.

Docs only — no code change.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* chore(deps): upgrade redis-rs 0.27 → 1.3 (+ redis::Value parser tests) (#99)

* chore(deps): upgrade redis-rs 0.27 → 1.3; add redis::Value parser test coverage

Bumps the `redis` crate (used by the Redis/RediSearch, Valkey, and VectorSets
engines + src/redis_client.rs) from 0.27.6 to 1.3.0.

The 0.27 → 1.x migration is a no-op for this codebase's API surface: the
`Value::BulkString/SimpleString/Array` renames and RESP3 variants landed back in
0.26, and the code already matches with wildcard `_ =>` arms; the removed
`Cmd/Pipeline::execute()` (→ `exec()`) is not used on any redis command here (the
`.execute()` calls are all postgres). `Client::open`, `get_connection`,
`set_read_timeout`, `cmd`/`pipe`/`query` are unchanged. No source edits were
needed to compile. NOTE: redis 1.3 requires Rust ≥ 1.88 / edition 2024 in the
compiler — CI's ubuntu-latest toolchain satisfies this.

Because the highest-risk surface is the manual RESP-array parsing (a RESP2↔RESP3
shift could silently reshape responses), this adds unit-test coverage for the
`redis::Value` parsers that previously had none:
- redis_utils.rs `parse_failed_calls` (INFO COMMANDSTATS): prefix/upcasing,
  failed_calls extraction, malformed-line skipping.
- valkey.rs + redis.rs `parse_ft_search_response` / `extract_vector_score` /
  `redis_value_to_json`: RESP2 FT.SEARCH id/score pairs, vector_score lookup,
  Value→JSON for every variant incl. the non-exhaustive fallthrough (an unknown
  variant maps to a non-empty string, never silently dropped).
- vectorsets.rs: extracted the inline VSIM reply parsing into a testable
  `parse_vsim_response` (behavior identical) + tests for RESP2 bulk and RESP3
  Int/Double shapes and the 1.0-score→distance conversion.

Verified on redis 1.3.0 against real servers: make check (fmt + strict clippy,
0 warnings), 49 lib + 100 bin unit tests (+15 new), 15/15 redis+vectorsets and
10/10 valkey integration tests (redis 8.8.0 / valkey-bundle latest).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(clippy): move vectorsets vsim test module to end of file

CI runs `clippy --all-targets -- -D warnings`, which flagged
items-after-test-module: the new vsim_parse_tests module was placed mid-file
(other fns followed it). Moved it to the end of the file. No logic change.

(make check uses non-strict clippy locally, which is why this slipped through;
verified now with the exact CI command.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(redis,valkey): parse FT.SEARCH under RESP3 (map) as well as RESP2 (array) (#100)

The engines connect with RESP2 by default, but a user can negotiate RESP3
(REDIS_URI=redis://host/?protocol=resp3). Under RESP3, FT.SEARCH returns a map
`{results:[{id, extra_attributes:{vector_score}}], ...}` — a completely
different shape from the RESP2 flat array `[count, id, fields, ...]`. The parser
only understood RESP2, so a RESP3 connection silently produced wrong results
(verified: recall 1.000 on RESP2 → 0.100 on RESP3 for redis/random-100).

Fix: parse_ft_search_response now takes the raw `redis::Value` and dispatches —
`Array` → RESP2, `Map` → RESP3 (reads `results[].id` + `extra_attributes.
vector_score`) — in both redis.rs and valkey.rs. The FT.SEARCH query now fetches
`redis::Value` instead of `Vec<redis::Value>` so both shapes deserialize.
Valkey gains a `VALKEY_PROTOCOL=resp3` switch (via a shared url suffix) so it,
too, can be driven over RESP3.

Coverage (the "does RESP3 work + filtered search" question):
- Unit tests for both the RESP2 array and RESP3 map shapes (redis + valkey).
- Integration tests running the filtered `match_any` search over RESP3:
  test_binary_redis_match_any_resp3 and test_binary_valkey_match_any_resp3.

Verified end-to-end on real servers: recall now 1.000 under BOTH RESP2 and RESP3
(redis 8.8.0 unfiltered), and >=0.9 for filtered match_any under RESP3 on both
engines. make check (fmt + strict clippy) clean; 102 bin unit tests; redis 16/16
and valkey 11/11 integration.

Note: VectorSets (VSIM) is RESP2-only by design (no protocol switch); its parser
was not extended.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(plot): --plot renders a QPS-vs-precision trade-off chart (SVG) (#101)

Adds a chart-export command (parity with the upstream Python tool's tradeoff
plots, which the Rust rewrite lacked). `--plot <out.svg>` reads the existing
`*-summary.json` result files, filters by --engines/--datasets, and writes a
self-contained SVG — one colored series per engine, precision on X, QPS on Y
("up and to the right is better"). No plotting dependency: the SVG is built by
hand (axes, gridlines, per-series polyline+points, legend, XML-escaped labels).

New src/bin/vector_db_benchmark/plot.rs + `plot: Option<String>` CLI flag +
main.rs dispatch (renders instead of benchmarking when set).

Verified: make check (fmt + strict clippy) clean; 4 unit tests (precision_summary
+ search_results parsing, SVG well-formedness + XML escaping, empty input);
generated a valid 2978-byte SVG end-to-end from real redis/glove-25 summaries.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(redis,valkey): bool/uuid/full-text/datetime filter datatypes (#102)

* feat(redis,valkey): bool/uuid/full-text/datetime filter datatypes

Add filter-datatype coverage matching (and exceeding) upstream qdrant
vector-db-benchmark for the redis and valkey engines:

- Schema map: bool/uuid -> TAG (SORTABLE UNF like keyword), datetime ->
  NUMERIC. Redis text -> TEXT; Valkey text -> tokenised TAG (no TEXT type).
- build_exact_match_filter: bool -> TAG match on "true"/"false".
- build_filter: full-text {"match":{"text":...}} branch. Redis emits
  @field:($p) over a TEXT field; Valkey degrades to a single-term TAG match.
- insert_number_param / build_range_filter: datetime bounds accept BOTH
  ISO-8601 strings (parsed to epoch seconds via chrono) AND numeric epochs
  (upstream is ISO-only).
- Schema-aware upload: datetime ISO values -> epoch seconds on HSET (both
  engines); Valkey text values -> whitespace-tokenised ';'-joined TAG set.
- Never-match safety for empty full-text queries (no silent full-scan).

Tests: unit tests for every new parser/upload branch in redis.rs and
valkey.rs; per-type compound-dataset fixtures in tests/common/mod.rs
(write_bool/uuid/fulltext/datetime_project) + test_binary_{redis,valkey}_*
recall tests (>= 0.9).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(valkey): inline NUMERIC range bounds; emit datetime epochs as Int

Local integration testing (which the implementation agent couldn't run) caught
two datetime issues:
- Valkey Search does NOT substitute $param inside NUMERIC range brackets
  (`@ts:[$p +inf]` → 'Invalid number'); only literals are accepted. Rewrote
  valkey build_range_filter to INLINE bounds (redis keeps params — they work
  there). Also made gt correctly exclusive.
- Datetime bounds now emit as integer epochs (whole seconds) in both engines.

Verified: clippy --all-targets -D warnings clean; 124 unit; redis 20/20 and
valkey 15/15 integration (incl. all bool/uuid/fulltext/datetime filter tests).

* review: apply 7-agent adversarial findings on filter datatypes

Confirmed defects from the 7-reviewer adversarial pass, all fixed + tested:
- #1 (HIGH) redis range `gt` was emitted INCLUSIVE (`[$p +inf]`, identical to
  `gte`) — now correctly EXCLUSIVE (`[($p +inf]`). Silent over-inclusion +
  redis/valkey divergence; was untested. Added a gt-exclusive/gte-inclusive test.
- #2 (HIGH) valkey exact keyword/uuid match inlined the raw value with NO
  escaping (unlike its match_any / text paths) — a value with a TAG metachar
  (| { } ( ) space \) → malformed query / wrong docs, error silently dropped.
  Now escapes via escape_tag_value.
- #3 (MED) valkey numeric exact-match and int match_any still used `$param`
  inside NUMERIC brackets — the exact construct valkey rejects ("Invalid
  number"), so they'd error on int/float fields. Now inlined literals.
- #4 (MED) datetime robustness: datetime_to_epoch_secs now also accepts naive
  (no-offset) and date-only bounds (interpreted UTC) — better than upstream's
  RFC-3339-only; and redis build_range_filter only emits a clause when the bound
  actually parsed (no dangling `$param` → FT.SEARCH hard-error). Tests added.

Added gate tests (gt-exclusive, datetime tolerance, no-dangling-param, two-field
AND). Deferred to follow-up (per synthesis, narrow CLI combo): #5 — priming
datetime_fields/text_tag_fields on the `--skip-upload` (non-skip-index) path.

Verified: clippy --all-targets -D warnings clean; 130 unit; redis 20/20, valkey
15/15 integration.

* style: cargo fmt the added filter tests

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(multi-tenancy): random-768-25-tenants scenario for redis & valkey (#103)

* feat(multi-tenancy): random-768-25-tenants scenario for redis & valkey

Add multi-tenancy benchmarking matching upstream qdrant/vector-db-benchmark's
random-768-*-tenants scenario: many tenants share ONE index and every search is
scoped to a single tenant via a keyword-equality filter on a `tenant` field,
with recall measured against the nearest neighbours WITHIN that tenant only.
Reuses the existing keyword-TAG filter path — no new engine parser code.

- tests/common: generalise write_filter_project into write_filter_project_multi
  (per-query condition + predicate) and add write_tenant_project fixture
  (25 tenants over 400 docs → 16 docs/tenant, each query targets one tenant).
  Add read_recalls() to read the per-query recalls array.
- integration tests: test_binary_{redis,valkey}_tenancy assert mean recall >= 0.9
  AND every per-query recall >= 0.9 (tenant-isolation: a leaked cross-tenant doc
  can't count toward tenant-local ground truth). Works over RESP2 and RESP3.
- datasets.json: register random-768-25-tenants (schema tenant:keyword), marked
  locally-generated (no download link) so auto-download of other datasets is
  unaffected.
- README: add a Multi-tenancy section.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(multi-tenancy): 7-agent review fixes — exact recall, all-tenant coverage, honest docs

Applies the confirmed findings from the 7-subagent adversarial review of the
multi-tenancy feature:

- Assert EXACT per-query recall (== 1.0, via > 0.999) instead of >= 0.9 in
  both redis and valkey tenancy tests. Search is exact (~16 docs/tenant), so a
  correct engine scores 1.0 on every query; any single cross-tenant leak now
  fails the check. Doc comments updated to match.
- Query EVERY tenant, not just the first 10: write_filter_project_multi gains an
  n_queries param; write_tenant_project passes N_TENANTS (25) so all 25 tenant
  labels — including two-digit ones — are exercised as query scopes. Existing
  single-condition filter fixtures pass N_QUERIES (unchanged behaviour).
- Drop the fabricated "vector_count": 1000000 from the random-768-25-tenants
  dataset registry entry (it is a locally-generated placeholder, not a 1M-row
  public dataset) and reword its description accordingly.
- README: correct "All datasets are automatically downloaded" (the tenancy
  dataset is a local placeholder) and soften the recall-as-isolation claim
  (recall is a strong signal, not a proof of zero leakage).

Verified: fmt, clippy --all-targets -D warnings, 130 unit tests, full redis
(9) + valkey (7) integration suites, tenancy 25/25 queries at recall 1.0.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(qdrant): hybrid dense+sparse RRF-fusion search (#104)

* feat(qdrant): hybrid (dense+sparse) RRF-fusion search path

Add an end-to-end HYBRID benchmark path for Qdrant that fuses a dense-vector
prefetch and a sparse-vector prefetch server-side via reciprocal-rank fusion
(RRF), without touching the Engine trait — dispatched on dataset.is_hybrid()
alongside the existing sparse branch.

- dataset.rs: is_hybrid(), read_hybrid_data(), read_hybrid_queries() over a
  type:"hybrid" dir (dense vectors.npy/queries.npy + sparse data.csr/queries.csr
  + shared neighbours.jsonl); reuses existing npy + CSR readers. Unit tests added.
- engine/qdrant.rs: create_hybrid_collection (named "dense" + "sparse" vectors),
  upload_hybrid (NamedVectors with both), and a hybrid search issuing two
  prefetches fused with Fusion::Rrf. Dense-only / sparse-only paths unchanged.
- tests/common: extract reusable write_sparse_project + add write_hybrid_project
  (deterministic StdRng; planted mutual-top-K ground truth; documented 0.9 floor).
- integration_qdrant.rs: sparse test now consumes write_sparse_project; add
  test_binary_qdrant_hybrid. Both assert their floor (observed 1.000).
- experiments/configurations/qdrant-hybrid.json: runnable hybrid config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(qdrant hybrid): address adversarial review — fusion-only fixture + 5 hardening fixes

Finding 1 (CRITICAL): redesign the hybrid fixture so ground truth is
recoverable ONLY by fusion. R is split into R_dense (dense-nearest, moderate
sparse) and R_sparse (sparse-top, moderate dense); dense-only (D_d) and
sparse-only (D_s) distractor rings make each modality individually return the
wrong top-K (recall ~0.5), while RRF gives every R doc two rank contributions
(one < K/2) so the fused top-K is exactly R for any k>=0. Adds a dense-only
jsonl VIEW + a NEGATIVE-CONTROL run asserting plain dense recall < 0.6 and
fusion beating it by a wide margin (observed: hybrid 1.000, dense-only 0.500).

Finding 2: create_hybrid_collection now honours collection_params
optimizers_config + quantization_config and throttles optimizer threads during
indexing, via helpers shared with the dense create path (was silently dropped).

Finding 3: read_hybrid_queries validates neighbours row count == query count
(was: OOB panic on a short neighbours.jsonl).

Finding 4: hybrid prefetch depth floored at top (pf_limit = max(cfg, top)) so
the RRF pool is never smaller than the requested result count.

Finding 5: neighbours.jsonl parsed strictly (read_neighbours_strict) — interior
blank/unparseable lines are rejected instead of silently shifting rows; only a
single trailing newline is tolerated. Unit tests added for 3/4/5.

Finding 6: README documents that qdrant-hybrid.json requires a type:"hybrid"
dataset and the exact on-disk layout (no fake dataset/link fabricated).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat(qdrant): product quantization + end-to-end quantization test coverage (#105)

* feat(qdrant): product quantization support + full quantization test coverage

Add PRODUCT quantization passthrough alongside the existing scalar/binary
arms in apply_optimizers_and_quantization: parse quantization_config.product
{compression, always_ram}, mapping the compression string (x4..x64) to the
CompressionRatio enum via a new parse_compression_ratio helper (clear Err on
unknown values, mirroring the scalar type-error style).

Add experiments/configurations/qdrant-single-node-pq-rps.json mirroring the
bq-rps grid with a product quantization_config + rescore/oversampling search
params.

Add end-to-end integration coverage: a new test runs the SAME generated
dataset (dim=64, 1000 docs, distinct queries) through SCALAR (int8), BINARY,
and PRODUCT quantization via the real CLI against live qdrant, each with
rescore+oversampling, asserting recall floors (all 0.9; observed scalar=1.000,
product=1.000, binary~0.99). Plus a unit test for the compression string->enum
mapping (valid + invalid).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* test(qdrant): give quantization test teeth, fixed seed, correct recall label

Address 3 adversarial-review findings on the new quantization integration test
(feature code in engine/qdrant.rs was verified correct, unchanged):

- FINDING 1 (teeth): the rescored modes test could pass even if the
  quantization_config were silently dropped (rescore re-ranks against full
  precision). The collection is dropped after each CLI run so config read-back
  is impossible; instead add test_binary_qdrant_quantization_is_applied, a
  NO-RESCORE negative control — full-precision baseline vs binary searched
  without rescore — asserting a >0.1 recall margin. Verified it fails closed:
  commenting out the quantization_config application makes binary-no-rescore
  recall jump from ~0.30 to 1.000 (margin 0.000) and the test goes RED.

- FINDING 2 (reproducibility): replace unseeded thread_rng with fixed-seed
  StdRng (matching tests/common convention) so corpus/queries/ground-truth and
  the recall floors are identical every run. Also fix the misleading "gaussian"
  comment — gen_range(-1.0..1.0) is UNIFORM.

- FINDING 3 (label): run_quantization_mode now reads mean_recall (via
  common::read_recall) so the "recall" labels match the measured metric.

Observed (fixed seed): scalar=1.000, product=1.000, binary=0.995-1.000;
teeth margin baseline(1.000) - binary_no_rescore(~0.29) = ~0.71.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* fix(milvus): escape exact-value strings via shared helper; document match_any single-valued limitation

FIX 1: factor quote_milvus_string() (backslash-then-doublequote escape, wraps
in double-quotes) and use it in BOTH the exact-value string branch and the
match_any string branch so a keyword value containing `"` or `\` (e.g.
15"laptop) no longer produces a malformed/injectable Milvus expression.

FIX 2: document on the match_any arm that Milvus match_any supports
SINGLE-VALUED keyword fields only; multi-valued `labels` contains-any is a
known limitation vs pgvector array-overlap / mongodb array $in (labels is
stored comma-joined as a single VarChar), tracked as follow-up. No storage
change shipped.

FIX 3: add unit tests asserting exact escaped output for both the match_any
and exact-value paths through the shared helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Weaviate Git Bot <83967968+weaviate-git-bot@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Andrew Kane <acekane1@gmail.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com>
Co-authored-by: filipe oliveira <filipecosta.90@gmail.com>
Co-authored-by: George <george.panchuk@qdrant.tech>
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: Eric O. Korman <eric.korman@gmail.com>
Co-authored-by: Anshu Avinash <anshu.avinash35@gmail.com>
Co-authored-by: Anshu Avinash <anshu.avinash@devrev.ai>
Co-authored-by: Dan <38852336+SebanDan@users.noreply.github.com>
Co-authored-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
Co-authored-by: tellet-q <elena.dubrovina@qdrant.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Andrey Vasnetsov <vasnetsov93@gmail.com>
Co-authored-by: LukasWestholt <support@lukaswestholt.de>
Co-authored-by: xzfc <xzfcpw@gmail.com>
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
Co-authored-by: Ubuntu <ubuntu@ip-172-31-46-179.eu-central-1.compute.internal>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Paulo Sousa <paulo.sousa@redis.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@IvanPleshkov IvanPleshkov IvanPleshkov approved these changes
@generall generall Awaiting requested review from generall
@KShivendu KShivendu Awaiting requested review from KShivendu
@tellet-q tellet-q Awaiting requested review from tellet-q
+1 more reviewer
@tellet tellet tellet approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@timvisee @filipecosta90 @KShivendu @tellet @IvanPleshkov