Serve-from-storage: cold semantic queries without attaching (Phase 5) - #12
Closed
EdgarBabajanyan wants to merge 8 commits into
Closed
Serve-from-storage: cold semantic queries without attaching (Phase 5)#12EdgarBabajanyan wants to merge 8 commits into
EdgarBabajanyan wants to merge 8 commits into
Conversation
Warm serverless had one wall left between it and true serverless: a query on an unattached collection paid a full index rebuild first — minutes at millions of vectors. This makes cold namespaces answer semantic queries directly from object storage in a handful of small range reads. Segment format CSEG0003 (v2 stays readable; pre-v0.5 readers fail loudly on v3 instead of silently dropping sections): - meta2/metaidx: per-chunk JSON rows + a sorted (id, offset, len) index so any chunk hydrates with one byte-range read - cent:/clu: per vector space past 5k rows: k-means centroids + a cluster directory (tiny, cacheable) and the vectors grouped by cluster, unit-normalized (cosine == dot). Below the threshold the flat emb: section remains and is brute-forced whole. - Compaction builds the clusters (deterministic sampled Lloyd's, sqrt(n) clusters capped at 4096) — background CPU, off the write path. Cold query path (search/cold.rs), COMPASS_COLD_SERVE=true (implies lazy attach; cloud only): - manifest read -> cached per-segment artifacts (TOC, centroids, tombstones, metadata index; immutable, keyed by segment id) -> nprobe nearest clusters fetched concurrently -> WAL tail brute-forced fresh -> newest-generation dedupe -> top candidates hydrated by byte range -> metadata filters -> top_k. - Freshness: the manifest is read per query, so cold reads see every committed write including the tail — read-your-writes holds by construction (min_seq validated against next_seq). - Composes with tenant partitions: partition namespaces cold-serve through the same router; isolation verified. - FTS on a cold namespace is a clear error (inverted indexes still need an attach); hybrid degrades the same way. - Warm promotion: COMPASS_WARM_AFTER cold hits (default 3) spawn a background attach so hot namespaces migrate to the fast path. - RAM per cold namespace: centroids + directories + metadata index — megabytes, independent of collection size. Also: partition_field now reads the bucket config instead of forcing an attach (routing metadata must not warm anything); futures becomes an unconditional dep (cold path uses concurrent range reads). Tests: IVF roundtrip + self-recall, CSEG0003 clustered roundtrip + v2 compat, cold search without attach (self-recall through clusters, tail visibility, tombstone exclusion, filters, FTS fence), partition composition, warm promotion. Suites: 106 local / 159 object-storage, clippy clean, all four build combinations verified. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The 500k live proof caught it: with compaction never having run (the ingest node was OOM-killed mid-run), the cold path brute-forced a 422-fragment tail EVERY query — 2.4s latency and 4.7GiB RSS, silently. A healthy namespace keeps its tail under the auto-compact threshold (32); past 2x that, cold serving now fails loudly with a compact-first hint rather than materializing the dataset per query. With the guard in and compaction run properly, the 300k proof shows the intended shape: fresh node, empty disk, boot 0.3s at 28MiB RSS, FIRST query 70ms (vs minutes of attach), steady-state p50 25ms at 66MiB RSS, filters correct. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Optional COLD= node section: answers without attach, filters apply, FTS rejected with guidance, writer-tail read-your-writes visible instantly on the cold node, metrics counting. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…ngest, error propagation Two independent audit passes (adversarial correctness + lean/pork) over the Phase 5+6 diff. Fixes, by severity: CRITICAL — cold reads applied every segment's tombstones to ALL segments; materialize() applies a segment's carried tombstones to OLDER segments only, so a chunk re-ingested after its delete folded would be served warm but permanently suppressed cold. Tombstones are now per-generation (a candidate dies only to a tombstone from a NEWER generation; tail replay stays seq-ordered). Regression test covers both directions. HIGH — ensure_partition required the parent to be ATTACHED, so first ingest of a new tenant failed on lazy/cold-serve nodes (exactly the node type Phase 5 creates); the partition template now falls back to the bucket config. And partition_field / is_partitioned_any_role swallowed transient storage errors as 'not partitioned', which would misroute tenant writes (or writer tombstones) into the parent namespace — storage errors now propagate; only genuine NotFound means unpartitioned. MEDIUM — warm promotion un-latched (>= + reset) so an evicted namespace can warm again; non-lazy LRU eviction restricted to partition namespaces (an evicted normal collection could never re-attach there); cold path now REJECTS what it cannot honor (hybrid with text, recency, boosts, relationship options) instead of silently returning different rankings than warm; filtered cold queries overfetch 8x deeper with the recall contract documented; writer-side stale-config partition bootstrap re-validates the parent with a fresh read (no resurrecting cascade-deleted collections); config-less-but-data namespaces cold-serve via manifest fallback (warm/cold parity). LOW/lean — v2 segments rejected with an upgrade hint (previously read whole then dropped every hit at hydration); cold 'contains' filter now matches string lists only (FilterIndex parity); LocalDiskStorage get_range clamps range ends like S3/GCS (416 parity for start-past-EOF); metaidx bounds validated; paged metadata index now testable and tested; partition create races adopt the namespace instead of destroying its config; writer delete errors point to a serving node; local partitioned create rolls back on allocator seed failure; shared segment magic consts; renamed encode_segment_v3/decode_segment_sectioned; comment and Cargo.toml drift fixed. Suites: 109 local / 163 object-storage, clippy clean, four build combinations verified. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…call contract Evals: 20k docs, exact numpy ground truth, structured vs adversarial datasets. Warm HNSW 1.000 recall@10 (structured) / 0.895 (uniform); cold IVF reaches 1.000 at the default nprobe=8 on structured data and degrades steeply on uniform data (0.41@8) — inherent IVF behavior, with the exhaustive-probe row (1.000) proving the pipeline itself is exact. FTS: 50/50 exact-token top-1, topical precision@10 = 1.000. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…uth sweep
An external-perspective audit of the repo as a stranger would clone it.
The blockers, all fixed:
- Telemetry contradicted the product's core promise: README says 'fully
offline, data never leaves the machine' while a PostHog startup event +
daily heartbeat defaulted ON. Telemetry is now strictly OPT-IN
(COMPASS_TELEMETRY=on), DO_NOT_TRACK honored, README gains an honest
Telemetry section, .env.example documents exactly what is sent.
- cargo publish would hard-fail: no license field on any crate. Added
Apache-2.0 to [workspace.package], inherited by all three crates.
- The release workflow's version gate grepped crates/compass/Cargo.toml,
which says 'version.workspace = true' — every tag would fail. It now
reads the workspace manifest.
- CONTRIBUTING told strangers to run commands that fail on a fresh clone
(cargo test --workspace pulls the CUDA-only crate; --features gpu does
not exist). Corrected to the exact CI invocations + Linux prerequisites
+ the Windows linker caveat.
- README was stale on exactly what v0.4 ships: it claimed 'not stateless
multi-node serving' and never mentioned writers, partitions, or cold
serve. Added the serverless-topologies and multi-tenant sections, fixed
the quickstart (FTS works with zero downloads; semantic points at
scripts/download-models.sh), documented unauthenticated /metrics.
- CHANGELOG's Unreleased section contradicted itself ('warm, not cold' in
the same release that ships the cold path) and cited pre-v0.5.
Also: new docs/deployment.md (the three topologies + honest fleet gaps),
ARCHITECTURE.md purged of the removed backend-selector section and taught
the real module map + CSEG0003, scale-envelope updated for partitions +
cold reads, security contact unified (security@runcaptain.com — verify
the mailbox exists before release), SECURITY.md no longer claims runtime
model downloads, duplicate issue templates removed, broken doc link
dropped, mangled .env.example section headers fixed.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
EdgarBabajanyan
force-pushed
the
feat/serve-from-storage
branch
from
July 4, 2026 22:13
cb65e11 to
def0682
Compare
Contributor
Author
|
Shipped in v0.4.0 via the consolidated release PR #13 (all commits, authorship, and sign-offs from this branch landed there verbatim). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Warm serverless had one wall left: a query on an unattached collection paid a full local index rebuild first. Measured on this hardware, that's minutes at hundreds of thousands of vectors:
For a multi-tenant fleet that's the difference between "serverless" and "warm pool you must keep fed."
Design
Compaction now writes cold-servable segments (format
CSEG0003; v2 stays readable, pre-v0.5 readers fail loudly rather than silently dropping sections):cent:/clu:per vector space past 5k rows — k-means centroids + a cluster directory (tiny) and the vectors grouped by cluster, unit-normalized. Deterministic sampled Lloyd's at compaction time, off the write path (~6s at 300k).meta2/metaidx— per-chunk JSON rows + a sorted byte-range index, so any chunk hydrates with one range read.With
COMPASS_COLD_SERVE=true(implies lazy attach; cloud only), a semantic query on an unattached namespace runs: manifest read → cached per-segment artifacts (TOC, centroids, tombstones, meta index — immutable, cached by segment id) →nprobenearest clusters fetched concurrently → WAL tail brute-forced fresh → newest-generation dedupe → byte-range hydration → metadata filters → top-k.Properties worth calling out:
COMPASS_WARM_AFTERcold hits (default 3) spawn a background attach, so hot namespaces migrate to the ~1ms hot path on their own.Evidence
Live proof (MinIO, 300k × 64d, compacted):
Suites: 106 local / 159 object-storage (9 new: IVF self-recall + roundtrip, CSEG0003 clustered roundtrip + v2 compat, cold search without attach — tail visibility, tombstone exclusion, filters, FTS fence — partition composition, warm promotion). Clippy clean, all four build combinations verified.
Notes / follow-ups
COMPASS_COLD_NPROBE, default 8); self-recall is test-enforced ≥90% at nprobe 4 on clustered data. Uniform random data wants higher nprobe — document per workload.clu:stores vectors normalized; raw magnitudes are not preserved through a compact→decode cycle (cosine search is invariant; documented behavior).