Release v0.4.0 — Serverless: stateless writers, tenant partitions, serve-from-storage - #13
Merged
Conversation
Public plan for evolving Compass from a cloud-durable single-node engine
(v0.3.0) to a fully serverless database, in eight phases across four releases
— every item additive and Apache 2.0. v0.4.0 targets Phases 0-3 ("warm
serverless"): stateless writes, lazy attach, manifest refresh.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The object-storage feature and its real-S3 integration tests never ran in CI — only the default build was tested, so a cloud-path regression would merge green. Add a test-cloud job running `cargo test --features object-storage` against a MinIO service container with a pre-created bucket, which also exercises the env-gated s3_integration tests (server-side ETag CAS, LSM lifecycle with GC, concurrent appends) on every PR. Also add a dependency-free DCO check requiring Signed-off-by on every PR commit, ahead of external contributions. Neither job touches the required-checks list. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Cold rebuild fabricated collection metadata: vector-space specs were inferred
from recovered embeddings (model:"recovered"), created_at was reset to now(),
and CollectionConfig — including embed_model — was silently replaced with
defaults. A detached/recovered collection therefore lost user configuration.
- create_collection (cloud mode) now writes a create-only
{ns}/collection.json (name, created_at, vector spaces, default space, dims,
config) plus an EMPTY manifest, so zero-ingest collections are discoverable
from a fresh disk and stateless writers can validate against real config.
Bucket failures roll the local creation back; colliding with a pre-v0.4
namespace that has data is refused.
- rebuild_collection_from_storage reads the bucket config; the embedding
inference remains only as a pre-v0.4 fallback and back-fills the bucket
config create-only (organic migration, runs at most once).
- Vector-space CRUD (add/delete/set-default/mark-active) is bucket-first:
short-lock validate -> CAS the bucket config with revalidation against the
latest doc -> write-lock local apply. No S3 round-trip ever holds the
collections lock (existing codebase rule).
- All gated on cloud_mode: in local mode a Storage-issued collection.json
would collide with the real local metadata file.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Chunk ids were minted from a per-node next_id counter, so a write required
the stateful node that had the collection attached — and two nodes writing
one namespace would collide. Recovery makes the collision concrete: a
rebuilt node computes next_id = max_id+1, which can land inside another
writer's active block.
- storage/id_alloc.rs: {ns}/id-alloc holds the next block start; blocks of
max(10_000, batch) are claimed via the proven get_versioned/put_if_match
CAS loop. Concurrent claims receive disjoint ranges (tested with 16
racing claimants). Crashed writers leak at most their pooled ranges —
gaps are fine; the defended invariant is no-reuse, and replay ordering
comes from manifest seq, not ids.
- In cloud mode EVERY ingest path allocates from blocks (attached nodes
pool on the collection; refills never hold the collections lock across
the S3 round-trip). next_id becomes a diagnostic high-water mark. Local
mode is untouched.
- Seeding: create_collection seeds the allocator at 0; pre-v0.4 namespaces
migrate on first claim by seeding from the bucket-derived high-water
mark (create-only, race-safe). Rolling caveat documented in the roadmap:
do not run v0.3 and v0.4 writers against one bucket.
- COMPASS_ROLE=writer: durable-append-only node. Boots instantly (no local
collections, no recovery), validates dims against the cached bucket
config (re-fetching once on validation failure so a stale cache never
poisons a durable fragment), claims ids, appends ONE WAL fragment, and
returns. Deletes append tombstones (idempotent on replay); relation
creates store target_status "missing" (re-resolved at read time on
serving nodes); queries and delete-by-filter are refused with clear
errors. Consistency contract: durable immediately, searchable after a
serving node's refresh/attach.
Tests: writer end-to-end with zero local state; writer/attached id
disjointness; pre-v0.4 allocator migration; bucket-config recovery
(real model specs + created_at survive cold rebuild, replacing the
model:"recovered" inference); zero-ingest collections survive node loss.
Suites: 103 default / 133 object-storage, all green.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…n_seq Serving nodes never re-read the manifest after boot, so two nodes on one bucket diverged forever, and a stateless writer's fragments were invisible until a restart. - SeqTracker per collection: `contiguous` frontier + out-of-band set. A node's own appends apply locally ahead of remote fragments with earlier seqs, so a single watermark would skip those remote fragments forever; the out-of-band set closes that hole. All five local append sites thread their seq into the tracker, and a check-and-mark under the write lock resolves the race where the refresher applies a node's own fragment between its S3 append and lock reacquisition (replay is idempotent). - apply_fragment_locally: one shared replay path mirroring materialize's kind dispatch (data/tombstone/relation upsert+delete), reused by ingest, delete, and the refresher. Data replay skips already-present ids (so chunk_count can't double-count) and QUARANTINES wrong-dims embeddings with a loud error instead of corrupting the mmap vector file. - refresh_collection: poll the manifest, apply fragments past the frontier in seq order. Two-branch compaction rule: watermark within our frontier -> skip segments and replay the tail; watermark PAST our frontier means fragments we never saw were folded -> full re-attach. Background refresher task (COMPASS_REFRESH_INTERVAL, default 5s, 0 disables) holds a Weak so it dies with the manager. - applied_seq persisted in collection metadata: a persistent-disk restart knows how fresh its local state is and catches up the delta instead of rebuilding (may lag on relation-only applies; replay is idempotent). - Read-your-writes: ingest/delete responses carry the fragment seq; SearchRequest.min_seq refreshes-then-serves with a bounded (2s) wait, rejecting seqs beyond the write history. Tests: two-node convergence (chunks, deletes, relations); no double-apply of own writes; both compaction branches (stale node re-attaches, current node skips); cross-node min_seq read-your-writes. Suites: 103 default / 137 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…budget Cloud boot rebuilt EVERY collection from the bucket before serving — cold start was O(total data) and attached state was unbounded, which caps how many namespaces one worker can host. - COMPASS_LAZY_ATTACH: boot registers bucket namespaces (one delimiter listing) and attaches on first request. A request stampede on a cold namespace rebuilds ONCE via a per-namespace mutex; the global collections lock is never held across the rebuild. Collections created by OTHER nodes after boot attach on demand too (registry miss falls through to a bucket existence check). Default off — eager boot remains today's behavior. - COMPASS_MAX_ATTACHED: LRU budget. Past it, the least-recently-used collection detaches: removed from the map under the write lock, local files deleted only after the lock is released. Detach is safe because the bucket now carries everything (config included — the prior commit); the namespace stays registered and re-attaches on demand with all data. - Every entry point (search, ingest, deletes, relations, facets, vector- space CRUD, compact, get/list) ensures attachment first; list/get answer from bucket configs for registered-but-unattached namespaces (live counts are known only once attached — documented). - new_with_storage_opts: fully-explicit constructor (role, lazy, budget) so tests avoid process-global env races; env parsing stays in new_with_storage. Writer-role nodes skip local loading and recovery entirely — instant boot. Tests: lazy boot does not rebuild; 8-way attach stampede attaches once; LRU eviction at budget 1 with full-data re-attach; foreign creates attach on demand. Suites: 103 default / 140 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…ion bugs An independent review round on the warm-serverless work found three critical and six high-severity issues before merge. All fixed here, each with a regression test: - Config convergence (critical): vector-space changes are CAS'd into the bucket config, not written as fragments, so already-attached nodes never learned about them — and then QUARANTINED chunks carrying the new space (silent per-node data loss). refresh_collection now syncs the bucket config (spaces, default, CollectionConfig) before replaying fragments. - Mark-after-apply (critical): delete/relations/ingest recorded a fragment seq as applied BEFORE applying it, so a failed local apply was invisible to the refresher forever (a node would serve deleted data indefinitely). All sites now mark only after a successful apply; replay is idempotent. - Serialized destructive transitions (critical): refresh-triggered full re-attach and LRU eviction now take the same per-namespace attach mutex as ensure_attached — two rebuilds (or a rebuild racing a file deletion) can no longer run into the same live index directory. - Delete/recreate detection: a vanished manifest detaches the collection (instead of warning forever while serving dead data); a recreated collection (bucket created_at differs) forces a full re-attach. Local caches (writer pools, configs, registry, attach locks) purge on delete. - Writer hardening: deletes validate the namespace (a tombstone for a bogus name used to CREATE a phantom bucket namespace) and reject ids at/past the allocator frontier (a u64::MAX id would poison max_id -> next_id overflow / id reuse on every future rebuild). Writers refuse create_collection and vector-space CRUD with clear errors. - True LRU: last_used is now stamped on search/ingest/facets/relations — eviction previously keyed on attach order and evicted the HOTTEST collection under budget pressure (rebuild thrash). - Refresher efficiency: fragment refs are filtered by seq BEFORE payload fetches (a caught-up node downloads nothing per tick) and fragments apply under per-fragment lock holds, so a large backlog can't cause a node-wide read outage. Eviction during an ingest's S3-append gap no longer erases the (healthy, durable) batch — evicted is distinguished from deleted. - delete_collection works on lazy/evicted collections, never holds the global lock across S3, and purges all namespace caches. - Keymap persistence (pre-existing, exposed by block ids): sub-1000-vector collections never persisted the HNSW keymap, silently relying on identity key->id mapping that broke as soon as ids were non-dense. The keymap is now saved on every build and synthesized as identity for pre-fix dirs. - CI: DCO ignores merge commits; the MinIO job fails loudly if the integration tests were silently skipped (env-drift guard). New tests: SeqTracker unit semantics; true-LRU eviction; config propagation across nodes; ingest racing a refresher loop; persistent-disk restart catching up a writer's delta; wrong-dims quarantine without corruption; min_seq local-mode + boundary; writer phantom-namespace/absurd-id rejection; delete+recreate detection. Suites: 103 default / 149 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The bitnami/minio service-container tag didn't exist, and GitHub service containers can't override the image command that minio/minio requires (`server /data`). Start MinIO with docker run (same images as docker-compose.minio.yml), wait on its health endpoint, and create the bucket with mc — the silent-skip guard already fails the job if the integration tests don't actually run. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Three scale walls fell together because they shared one root: segments were a single JSON object produced by an O(collection) fold. - JSON segments encoded f32 embeddings as decimal text (~10x bloat) and were written with a single PUT (S3 caps those at 5GB) — compaction simply failed once the live set crossed a few GB, roughly 1-3M chunks. - Compaction materialized the ENTIRE collection into RAM every 32 fragments (O(collection) memory + IO per cycle; quadratic write amplification). Now: - Segment v2: magic-tagged sectioned binary — chunk metadata (JSON, sans embeddings), embeddings per space as raw f32 LE rows keyed by id, relations, cross-segment tombstone sections, max_id. v1 JSON and the oldest bare-array form still decode (versioned fallback); unknown sections are skipped for forward compat. - Storage::put_large: multipart upload on the object-store backend (16MB parts) past a single-PUT threshold; segments are immutable and UUID-keyed so no CAS token is needed on them. - Partitioned compaction: the routine cycle folds ONLY the WAL tail into an APPENDED segment — O(batch), not O(collection). Deletes that don't match anything within the tail are carried as segment tombstone sections and applied against older segments at materialize time (relation deletes too). A full merge into one segment runs only past 8 accumulated segments — the sole O(live-set) operation, 1/8th as often. GC discipline is unchanged: folded objects stage one cycle, CAS losers delete their orphan segment. Tests: v2 round-trip (multi-space embeddings, tombstones), cross-segment delete via a real two-fold sequence, GC bounds re-verified across 14 cycles crossing the merge threshold. Suites: 105 default / 151 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…d HNSW saves Two more scale walls: every delete (and every replayed fragment) rebuilt the ENTIRE filter index, and every ingest batch rewrote the ENTIRE HNSW index file — both O(collection) per write, disqualifying past ~10M chunks. - FilterIndex: the numeric range index moves from a sorted Vec (O(N) per update) to a BTreeMap keyed by total-order-encoded f64 bits (O(log N) inserts/removes, range scans union pre-grouped treemaps). New remove() reverses insert() exactly, pruning empty entries. Ingest/replay now insert incrementally; deletes and ingest-failure compensation remove incrementally; full rebuilds remain only on attach/recovery. finalize() is a no-op kept for API compat; the (unwired) persistence codec is updated for the new layout. - HNSW: the in-RAM index stays mutable across batches (first mutation loads from disk once); the index FILE is rewritten every 16 batches instead of every batch. Vectors are already durable per batch in the mmap file, so a crash between saves leaves only a stale index — detected at load (index.size() < keymap length) and rebuilt from the mmap, then saved. Save-batching state lives on the collection, not the vector state. Suites: 105 default / 151 object-storage — including the delete/facet tests that verify incremental filter maintenance agrees with the old full rebuild. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
LoadedCollection held EVERY chunk — embeddings included — in a RAM HashMap, capping a serving node somewhere in the 5-20M chunk range regardless of compute. The bounded LRU read-through cache existed (search/chunk_cache.rs) but was never wired. - chunk_store is now the ChunkCache (bounded LRU over redb, default 100k resident): search assembly reads hits via get/get_batch; scoring metadata batches through it; parent enrichment reads through it; deletes fetch metadata for incremental filter-index removal through it. - Existence checks (relation target_status, delete eligibility, replay dedup) go through the filter index universe — a treemap of LIVE ids maintained incrementally on every path, so no chunk load is needed to answer "does id X exist". Replay dedup additionally consults tombstones (a deleted id must not be re-applied just because it left the universe). - Full scans (delete-by-filter, rebuild-job export, TAMS segment lookup) stream from redb via for_each instead of iterating a resident map. Boot rehydration builds the filter index streaming and keeps only the max-id scan; the chunks themselves stay on disk. The entire existing suite — search, relations, deletes, facets, TAMS, cloud convergence, lazy attach — passes unchanged on the out-of-core path: 105 default / 151 object-storage, clippy clean. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Operating multi-node deployments blind was a production blocker. /metrics (Prometheus text, unauthenticated like /health) exposes ingest/search/delete counters, refresh convergence (fragments applied, forced re-attaches), attach counts + cumulative time, compactions, and quarantined chunks, plus per-collection chunk-count and applied-seq gauges. Hand-rolled atomics — no new metrics dependency; call sites go through free functions so a real facade can replace the implementation later. COMPASS_MAX_CONCURRENCY bounds in-flight requests (tower global concurrency limit) instead of queueing without limit; unset = unlimited. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
COMPASS_SCALE_N gates a harness that ingests N synthetic chunks through the full cloud path (local-disk Storage backend — same code, disk-bound), measures ingest throughput, cold-attach time, and search latency, and skips cleanly when unset. First measured point (250k chunks, 128 dims, under ARM emulation): 1,603 chunks/s ingest, 134.5s cold attach, 5.5ms search. docs/scale-envelope.md records measured numbers only, names cold-attach as the binding constraint now that the RAM/segment/compaction/per-write walls are gone, and states plainly that billion-vector serving waits on Phase 6 serve-from-storage indexes. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…action safety An adversarial review of the five scale-wall commits found two criticals and a set of high/medium issues; all fixed: - Boot panic on DEFAULT env (critical): the concurrency layer passed usize::MAX/2 permits to tokio's Semaphore, whose hard cap is usize::MAX>>3 — the server asserted at startup whenever COMPASS_MAX_CONCURRENCY was unset. Capped under MAX_PERMITS. - HNSW hole (critical): a batch erroring after in-RAM adds but before a save left the next batch reloading a STALE index file, adding only new vectors, and saving — permanently missing up to 15 committed batches (recall silently degraded until a cold attach). The fresh-load path now detects index.size() < base_key and heals from the mmap before adding. - Compaction folds now use the STRICT fragment reader (the tolerant one skips NotFound — advancing the watermark past an unread fragment would be silent data loss), and the segment merge NEVER runs in the same invocation as a fold, restoring the one-cycle GC grace for readers holding the pre-fold manifest. - Stale-index rebuild truncates at the keymap length (orphan mmap tail rows can't be indexed under fabricated ids); ambiguous manifest-commit failures no longer delete the possibly-referenced new segment (only a definite CAS conflict does); skipped filter-index removals are logged loudly; /metrics reads attached collections only (it was an unauthenticated one-S3-GET-per-namespace-per-scrape amplifier in lazy mode); the scale harness exercises put_large. - fold_tail keeps carried tombstones on re-create (safe: materialize applies segment tombstones before the segment's own chunks) so folded and unfolded relation state can never diverge. Second measured point (500k chunks, 128 dims, emulated): 1,392 chunks/s ingest, 369.3s cold attach, 11.2ms search — attach scales linearly, as the envelope doc now states along with the merge/attach RAM caveat. Suites: 105 default / 152 object-storage, clippy clean. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…clude deleted
The live E2E harness (scripts/e2e.sh, added here) caught facets returning
{} on a healthy collection. Three distinct bugs, all in the same design:
- build_index returned facet bitsets for the new batch only, and
apply_ingest_commit replaced the collection's facet state with them —
every ingest after the first wiped all prior facets (latent since v0.2).
- open_index returned empty facet state and nothing ever rebuilt it, so
facets were permanently empty after any restart or re-attach.
- The bitsets were dense arrays keyed by insertion position while the
query side intersected them by chunk id; id-block allocation (cloud
mode) breaks the dense-id assumption entirely.
Fix: facets are now HashMap<field, HashMap<value, RoaringTreemap>> keyed
by chunk id. Ingest absorbs prior state instead of replacing it; the
load/rebuild chunk scans rebuild facets in the same pass that builds the
filter index; and get_facets intersects each value's treemap with the
FilterIndex live-id universe, so tombstoned chunks stop inflating counts
(previously deleted chunks were counted until a full FTS rebuild).
scripts/e2e.sh is a 44-check live-stack harness covering every endpoint,
every filter operator, writer-role behavior, refresh visibility, and
compaction survival; it is what caught this.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The v0.3.0 comparison bench caught a warm-restart regression: 20.2s vs
1.1s at 100k chunks. Boot logs show why:
HNSW index at /app/data/benchcoll/vectors/default.index is stale
(97000 < 100000); rebuilding from mmap
Batched HNSW persistence (HNSW_SAVE_EVERY=16) means a clean shutdown can
leave the index file up to 15 batches behind the per-batch-durable mmap
and keymap — so nearly every warm restart of an actively-written
collection took the stale path, and that path re-inserted ALL vectors
(O(collection), ~20s per 100k under emulation).
The runtime heal in apply_ingest_commit already does this right: load
the existing graph and append only rows index.size()..keymap.len() from
the mmap. Mirror it at load time, then re-view the saved file so the
healed index stays mmap-backed instead of RAM-resident. The keymap is
persisted per batch, so the index file is only ever behind it, never
ahead — appending the tail is always sufficient.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Three independent audit passes (dead code, additive-ness, over-engineering) swept the codebase. This commit acts on the mechanical findings: Deleted (grep-proven zero call sites in any target): - BitSet dense-bitset facet structure — fully replaced by chunk-id-keyed RoaringTreemaps; the query side had already stopped using it - search/backend.rs (VectorIndex wiring, UsearchHnswIndex, build_backend, COMPASS_BACKEND env) — zero constructors anywhere; the live vector path calls usearch directly. With it go the compass-index-api dependency and the gpu feature, which gated code no runtime path could ever select. The compass-index-api / compass-vector-gpu workspace crates remain for a future real wiring but no longer ship weight into the binary - lsm::compact generic primitive — the live compaction path is read_uncompacted_fragments_strict + append_segment / replace_with_single_segment; its three tests are PORTED to those primitives (fold, watermark advance, strict-read abort on missing fragment), not dropped - filter.rs (matches_filters + 7 tests): delete_by_filter now resolves ids through the same roaring FilterIndex::eligible pushdown search uses, so search and delete-by-filter can never disagree on filter semantics - FilterExpr::eval + eval_predicate + stringify_metadata: the third, never-called filter evaluator - FilterIndex serialize/deserialize + MetadataKey codec (~200 lines of never-wired persistence scaffolding), FilterIndex::finalize() no-op and its call sites, FilterIndex::is_empty - save_vectors legacy writer (reader stays for migration), VectorState.dims and five FtsState field handles nobody read, three unread FilteredSearchExplain fields, SharedStorage alias, build_filter_index_from_chunks, RelationshipGraph::len - rayon dependency: its only use was current_num_threads() inside 128.max(...) — now std::thread::available_parallelism via one helper Fixed (dead-code finding that was actually a live bug): - mark_vector_space_active was never called: a completed vector-space rebuild updated only the in-RAM progress tracker — the space stayed status=building in collection metadata and the rebuilt index was never hot-loaded until restart. start_rebuild now takes the manager and calls it on completion; activation failure is reported as a failed rebuild - The unreachable filters branch in tantivy_fts::search (with its stale TODO) and the vestigial existing_count param of build_index are gone Lint hygiene: dead_code removed from the crate-level allow list — it had been suppressing ALL dead-code detection, which is how this pile accumulated. Test-only helpers are now cfg(test); the two deliberate API-surface items (Storage::get_range, ObjectMeta size/version — the sectioned-segment range-read contract) carry targeted allows with reasons. Net -1,178 lines. Suites: 94 local / 141 object-storage, clippy clean under -D warnings with the lint live. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
- local_mode_writes_no_wal now also asserts the id-block allocator object is never seeded locally and that ids stay dense next_id values across batches (the audit flagged both as untested local-mode invariants) - New writer_role_is_neutralized_in_local_mode: a stray COMPASS_ROLE=writer on a local-disk deployment must not disable serving — cloud_mode forces the role to Full; previously only code-verified - ARCHITECTURE.md data layout matched a format that no longer exists (meta.json/chunks.bin/index.usearch); now documents the real one (collection.json/chunks.redb/tantivy//vectors/<space>.*) plus the cloud bucket layout (manifest, wal/, segments/, id-alloc) - CLAUDE.md/README API tables gain GET /metrics and /segments/at; CLAUDE.md stops claiming a wired GPU feature (the crate is standalone until wired) - serverless-roadmap status: Phases 0-3 shipped on this branch - .env.example documents COMPASS_MAX_CONCURRENCY - Deleted an orphaned doc block and a stale 'Planned follow-up' note that described already-shipped filter pushdown Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Pure file move: 49% of the 7,118-line module was #[cfg(test)] code. Each module is now a child file (same module tree, so super::* keeps private access); mod.rs is 3,720 lines of engine code. No test changed: 142 pass. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Searching a missing collection returned HTTP 500 (api/search.rs mapped
every engine error to INTERNAL_SERVER_ERROR) and ingesting into one
returned 400 — stringly Box<dyn Error> carried no classification, and
three handlers (delete, relations, segments) each had their own copy of
a substring-sniffing mapper ('msg.contains("not found")').
All 32 not-found construction sites in the collection manager now build a
typed NotFound error; one shared api::error_response maps it to 404 by
downcast (no string sniffing), logs 500-class details server-side, and
keeps each handler's default for everything else. The three duplicate
mappers are gone.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The dead-code pass gated it #[cfg(all(test, feature))] but the
object-store backend calls it at RUNTIME (object_store_backend.rs:219).
No CI job compiles the non-test object-storage combination — only the
release Docker build does — so the gate slipped through green checks:
error[E0599]: no method named `is_empty` found for `&Version`
Gate is now #[cfg(any(test, feature = "object-storage"))]: present for
the backend and the integration tests, still dead-code-checked in the
default build.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Nothing in CI compiled the cloud feature without cfg(test) — the exact combination the release Docker image builds — so a mis-scoped cfg gate broke the image while every check stayed green. One cargo check line closes the gap. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…on (Phase 6 core)
Create a collection with config.partition_by = "tenant_id" and every
chunk routes to an internal per-tenant namespace. A partition IS a full
collection under the hood — own LSM manifest/WAL/segments, Tantivy dir,
vector files, attach/evict lifecycle — so Phase 6 is a router at the
manager entry points over the existing engine, not new machinery. This
is what moves the scale envelope from per-COLLECTION to per-TENANT: a
query for tenant T attaches T's slice only, LRU eviction and refresh
scale with the hot-tenant set, and RAM is bounded regardless of how
many tenants (or how many billion vectors) the collection holds.
Mechanics:
- Partition namespace: {parent}--part--{value} (separator reserved at
create; partition values validated kebab-case, len <= 64). Namespaces
reuse every existing storage/path/attach rule unchanged.
- Ingest groups by metadata[partition_by] and delegates per group,
auto-creating partitions on first sight (inheriting the parent's
vector spaces + embed model). Writer role routes the same way,
bootstrapping partition bucket objects with create-only writes.
- Chunk ids are COLLECTION-unique: partitions claim CAS-leased blocks
from the PARENT's allocator — in local mode too (the allocator is a
CAS-updated file; no WAL/manifest objects locally). Without this,
delete-by-id and search results would be ambiguous across tenants.
- Search and delete-by-filter require a filter on the partition field:
exact match routes to one partition, set membership fans out (<=16,
merged by score). A tenant with no data is empty, not an error.
Bare-id deletes are rejected with guidance (ids don't name their
partition; probing every partition is unbounded).
- Partitions attach on demand even in non-lazy cloud mode (a writer can
mint one at any time — 'all attached at boot' can never hold), are
hidden from listings, and are cascade-deleted with the parent from
disk, registry, and bucket.
- Fenced with clear errors until routed: relations, facets, TAMS
lookup, vector-space CRUD, min_seq across a multi-partition fan-out.
Writer-role fences consult the bucket config, so a writer can't
durably append relations/tombstones into the parent namespace that no
serving node ever materializes.
Tests (11 new): tenant isolation on shared query terms, id uniqueness
under interleaved multi-tenant ingest, filter-routed deletes, listing/
cascade behavior, fence + validation errors, restart persistence,
writer-role ingest visible on a serving node that predates the
partitions, and cold rebuild of a partitioned collection from the
bucket alone (ids keep minting without reuse). Suites: 105 local / 153
object-storage, clippy clean, all four build combinations verified.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Warm serverless had one wall left between it and true serverless: a query on an unattached collection paid a full index rebuild first — minutes at millions of vectors. This makes cold namespaces answer semantic queries directly from object storage in a handful of small range reads. Segment format CSEG0003 (v2 stays readable; pre-v0.5 readers fail loudly on v3 instead of silently dropping sections): - meta2/metaidx: per-chunk JSON rows + a sorted (id, offset, len) index so any chunk hydrates with one byte-range read - cent:/clu: per vector space past 5k rows: k-means centroids + a cluster directory (tiny, cacheable) and the vectors grouped by cluster, unit-normalized (cosine == dot). Below the threshold the flat emb: section remains and is brute-forced whole. - Compaction builds the clusters (deterministic sampled Lloyd's, sqrt(n) clusters capped at 4096) — background CPU, off the write path. Cold query path (search/cold.rs), COMPASS_COLD_SERVE=true (implies lazy attach; cloud only): - manifest read -> cached per-segment artifacts (TOC, centroids, tombstones, metadata index; immutable, keyed by segment id) -> nprobe nearest clusters fetched concurrently -> WAL tail brute-forced fresh -> newest-generation dedupe -> top candidates hydrated by byte range -> metadata filters -> top_k. - Freshness: the manifest is read per query, so cold reads see every committed write including the tail — read-your-writes holds by construction (min_seq validated against next_seq). - Composes with tenant partitions: partition namespaces cold-serve through the same router; isolation verified. - FTS on a cold namespace is a clear error (inverted indexes still need an attach); hybrid degrades the same way. - Warm promotion: COMPASS_WARM_AFTER cold hits (default 3) spawn a background attach so hot namespaces migrate to the fast path. - RAM per cold namespace: centroids + directories + metadata index — megabytes, independent of collection size. Also: partition_field now reads the bucket config instead of forcing an attach (routing metadata must not warm anything); futures becomes an unconditional dep (cold path uses concurrent range reads). Tests: IVF roundtrip + self-recall, CSEG0003 clustered roundtrip + v2 compat, cold search without attach (self-recall through clusters, tail visibility, tombstone exclusion, filters, FTS fence), partition composition, warm promotion. Suites: 106 local / 159 object-storage, clippy clean, all four build combinations verified. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The 500k live proof caught it: with compaction never having run (the ingest node was OOM-killed mid-run), the cold path brute-forced a 422-fragment tail EVERY query — 2.4s latency and 4.7GiB RSS, silently. A healthy namespace keeps its tail under the auto-compact threshold (32); past 2x that, cold serving now fails loudly with a compact-first hint rather than materializing the dataset per query. With the guard in and compaction run properly, the 300k proof shows the intended shape: fresh node, empty disk, boot 0.3s at 28MiB RSS, FIRST query 70ms (vs minutes of attach), steady-state p50 25ms at 66MiB RSS, filters correct. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Optional COLD= node section: answers without attach, filters apply, FTS rejected with guidance, writer-tail read-your-writes visible instantly on the cold node, metrics counting. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…ngest, error propagation Two independent audit passes (adversarial correctness + lean/pork) over the Phase 5+6 diff. Fixes, by severity: CRITICAL — cold reads applied every segment's tombstones to ALL segments; materialize() applies a segment's carried tombstones to OLDER segments only, so a chunk re-ingested after its delete folded would be served warm but permanently suppressed cold. Tombstones are now per-generation (a candidate dies only to a tombstone from a NEWER generation; tail replay stays seq-ordered). Regression test covers both directions. HIGH — ensure_partition required the parent to be ATTACHED, so first ingest of a new tenant failed on lazy/cold-serve nodes (exactly the node type Phase 5 creates); the partition template now falls back to the bucket config. And partition_field / is_partitioned_any_role swallowed transient storage errors as 'not partitioned', which would misroute tenant writes (or writer tombstones) into the parent namespace — storage errors now propagate; only genuine NotFound means unpartitioned. MEDIUM — warm promotion un-latched (>= + reset) so an evicted namespace can warm again; non-lazy LRU eviction restricted to partition namespaces (an evicted normal collection could never re-attach there); cold path now REJECTS what it cannot honor (hybrid with text, recency, boosts, relationship options) instead of silently returning different rankings than warm; filtered cold queries overfetch 8x deeper with the recall contract documented; writer-side stale-config partition bootstrap re-validates the parent with a fresh read (no resurrecting cascade-deleted collections); config-less-but-data namespaces cold-serve via manifest fallback (warm/cold parity). LOW/lean — v2 segments rejected with an upgrade hint (previously read whole then dropped every hit at hydration); cold 'contains' filter now matches string lists only (FilterIndex parity); LocalDiskStorage get_range clamps range ends like S3/GCS (416 parity for start-past-EOF); metaidx bounds validated; paged metadata index now testable and tested; partition create races adopt the namespace instead of destroying its config; writer delete errors point to a serving node; local partitioned create rolls back on allocator seed failure; shared segment magic consts; renamed encode_segment_v3/decode_segment_sectioned; comment and Cargo.toml drift fixed. Suites: 109 local / 163 object-storage, clippy clean, four build combinations verified. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…call contract Evals: 20k docs, exact numpy ground truth, structured vs adversarial datasets. Warm HNSW 1.000 recall@10 (structured) / 0.895 (uniform); cold IVF reaches 1.000 at the default nprobe=8 on structured data and degrades steeply on uniform data (0.41@8) — inherent IVF behavior, with the exhaustive-probe row (1.000) proving the pipeline itself is exact. FTS: 50/50 exact-token top-1, topical precision@10 = 1.000. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…uth sweep
An external-perspective audit of the repo as a stranger would clone it.
The blockers, all fixed:
- Telemetry contradicted the product's core promise: README says 'fully
offline, data never leaves the machine' while a PostHog startup event +
daily heartbeat defaulted ON. Telemetry is now strictly OPT-IN
(COMPASS_TELEMETRY=on), DO_NOT_TRACK honored, README gains an honest
Telemetry section, .env.example documents exactly what is sent.
- cargo publish would hard-fail: no license field on any crate. Added
Apache-2.0 to [workspace.package], inherited by all three crates.
- The release workflow's version gate grepped crates/compass/Cargo.toml,
which says 'version.workspace = true' — every tag would fail. It now
reads the workspace manifest.
- CONTRIBUTING told strangers to run commands that fail on a fresh clone
(cargo test --workspace pulls the CUDA-only crate; --features gpu does
not exist). Corrected to the exact CI invocations + Linux prerequisites
+ the Windows linker caveat.
- README was stale on exactly what v0.4 ships: it claimed 'not stateless
multi-node serving' and never mentioned writers, partitions, or cold
serve. Added the serverless-topologies and multi-tenant sections, fixed
the quickstart (FTS works with zero downloads; semantic points at
scripts/download-models.sh), documented unauthenticated /metrics.
- CHANGELOG's Unreleased section contradicted itself ('warm, not cold' in
the same release that ships the cold path) and cited pre-v0.5.
Also: new docs/deployment.md (the three topologies + honest fleet gaps),
ARCHITECTURE.md purged of the removed backend-selector section and taught
the real module map + CSEG0003, scale-envelope updated for partitions +
cold reads, security contact unified (security@runcaptain.com — verify
the mailbox exists before release), SECURITY.md no longer claims runtime
model downloads, duplicate issue templates removed, broken doc link
dropped, mangled .env.example section headers fixed.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
- Workspace version 0.3.0 -> 0.4.0 (internal compass-index-api pin in lockstep) - CHANGELOG: stamp [Unreleased] as [0.4.0] - 2026-07-04 - Cargo.lock regenerated Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR is
The complete v0.4.0 release in one merge: it contains the full stacked feature set (#10 warm serverless → #11 tenant partitions → #12 serve-from-storage) plus the release commit (version bump to 0.4.0, CHANGELOG stamp, lockfile). Merging this ships everything; #10/#11/#12 can then be closed as superseded (their commits land via this PR, authorship and sign-offs intact).
What v0.4.0 ships
Warm serverless — object storage as the source of truth with disposable nodes: stateless writer role (
COMPASS_ROLE=writer, instant boot, append-only), CAS id-block allocator (no id collisions across nodes), bucket-durable collection config, background manifest refresher +min_seqread-your-writes, lazy attach + LRU detach.Tenant partitions —
"config": {"partition_by": "tenant_id"}routes every chunk to an internal per-tenant namespace behind one collection API. Ids stay collection-unique; partitions auto-create on first ingest (writers included), attach on demand, hide from listings, cascade-delete. Serving RAM tracks the hot-tenant set, not tenant count. Works fully in local mode.Serve-from-storage —
COMPASS_COLD_SERVE=true: semantic queries on never-attached collections answered from object-storage range reads (segment format CSEG0003: IVF-clustered vectors + row-addressable metadata). First cold query ~70ms vs minutes of index rebuild; read-your-writes by construction; warm promotion after repeated hits.Fixed along the way: facets broken since v0.2 (wiped per ingest, empty after restart, counted deleted chunks); O(collection) warm restarts (20.2s → 1.6s at 100k); vector-space rebuilds that never activated; 500s for missing collections (now 404); sub-1000-vector keymap bug; telemetry flipped to strictly opt-in.
Lean pass: ~1,200 lines of dead weight removed, dead_code lint re-enabled crate-wide, one filter semantics for search + deletes, docs truth-swept (README/ARCHITECTURE/CONTRIBUTING now describe v0.4 reality; new docs/deployment.md + docs/search-quality.md).
Evidence
Migration notes
COMPASS_TELEMETRY=on).After merge: tag
v0.4.0(the release workflow's version gate now reads the workspace manifest — fixed in this PR; it would have failed before).