Warm serverless: stateless writers, manifest refresh, lazy attach (roadmap Phases 0-3) - #10
Closed
EdgarBabajanyan wants to merge 27 commits into
Closed
Warm serverless: stateless writers, manifest refresh, lazy attach (roadmap Phases 0-3)#10EdgarBabajanyan wants to merge 27 commits into
EdgarBabajanyan wants to merge 27 commits into
Conversation
Public plan for evolving Compass from a cloud-durable single-node engine
(v0.3.0) to a fully serverless database, in eight phases across four releases
— every item additive and Apache 2.0. v0.4.0 targets Phases 0-3 ("warm
serverless"): stateless writes, lazy attach, manifest refresh.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The object-storage feature and its real-S3 integration tests never ran in CI — only the default build was tested, so a cloud-path regression would merge green. Add a test-cloud job running `cargo test --features object-storage` against a MinIO service container with a pre-created bucket, which also exercises the env-gated s3_integration tests (server-side ETag CAS, LSM lifecycle with GC, concurrent appends) on every PR. Also add a dependency-free DCO check requiring Signed-off-by on every PR commit, ahead of external contributions. Neither job touches the required-checks list. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Cold rebuild fabricated collection metadata: vector-space specs were inferred
from recovered embeddings (model:"recovered"), created_at was reset to now(),
and CollectionConfig — including embed_model — was silently replaced with
defaults. A detached/recovered collection therefore lost user configuration.
- create_collection (cloud mode) now writes a create-only
{ns}/collection.json (name, created_at, vector spaces, default space, dims,
config) plus an EMPTY manifest, so zero-ingest collections are discoverable
from a fresh disk and stateless writers can validate against real config.
Bucket failures roll the local creation back; colliding with a pre-v0.4
namespace that has data is refused.
- rebuild_collection_from_storage reads the bucket config; the embedding
inference remains only as a pre-v0.4 fallback and back-fills the bucket
config create-only (organic migration, runs at most once).
- Vector-space CRUD (add/delete/set-default/mark-active) is bucket-first:
short-lock validate -> CAS the bucket config with revalidation against the
latest doc -> write-lock local apply. No S3 round-trip ever holds the
collections lock (existing codebase rule).
- All gated on cloud_mode: in local mode a Storage-issued collection.json
would collide with the real local metadata file.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Chunk ids were minted from a per-node next_id counter, so a write required
the stateful node that had the collection attached — and two nodes writing
one namespace would collide. Recovery makes the collision concrete: a
rebuilt node computes next_id = max_id+1, which can land inside another
writer's active block.
- storage/id_alloc.rs: {ns}/id-alloc holds the next block start; blocks of
max(10_000, batch) are claimed via the proven get_versioned/put_if_match
CAS loop. Concurrent claims receive disjoint ranges (tested with 16
racing claimants). Crashed writers leak at most their pooled ranges —
gaps are fine; the defended invariant is no-reuse, and replay ordering
comes from manifest seq, not ids.
- In cloud mode EVERY ingest path allocates from blocks (attached nodes
pool on the collection; refills never hold the collections lock across
the S3 round-trip). next_id becomes a diagnostic high-water mark. Local
mode is untouched.
- Seeding: create_collection seeds the allocator at 0; pre-v0.4 namespaces
migrate on first claim by seeding from the bucket-derived high-water
mark (create-only, race-safe). Rolling caveat documented in the roadmap:
do not run v0.3 and v0.4 writers against one bucket.
- COMPASS_ROLE=writer: durable-append-only node. Boots instantly (no local
collections, no recovery), validates dims against the cached bucket
config (re-fetching once on validation failure so a stale cache never
poisons a durable fragment), claims ids, appends ONE WAL fragment, and
returns. Deletes append tombstones (idempotent on replay); relation
creates store target_status "missing" (re-resolved at read time on
serving nodes); queries and delete-by-filter are refused with clear
errors. Consistency contract: durable immediately, searchable after a
serving node's refresh/attach.
Tests: writer end-to-end with zero local state; writer/attached id
disjointness; pre-v0.4 allocator migration; bucket-config recovery
(real model specs + created_at survive cold rebuild, replacing the
model:"recovered" inference); zero-ingest collections survive node loss.
Suites: 103 default / 133 object-storage, all green.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…n_seq Serving nodes never re-read the manifest after boot, so two nodes on one bucket diverged forever, and a stateless writer's fragments were invisible until a restart. - SeqTracker per collection: `contiguous` frontier + out-of-band set. A node's own appends apply locally ahead of remote fragments with earlier seqs, so a single watermark would skip those remote fragments forever; the out-of-band set closes that hole. All five local append sites thread their seq into the tracker, and a check-and-mark under the write lock resolves the race where the refresher applies a node's own fragment between its S3 append and lock reacquisition (replay is idempotent). - apply_fragment_locally: one shared replay path mirroring materialize's kind dispatch (data/tombstone/relation upsert+delete), reused by ingest, delete, and the refresher. Data replay skips already-present ids (so chunk_count can't double-count) and QUARANTINES wrong-dims embeddings with a loud error instead of corrupting the mmap vector file. - refresh_collection: poll the manifest, apply fragments past the frontier in seq order. Two-branch compaction rule: watermark within our frontier -> skip segments and replay the tail; watermark PAST our frontier means fragments we never saw were folded -> full re-attach. Background refresher task (COMPASS_REFRESH_INTERVAL, default 5s, 0 disables) holds a Weak so it dies with the manager. - applied_seq persisted in collection metadata: a persistent-disk restart knows how fresh its local state is and catches up the delta instead of rebuilding (may lag on relation-only applies; replay is idempotent). - Read-your-writes: ingest/delete responses carry the fragment seq; SearchRequest.min_seq refreshes-then-serves with a bounded (2s) wait, rejecting seqs beyond the write history. Tests: two-node convergence (chunks, deletes, relations); no double-apply of own writes; both compaction branches (stale node re-attaches, current node skips); cross-node min_seq read-your-writes. Suites: 103 default / 137 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…budget Cloud boot rebuilt EVERY collection from the bucket before serving — cold start was O(total data) and attached state was unbounded, which caps how many namespaces one worker can host. - COMPASS_LAZY_ATTACH: boot registers bucket namespaces (one delimiter listing) and attaches on first request. A request stampede on a cold namespace rebuilds ONCE via a per-namespace mutex; the global collections lock is never held across the rebuild. Collections created by OTHER nodes after boot attach on demand too (registry miss falls through to a bucket existence check). Default off — eager boot remains today's behavior. - COMPASS_MAX_ATTACHED: LRU budget. Past it, the least-recently-used collection detaches: removed from the map under the write lock, local files deleted only after the lock is released. Detach is safe because the bucket now carries everything (config included — the prior commit); the namespace stays registered and re-attaches on demand with all data. - Every entry point (search, ingest, deletes, relations, facets, vector- space CRUD, compact, get/list) ensures attachment first; list/get answer from bucket configs for registered-but-unattached namespaces (live counts are known only once attached — documented). - new_with_storage_opts: fully-explicit constructor (role, lazy, budget) so tests avoid process-global env races; env parsing stays in new_with_storage. Writer-role nodes skip local loading and recovery entirely — instant boot. Tests: lazy boot does not rebuild; 8-way attach stampede attaches once; LRU eviction at budget 1 with full-data re-attach; foreign creates attach on demand. Suites: 103 default / 140 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…ion bugs An independent review round on the warm-serverless work found three critical and six high-severity issues before merge. All fixed here, each with a regression test: - Config convergence (critical): vector-space changes are CAS'd into the bucket config, not written as fragments, so already-attached nodes never learned about them — and then QUARANTINED chunks carrying the new space (silent per-node data loss). refresh_collection now syncs the bucket config (spaces, default, CollectionConfig) before replaying fragments. - Mark-after-apply (critical): delete/relations/ingest recorded a fragment seq as applied BEFORE applying it, so a failed local apply was invisible to the refresher forever (a node would serve deleted data indefinitely). All sites now mark only after a successful apply; replay is idempotent. - Serialized destructive transitions (critical): refresh-triggered full re-attach and LRU eviction now take the same per-namespace attach mutex as ensure_attached — two rebuilds (or a rebuild racing a file deletion) can no longer run into the same live index directory. - Delete/recreate detection: a vanished manifest detaches the collection (instead of warning forever while serving dead data); a recreated collection (bucket created_at differs) forces a full re-attach. Local caches (writer pools, configs, registry, attach locks) purge on delete. - Writer hardening: deletes validate the namespace (a tombstone for a bogus name used to CREATE a phantom bucket namespace) and reject ids at/past the allocator frontier (a u64::MAX id would poison max_id -> next_id overflow / id reuse on every future rebuild). Writers refuse create_collection and vector-space CRUD with clear errors. - True LRU: last_used is now stamped on search/ingest/facets/relations — eviction previously keyed on attach order and evicted the HOTTEST collection under budget pressure (rebuild thrash). - Refresher efficiency: fragment refs are filtered by seq BEFORE payload fetches (a caught-up node downloads nothing per tick) and fragments apply under per-fragment lock holds, so a large backlog can't cause a node-wide read outage. Eviction during an ingest's S3-append gap no longer erases the (healthy, durable) batch — evicted is distinguished from deleted. - delete_collection works on lazy/evicted collections, never holds the global lock across S3, and purges all namespace caches. - Keymap persistence (pre-existing, exposed by block ids): sub-1000-vector collections never persisted the HNSW keymap, silently relying on identity key->id mapping that broke as soon as ids were non-dense. The keymap is now saved on every build and synthesized as identity for pre-fix dirs. - CI: DCO ignores merge commits; the MinIO job fails loudly if the integration tests were silently skipped (env-drift guard). New tests: SeqTracker unit semantics; true-LRU eviction; config propagation across nodes; ingest racing a refresher loop; persistent-disk restart catching up a writer's delta; wrong-dims quarantine without corruption; min_seq local-mode + boundary; writer phantom-namespace/absurd-id rejection; delete+recreate detection. Suites: 103 default / 149 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The bitnami/minio service-container tag didn't exist, and GitHub service containers can't override the image command that minio/minio requires (`server /data`). Start MinIO with docker run (same images as docker-compose.minio.yml), wait on its health endpoint, and create the bucket with mc — the silent-skip guard already fails the job if the integration tests don't actually run. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Three scale walls fell together because they shared one root: segments were a single JSON object produced by an O(collection) fold. - JSON segments encoded f32 embeddings as decimal text (~10x bloat) and were written with a single PUT (S3 caps those at 5GB) — compaction simply failed once the live set crossed a few GB, roughly 1-3M chunks. - Compaction materialized the ENTIRE collection into RAM every 32 fragments (O(collection) memory + IO per cycle; quadratic write amplification). Now: - Segment v2: magic-tagged sectioned binary — chunk metadata (JSON, sans embeddings), embeddings per space as raw f32 LE rows keyed by id, relations, cross-segment tombstone sections, max_id. v1 JSON and the oldest bare-array form still decode (versioned fallback); unknown sections are skipped for forward compat. - Storage::put_large: multipart upload on the object-store backend (16MB parts) past a single-PUT threshold; segments are immutable and UUID-keyed so no CAS token is needed on them. - Partitioned compaction: the routine cycle folds ONLY the WAL tail into an APPENDED segment — O(batch), not O(collection). Deletes that don't match anything within the tail are carried as segment tombstone sections and applied against older segments at materialize time (relation deletes too). A full merge into one segment runs only past 8 accumulated segments — the sole O(live-set) operation, 1/8th as often. GC discipline is unchanged: folded objects stage one cycle, CAS losers delete their orphan segment. Tests: v2 round-trip (multi-space embeddings, tombstones), cross-segment delete via a real two-fold sequence, GC bounds re-verified across 14 cycles crossing the merge threshold. Suites: 105 default / 151 object-storage. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…d HNSW saves Two more scale walls: every delete (and every replayed fragment) rebuilt the ENTIRE filter index, and every ingest batch rewrote the ENTIRE HNSW index file — both O(collection) per write, disqualifying past ~10M chunks. - FilterIndex: the numeric range index moves from a sorted Vec (O(N) per update) to a BTreeMap keyed by total-order-encoded f64 bits (O(log N) inserts/removes, range scans union pre-grouped treemaps). New remove() reverses insert() exactly, pruning empty entries. Ingest/replay now insert incrementally; deletes and ingest-failure compensation remove incrementally; full rebuilds remain only on attach/recovery. finalize() is a no-op kept for API compat; the (unwired) persistence codec is updated for the new layout. - HNSW: the in-RAM index stays mutable across batches (first mutation loads from disk once); the index FILE is rewritten every 16 batches instead of every batch. Vectors are already durable per batch in the mmap file, so a crash between saves leaves only a stale index — detected at load (index.size() < keymap length) and rebuilt from the mmap, then saved. Save-batching state lives on the collection, not the vector state. Suites: 105 default / 151 object-storage — including the delete/facet tests that verify incremental filter maintenance agrees with the old full rebuild. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
LoadedCollection held EVERY chunk — embeddings included — in a RAM HashMap, capping a serving node somewhere in the 5-20M chunk range regardless of compute. The bounded LRU read-through cache existed (search/chunk_cache.rs) but was never wired. - chunk_store is now the ChunkCache (bounded LRU over redb, default 100k resident): search assembly reads hits via get/get_batch; scoring metadata batches through it; parent enrichment reads through it; deletes fetch metadata for incremental filter-index removal through it. - Existence checks (relation target_status, delete eligibility, replay dedup) go through the filter index universe — a treemap of LIVE ids maintained incrementally on every path, so no chunk load is needed to answer "does id X exist". Replay dedup additionally consults tombstones (a deleted id must not be re-applied just because it left the universe). - Full scans (delete-by-filter, rebuild-job export, TAMS segment lookup) stream from redb via for_each instead of iterating a resident map. Boot rehydration builds the filter index streaming and keeps only the max-id scan; the chunks themselves stay on disk. The entire existing suite — search, relations, deletes, facets, TAMS, cloud convergence, lazy attach — passes unchanged on the out-of-core path: 105 default / 151 object-storage, clippy clean. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Operating multi-node deployments blind was a production blocker. /metrics (Prometheus text, unauthenticated like /health) exposes ingest/search/delete counters, refresh convergence (fragments applied, forced re-attaches), attach counts + cumulative time, compactions, and quarantined chunks, plus per-collection chunk-count and applied-seq gauges. Hand-rolled atomics — no new metrics dependency; call sites go through free functions so a real facade can replace the implementation later. COMPASS_MAX_CONCURRENCY bounds in-flight requests (tower global concurrency limit) instead of queueing without limit; unset = unlimited. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
COMPASS_SCALE_N gates a harness that ingests N synthetic chunks through the full cloud path (local-disk Storage backend — same code, disk-bound), measures ingest throughput, cold-attach time, and search latency, and skips cleanly when unset. First measured point (250k chunks, 128 dims, under ARM emulation): 1,603 chunks/s ingest, 134.5s cold attach, 5.5ms search. docs/scale-envelope.md records measured numbers only, names cold-attach as the binding constraint now that the RAM/segment/compaction/per-write walls are gone, and states plainly that billion-vector serving waits on Phase 6 serve-from-storage indexes. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…action safety An adversarial review of the five scale-wall commits found two criticals and a set of high/medium issues; all fixed: - Boot panic on DEFAULT env (critical): the concurrency layer passed usize::MAX/2 permits to tokio's Semaphore, whose hard cap is usize::MAX>>3 — the server asserted at startup whenever COMPASS_MAX_CONCURRENCY was unset. Capped under MAX_PERMITS. - HNSW hole (critical): a batch erroring after in-RAM adds but before a save left the next batch reloading a STALE index file, adding only new vectors, and saving — permanently missing up to 15 committed batches (recall silently degraded until a cold attach). The fresh-load path now detects index.size() < base_key and heals from the mmap before adding. - Compaction folds now use the STRICT fragment reader (the tolerant one skips NotFound — advancing the watermark past an unread fragment would be silent data loss), and the segment merge NEVER runs in the same invocation as a fold, restoring the one-cycle GC grace for readers holding the pre-fold manifest. - Stale-index rebuild truncates at the keymap length (orphan mmap tail rows can't be indexed under fabricated ids); ambiguous manifest-commit failures no longer delete the possibly-referenced new segment (only a definite CAS conflict does); skipped filter-index removals are logged loudly; /metrics reads attached collections only (it was an unauthenticated one-S3-GET-per-namespace-per-scrape amplifier in lazy mode); the scale harness exercises put_large. - fold_tail keeps carried tombstones on re-create (safe: materialize applies segment tombstones before the segment's own chunks) so folded and unfolded relation state can never diverge. Second measured point (500k chunks, 128 dims, emulated): 1,392 chunks/s ingest, 369.3s cold attach, 11.2ms search — attach scales linearly, as the envelope doc now states along with the merge/attach RAM caveat. Suites: 105 default / 152 object-storage, clippy clean. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
…clude deleted
The live E2E harness (scripts/e2e.sh, added here) caught facets returning
{} on a healthy collection. Three distinct bugs, all in the same design:
- build_index returned facet bitsets for the new batch only, and
apply_ingest_commit replaced the collection's facet state with them —
every ingest after the first wiped all prior facets (latent since v0.2).
- open_index returned empty facet state and nothing ever rebuilt it, so
facets were permanently empty after any restart or re-attach.
- The bitsets were dense arrays keyed by insertion position while the
query side intersected them by chunk id; id-block allocation (cloud
mode) breaks the dense-id assumption entirely.
Fix: facets are now HashMap<field, HashMap<value, RoaringTreemap>> keyed
by chunk id. Ingest absorbs prior state instead of replacing it; the
load/rebuild chunk scans rebuild facets in the same pass that builds the
filter index; and get_facets intersects each value's treemap with the
FilterIndex live-id universe, so tombstoned chunks stop inflating counts
(previously deleted chunks were counted until a full FTS rebuild).
scripts/e2e.sh is a 44-check live-stack harness covering every endpoint,
every filter operator, writer-role behavior, refresh visibility, and
compaction survival; it is what caught this.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The v0.3.0 comparison bench caught a warm-restart regression: 20.2s vs
1.1s at 100k chunks. Boot logs show why:
HNSW index at /app/data/benchcoll/vectors/default.index is stale
(97000 < 100000); rebuilding from mmap
Batched HNSW persistence (HNSW_SAVE_EVERY=16) means a clean shutdown can
leave the index file up to 15 batches behind the per-batch-durable mmap
and keymap — so nearly every warm restart of an actively-written
collection took the stale path, and that path re-inserted ALL vectors
(O(collection), ~20s per 100k under emulation).
The runtime heal in apply_ingest_commit already does this right: load
the existing graph and append only rows index.size()..keymap.len() from
the mmap. Mirror it at load time, then re-view the saved file so the
healed index stays mmap-backed instead of RAM-resident. The keymap is
persisted per batch, so the index file is only ever behind it, never
ahead — appending the tail is always sufficient.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Three independent audit passes (dead code, additive-ness, over-engineering) swept the codebase. This commit acts on the mechanical findings: Deleted (grep-proven zero call sites in any target): - BitSet dense-bitset facet structure — fully replaced by chunk-id-keyed RoaringTreemaps; the query side had already stopped using it - search/backend.rs (VectorIndex wiring, UsearchHnswIndex, build_backend, COMPASS_BACKEND env) — zero constructors anywhere; the live vector path calls usearch directly. With it go the compass-index-api dependency and the gpu feature, which gated code no runtime path could ever select. The compass-index-api / compass-vector-gpu workspace crates remain for a future real wiring but no longer ship weight into the binary - lsm::compact generic primitive — the live compaction path is read_uncompacted_fragments_strict + append_segment / replace_with_single_segment; its three tests are PORTED to those primitives (fold, watermark advance, strict-read abort on missing fragment), not dropped - filter.rs (matches_filters + 7 tests): delete_by_filter now resolves ids through the same roaring FilterIndex::eligible pushdown search uses, so search and delete-by-filter can never disagree on filter semantics - FilterExpr::eval + eval_predicate + stringify_metadata: the third, never-called filter evaluator - FilterIndex serialize/deserialize + MetadataKey codec (~200 lines of never-wired persistence scaffolding), FilterIndex::finalize() no-op and its call sites, FilterIndex::is_empty - save_vectors legacy writer (reader stays for migration), VectorState.dims and five FtsState field handles nobody read, three unread FilteredSearchExplain fields, SharedStorage alias, build_filter_index_from_chunks, RelationshipGraph::len - rayon dependency: its only use was current_num_threads() inside 128.max(...) — now std::thread::available_parallelism via one helper Fixed (dead-code finding that was actually a live bug): - mark_vector_space_active was never called: a completed vector-space rebuild updated only the in-RAM progress tracker — the space stayed status=building in collection metadata and the rebuilt index was never hot-loaded until restart. start_rebuild now takes the manager and calls it on completion; activation failure is reported as a failed rebuild - The unreachable filters branch in tantivy_fts::search (with its stale TODO) and the vestigial existing_count param of build_index are gone Lint hygiene: dead_code removed from the crate-level allow list — it had been suppressing ALL dead-code detection, which is how this pile accumulated. Test-only helpers are now cfg(test); the two deliberate API-surface items (Storage::get_range, ObjectMeta size/version — the sectioned-segment range-read contract) carry targeted allows with reasons. Net -1,178 lines. Suites: 94 local / 141 object-storage, clippy clean under -D warnings with the lint live. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
- local_mode_writes_no_wal now also asserts the id-block allocator object is never seeded locally and that ids stay dense next_id values across batches (the audit flagged both as untested local-mode invariants) - New writer_role_is_neutralized_in_local_mode: a stray COMPASS_ROLE=writer on a local-disk deployment must not disable serving — cloud_mode forces the role to Full; previously only code-verified - ARCHITECTURE.md data layout matched a format that no longer exists (meta.json/chunks.bin/index.usearch); now documents the real one (collection.json/chunks.redb/tantivy//vectors/<space>.*) plus the cloud bucket layout (manifest, wal/, segments/, id-alloc) - CLAUDE.md/README API tables gain GET /metrics and /segments/at; CLAUDE.md stops claiming a wired GPU feature (the crate is standalone until wired) - serverless-roadmap status: Phases 0-3 shipped on this branch - .env.example documents COMPASS_MAX_CONCURRENCY - Deleted an orphaned doc block and a stale 'Planned follow-up' note that described already-shipped filter pushdown Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Pure file move: 49% of the 7,118-line module was #[cfg(test)] code. Each module is now a child file (same module tree, so super::* keeps private access); mod.rs is 3,720 lines of engine code. No test changed: 142 pass. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Searching a missing collection returned HTTP 500 (api/search.rs mapped
every engine error to INTERNAL_SERVER_ERROR) and ingesting into one
returned 400 — stringly Box<dyn Error> carried no classification, and
three handlers (delete, relations, segments) each had their own copy of
a substring-sniffing mapper ('msg.contains("not found")').
All 32 not-found construction sites in the collection manager now build a
typed NotFound error; one shared api::error_response maps it to 404 by
downcast (no string sniffing), logs 500-class details server-side, and
keeps each handler's default for everything else. The three duplicate
mappers are gone.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The dead-code pass gated it #[cfg(all(test, feature))] but the
object-store backend calls it at RUNTIME (object_store_backend.rs:219).
No CI job compiles the non-test object-storage combination — only the
release Docker build does — so the gate slipped through green checks:
error[E0599]: no method named `is_empty` found for `&Version`
Gate is now #[cfg(any(test, feature = "object-storage"))]: present for
the backend and the integration tests, still dead-code-checked in the
default build.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Nothing in CI compiled the cloud feature without cfg(test) — the exact combination the release Docker image builds — so a mis-scoped cfg gate broke the image while every check stayed green. One cargo check line closes the gap. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
The any(test, feature) gate made it dead code in test-without-feature builds, failing clippy --all-targets and the default-feature test jobs. Every user (backend runtime + s3_integration tests) is behind the feature, so gate on the feature alone. All four build combinations verified: default test, feature test, feature non-test, clippy --all-targets. Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
This was referenced Jul 4, 2026
Contributor
Author
|
Shipped in v0.4.0 via the consolidated release PR #13 (all commits, authorship, and sign-offs from this branch landed there verbatim). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Compass v0.3.0 made object storage durable, but serving stayed fully stateful: every write required the node that had the collection attached, boot rebuilt every collection before serving (O(total data)), two nodes on one bucket diverged forever, and attached state was unbounded. None of that scales toward serverless.
Fix — the "warm serverless" milestone (roadmap Phases 0–3, all opt-in, local mode untouched)
COMPASS_ROLE=writernodes boot instantly, validate against the bucket's collection config, mint ids from CAS-leased blocks ({ns}/id-alloc— writers and attached nodes can never collide), append one durable WAL fragment, and return itsseq.{ns}/collection.jsonmakes specs/created_at/CollectionConfigdurable (cold rebuild previously fabricated them and silently lostembed_model); vector-space CRUD is bucket-first CAS.min_seqon search gives cross-node read-your-writes with a bounded wait.COMPASS_LAZY_ATTACHboots in O(namespaces) and attaches on first request (stampede-safe);COMPASS_MAX_ATTACHEDdetaches least-recently-used collections, which re-attach on demand.Adversarial review (fixed pre-PR, each with a regression test)
An independent review round found 3 critical + 6 high issues, all fixed in the final commit: config changes never converged to attached nodes (and quarantined their chunks — silent data loss); seqs were marked applied before the apply (a failed apply became permanently invisible); concurrent rebuilds could run into the same live index directory; "LRU" was actually FIFO-by-attach; writer deletes could create phantom namespaces and poison the id high-water mark; delete/recreate left stale nodes serving dead data. Also fixed a latent v0.2 bug the block allocator exposed: sub-1000-vector collections never persisted the vector keymap and silently served wrong chunk ids once ids were non-dense.
Testing
min_seqread-your-writes on the serving node → background-refresh convergence → writer delete replication → lazy node boots registering (not rebuilding) and attaches on demand in 0.11sNotes / follow-ups
min_seqcovers read-your-writescreated_at(documented in the roadmap as a v0.5 hardening item)