Release v0.4.0 — Serverless: stateless writers, tenant partitions, serve-from-storage

.env.example

@@ -59,6 +59,42 @@ RUST_LOG=compass=info
 # AZURE_STORAGE_CONNECTION_STRING=
 # AZURE_STORAGE_SAS_KEY=
 
-# ── Telemetry (anonymous; opt out) ──────────────────────────────────────────
-# COMPASS_TELEMETRY=off
-# DO_NOT_TRACK=1
+# ── Warm serverless (cloud mode only) ───────────────────────────────────────
+# Node role: `full` (default — serve reads + writes with local indexes) or
+# `writer` (durable-append-only: no local indexes, no read serving, instant
+# boot). Writers validate against the bucket's collection config.
+# COMPASS_ROLE=full
+
+# Seconds between manifest refreshes (convergence with other nodes' writes).
+# Default 5; 0 disables the background refresher.
+# COMPASS_REFRESH_INTERVAL=5
+
+# Lazy attach: register bucket collections at boot and attach (rebuild local
+# indexes) on first request instead of eagerly. Default false.
+# COMPASS_LAZY_ATTACH=false
+
+# Max simultaneously-attached collections when lazy attach is on (LRU detach
+# past the budget; detached collections re-attach on demand). 0 = unbounded.
+# COMPASS_MAX_ATTACHED=0
+
+# ── Serve-from-storage (cold reads) ─────────────────────────────────────────
+# Serve-from-storage: semantic queries on UNATTACHED collections are answered
+# directly from object storage (a few range reads, ~100s of ms) instead of
+# waiting for a full index rebuild. Implies COMPASS_LAZY_ATTACH. Cloud only.
+# COMPASS_COLD_SERVE=true
+# Cold hits on a namespace before a background attach warms it (0 = never).
+# COMPASS_WARM_AFTER=3
+# IVF clusters probed per segment per cold query (recall/latency knob).
+# Default 8 reaches warm-parity recall on clustered embedding spaces; raise
+# it for unstructured vector data (see docs/search-quality.md).
+# COMPASS_COLD_NPROBE=8
+
+# Global in-flight request cap (backpressure). Unset = effectively unlimited.
+# COMPASS_MAX_CONCURRENCY=1024
+
+# ── Telemetry (anonymous; OPT-IN, off by default) ───────────────────────────
+# Compass never phones home unless you set this. When on, it sends a startup
+# event + daily heartbeat (random instance id, version, OS/arch, collection
+# and vector counts — never document content or queries). DO_NOT_TRACK=1 is
+# honored even when opted in.
+# COMPASS_TELEMETRY=on

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +1,5 @@
 blank_issues_enabled: true
 contact_links:
   - name: Security vulnerability
-    url: mailto:security@runcaptain.com
+    url: mailto:support@runcaptain.com
     about: Report security issues privately via email — do not open a public issue.

.github/workflows/ci.yml

@@ -4,7 +4,8 @@ on:
   push:
     branches: [main]
   pull_request:
-    branches: [main]
+    # feat/** so stacked PRs (feature targeting feature) get CI too.
+    branches: [main, "feat/**"]
 
 env:
   CARGO_TERM_COLOR: always
@@ -57,6 +58,71 @@ jobs:
           sudo apt-get install -y cmake pkg-config libssl-dev
       - run: cargo test --workspace --exclude compass-vector-gpu
 
+  # Cloud-feature coverage: the object-storage build + the real-S3 integration
+  # tests against MinIO. Not in the required-checks list (new job), but a
+  # failure here still blocks review attention.
+  test-cloud:
+    runs-on: ubuntu-24.04
+    env:
+      COMPASS_TEST_S3_BUCKET: compass-data
+      COMPASS_S3_ENDPOINT: http://localhost:9000
+      COMPASS_S3_ALLOW_HTTP: "true"
+      AWS_ACCESS_KEY_ID: minioadmin
+      AWS_SECRET_ACCESS_KEY: minioadmin
+      AWS_DEFAULT_REGION: us-east-1
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+      - run: |
+          sudo apt-get update
+          sudo apt-get install -y cmake pkg-config libssl-dev
+      # MinIO as a plain container (service containers can't override the
+      # image command, and minio/minio needs `server /data`). Same images as
+      # docker-compose.minio.yml.
+      - run: |
+          docker run -d --name minio -p 9000:9000 \
+            -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
+            minio/minio:latest server /data
+          for i in $(seq 1 30); do
+            curl -sf http://localhost:9000/minio/health/live && break
+            sleep 1
+          done
+          docker run --rm --network host --entrypoint sh minio/mc:latest -c \
+            "mc alias set local http://localhost:9000 minioadmin minioadmin && mc mb -p local/compass-data"
+      # NON-TEST compile of the cloud feature: no other job builds this
+      # combination (clippy/msrv build without the feature; tests build with
+      # cfg(test)), so a cfg gate that hides an item from the release cloud
+      # build otherwise sails through green checks and breaks docker builds.
+      - run: cargo check -p compass --features object-storage
+      # The s3_integration tests skip silently without the env; guard against
+      # env-name drift turning this job into a green no-op.
+      - run: |
+          cargo test -p compass --features object-storage -- --nocapture 2>&1 | tee /tmp/cloud-tests.log
+          if grep -q '^skipped: COMPASS_TEST_S3_BUCKET' /tmp/cloud-tests.log; then
+            echo '::error::s3_integration tests were skipped — MinIO env wiring is broken'
+            exit 1
+          fi
+
+  # Developer Certificate of Origin: every PR commit carries a Signed-off-by
+  # trailer. Dependency-free check over the PR range.
+  dco:
+    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - run: |
+          missing=0
+          for sha in $(git rev-list --no-merges ${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }}); do
+            if ! git log -1 --format=%B "$sha" | grep -q '^Signed-off-by: '; then
+              echo "::error::commit $sha is missing a Signed-off-by trailer (git commit -s)"
+              missing=1
+            fi
+          done
+          exit $missing
+
   msrv:
     runs-on: ubuntu-24.04
     steps:

.github/workflows/release.yml

@@ -27,7 +27,9 @@ jobs:
 
       - name: Verify tag matches Cargo.toml
         run: |
-          CARGO_VERSION=$(grep "^version" crates/compass/Cargo.toml | head -1 | sed 's/.*"\([^"]*\)".*/\1/')
+          # The version lives in [workspace.package] in the ROOT manifest;
+          # crate manifests say `version.workspace = true`.
+          CARGO_VERSION=$(grep "^version" Cargo.toml | head -1 | sed 's/.*"\([^"]*\)".*/\1/')
           if [ "${{ steps.version.outputs.version }}" != "$CARGO_VERSION" ]; then
             echo "Tag version ${{ steps.version.outputs.version }} does not match Cargo.toml version $CARGO_VERSION"
             exit 1

ARCHITECTURE.md

@@ -43,42 +43,50 @@ crates/compass/src/
     mod.rs           SearchMode enum + re-exports.
     backend.rs       VectorIndex trait shim. UsearchHnswIndex (CPU) lives here.
     vector.rs        USearch HNSW build + search + persistence (CPU primitives).
-    tantivy_fts.rs   Full-text search via Tantivy (BM25).
+    tantivy_fts.rs   Full-text search via Tantivy (BM25) + facet treemaps.
     hybrid.rs        Reciprocal Rank Fusion (RRF, k=60) over FTS + semantic.
+    ivf.rs           IVF clustering built at compaction (cold-read layout).
+    cold.rs          Serve-from-storage query path (range reads, no attach).
+    filter_index.rs  Roaring-treemap metadata filter index (warm pushdown).
+    chunk_store.rs / chunk_cache.rs   redb chunk store + bounded LRU cache.
+  collections/
+    partitions.rs    Tenant-partition routing helpers.
+    cloud.rs         Segment codec (CSEG0003), materialize, bucket config.
+  storage/           Storage trait + local disk + object-store backends + LSM.
+  metrics.rs         /metrics counters. telemetry.rs: opt-in usage pings.
 ```
 
-## Vector backend abstraction
+## Vector backends
 
-All vector backends implement `compass_index_api::VectorIndex`. The default backend is `UsearchHnswIndex` (CPU, mmap-backed, disk-persistent). The opt-in GPU backend is `compass_vector_gpu::CuvsHnswIndex` (CAGRA build on GPU, HNSW search on CPU).
-
-Selection happens at startup in `search::backend::build_backend`, driven by the `COMPASS_BACKEND` environment variable:
-
-| Value | Behavior |
-|-------|----------|
-| `cpu` (default) | USearch on CPU. Always available. |
-| `gpu` | cuVS on GPU. Requires the `gpu` feature and a CUDA-capable device. Falls back to CPU with a warning if either is missing. |
-| `auto` | Probe for GPU, fall back to CPU silently if unavailable. |
-
-The trait is intentionally narrow: `build`, `add`, `search`, `len`, `dims`, `save`, `backend_name`. New backends should fit through this surface or extend it via a follow-up trait, not by branching on a concrete type.
+The engine uses USearch HNSW directly (CPU, mmap-backed, disk-persistent)
+for warm serving, plus an IVF layout inside segments (`search/ivf.rs`) for
+serve-from-storage cold reads. `compass-index-api` (a narrow `VectorIndex`
+trait) and `compass-vector-gpu` (cuVS) exist as standalone crates for a
+future GPU integration but are NOT wired into the engine — there is no
+`COMPASS_BACKEND` knob and no `gpu` feature on the `compass` crate today.
 
 ## Storage layout
 
 Per-collection state lives under `$DATA_DIR/<collection>/`:
 
 ```
 data/<collection>/
-  meta.json                 CollectionMetadata (name, default vector space, vector_spaces map)
-  chunks.bin                Append-only log of Chunk records
-  metadata.bin              Per-chunk metadata (typed values, bitset-faceted)
-  fts/                      Tantivy directory
-  vectors/<space>/
-    index.usearch           USearch HNSW (CPU) — mmap-backed
-    index.cuvs              cuVS HNSW (GPU build) — when COMPASS_BACKEND=gpu
-    index.keymap            Internal HNSW key -> external chunk id mapping
-    vectors.bin             Raw float buffer (used for brute-force fallback + rebuilds)
+  collection.json           Collection metadata (name, config, vector_spaces map, applied_seq)
+  chunks.redb               Chunk bodies + metadata (redb; disk source of truth)
+  relations.redb            Typed many-to-many chunk relations (redb)
   relationships.bin         Parent-child + sibling edges
+  tantivy/                  Tantivy FTS index directory
+  vectors/
+    <space>.index           USearch HNSW graph — mmap-backed
+    <space>.keymap          Internal HNSW key -> external chunk id mapping
+    <space>.bin             CMV2 mmap vector file (torn-append-safe, per-batch durable)
 ```
 
+In cloud mode the object-storage bucket additionally holds, per collection:
+`collection.json` (bucket config), `manifest` (LSM manifest, CAS-committed),
+`wal/{uuid}.frag` (WAL fragments), `segments/{uuid}` (CSEG0003 sectioned
+segments: row-addressable metadata + IVF-clustered vectors; v2 readable), and `id-alloc` (CAS-leased chunk-id blocks).
+
 The disk format is the contract. Bumping it requires a migration path documented in CHANGELOG.md.
 
 ## Rebuild flow (model upgrades)
@@ -114,7 +122,8 @@ cuVS CAGRA build on an A10G runs ~12x faster than USearch CPU build at the same
 1. Create a new crate `crates/compass-vector-<name>/`.
 2. Depend on `compass-index-api` (workspace dep) and your backend library.
 3. Implement `VectorIndex` (and `LoadableIndex` if loading from disk makes sense).
-4. Add a `#[cfg(feature = "<name>")]`-gated branch in `search::backend::build_backend`.
+4. Wire it into the engine (there is currently no runtime backend selector —
+   proposing that wiring is part of such a PR; open an issue first).
 5. Document the build prerequisites in `ARCHITECTURE.md` (this file).
 6. Add a smoke binary under `src/bin/` that builds, queries, and prints latency.