Tenant-partitioned collections: multi-tenant scale in one collection (Phase 6 core) - #11
Closed
EdgarBabajanyan wants to merge 3 commits into
Closed
Tenant-partitioned collections: multi-tenant scale in one collection (Phase 6 core)#11EdgarBabajanyan wants to merge 3 commits into
EdgarBabajanyan wants to merge 3 commits into
Conversation
…on (Phase 6 core)
Create a collection with config.partition_by = "tenant_id" and every
chunk routes to an internal per-tenant namespace. A partition IS a full
collection under the hood — own LSM manifest/WAL/segments, Tantivy dir,
vector files, attach/evict lifecycle — so Phase 6 is a router at the
manager entry points over the existing engine, not new machinery. This
is what moves the scale envelope from per-COLLECTION to per-TENANT: a
query for tenant T attaches T's slice only, LRU eviction and refresh
scale with the hot-tenant set, and RAM is bounded regardless of how
many tenants (or how many billion vectors) the collection holds.
Mechanics:
- Partition namespace: {parent}--part--{value} (separator reserved at
create; partition values validated kebab-case, len <= 64). Namespaces
reuse every existing storage/path/attach rule unchanged.
- Ingest groups by metadata[partition_by] and delegates per group,
auto-creating partitions on first sight (inheriting the parent's
vector spaces + embed model). Writer role routes the same way,
bootstrapping partition bucket objects with create-only writes.
- Chunk ids are COLLECTION-unique: partitions claim CAS-leased blocks
from the PARENT's allocator — in local mode too (the allocator is a
CAS-updated file; no WAL/manifest objects locally). Without this,
delete-by-id and search results would be ambiguous across tenants.
- Search and delete-by-filter require a filter on the partition field:
exact match routes to one partition, set membership fans out (<=16,
merged by score). A tenant with no data is empty, not an error.
Bare-id deletes are rejected with guidance (ids don't name their
partition; probing every partition is unbounded).
- Partitions attach on demand even in non-lazy cloud mode (a writer can
mint one at any time — 'all attached at boot' can never hold), are
hidden from listings, and are cascade-deleted with the parent from
disk, registry, and bucket.
- Fenced with clear errors until routed: relations, facets, TAMS
lookup, vector-space CRUD, min_seq across a multi-partition fan-out.
Writer-role fences consult the bucket config, so a writer can't
durably append relations/tombstones into the parent namespace that no
serving node ever materializes.
Tests (11 new): tenant isolation on shared query terms, id uniqueness
under interleaved multi-tenant ingest, filter-routed deletes, listing/
cascade behavior, fence + validation errors, restart persistence,
writer-role ingest visible on a serving node that predates the
partitions, and cold rebuild of a partitioned collection from the
bucket alone (ids keep minting without reuse). Suites: 105 local / 153
object-storage, clippy clean, all four build combinations verified.
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
Signed-off-by: Edgar Babajanyan <bedgar2005@gmail.com>
This was referenced Jul 4, 2026
Contributor
Author
|
Shipped in v0.4.0 via the consolidated release PR #13 (all commits, authorship, and sign-offs from this branch landed there verbatim). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
One collection cannot hold a large multi-tenant corpus: every attached collection costs RAM/attach-time proportional to its FULL size, so a billion vectors across thousands of tenants in one namespace is unservable — the measured single-namespace envelope is ~10–50M chunks.
Design
A partition IS a full internal collection.
config.partition_by = "tenant_id"at create marks the parent; every chunk routes to{parent}--part--{value}— its own LSM manifest/WAL/segments, Tantivy dir, vector files, and attach/evict lifecycle, reusing the existing engine wholesale. Phase 6 is a router at the manager entry points, not a new engine: the fundamentals (collections, local-first behavior, unpartitioned paths) are untouched, andpartition_byunset means zero code-path change.metadata[partition_by], auto-creates partitions on first sight (writer role bootstraps bucket objects with create-only writes).{"in": [...]}fans out (≤16, merged by score). Empty tenant = empty results, not an error.min_seqacross multi-partition fan-out. Writer-role fences consult the bucket config so a writer can't durably append into the parent namespace that no serving node materializes.Evidence
Boot and serving RAM are independent of tenant count — RAM tracks
cap × per-tenant size. That moves the envelope from per-collection to per-tenant: 1B vectors across 10k × 100k-vector tenants serves on the same bounded footprint.Notes / follow-ups
feat/warm-serverless); merges after it, in that order.min_seqmap, relations within a partition, vector-space CRUD propagation, re-partitioning tooling.