Search & RAG/The Search Engine
Clusters, Tenants, Collections
Search data lives along three axes, and the design's one big idea is that they are DECOUPLED: where data is stored (cluster), who is isolated from whom (tenant), and how datasets are scoped (collection) are three independent decisions. Systems that fuse them (one index per customer, one database per dataset) force every isolation change to become a data migration. Here, moving a boundary is re-granting a key, not moving rows.
1The three axes, and what each one decides#
| Axis | It decides | It never decides |
|---|---|---|
| Cluster | WHERE data physically lives: which storage engine, which database server, which machine. Operator-created in the Search section, addressed by a chosen cluster_id. |
Who reaches the data. A cluster is deployment, not access |
| Tenant | The ISOLATION and CONFIGURATION unit: access is granted per tenant, and everything quality-relevant (embedding model, search modes, OCR, normalization, reranker, query model) is tenant-wide. Server-generated GUID. | Where data lives (its cluster fixed at creation) or how queries are scoped |
| Collection | Dataset SCOPING inside a tenant: documents are indexed into collections, queries target one or several (collection_ids). Caller-chosen GUID, created idempotently. |
Isolation. Collections share their tenant's access and settings |
Two rules position any domain onto the axes:
- The tenant is the smallest unit you will ever grant or revoke independently. If two audiences must never see each other's documents, those documents belong in different tenants. Anything below that granularity is a collection (scoped at query time by the caller) or a metadata filter.
- Metadata filters are conveniences, never boundaries. The caller writes the filter, so a filter can narrow what the caller ASKS for, not what it MAY see. Every real boundary is a grant (see the Access Model).
2Mapping an existing design#
The common starting point is one database segmented by schema per program and per audience. The translation:
| You had | You model as |
|---|---|
| The physical database | One cluster, usually the default one, so cluster_id can be omitted from every request. Add clusters only for genuinely separate storage or compliance domains |
| A schema per program | A tenant per program. Tenant-wide settings (embedding model, normalization) then follow the program |
| Access per audience or application | One API key per audience, granted onto exactly the tenants it may reach. The grant matrix IS the access model |
| Datasets inside a program | Collections inside the tenant, queried together or separately with collection_ids |
| Row-level or attribute segmentation | Custom metadata plus query-time filters, convenient but never an access boundary. Audiences that must not see each other's rows get separate tenants |
The same translation covers most systems: replace "program" with customer, department, product, or environment, and the table holds.
3The isolation menu#
Isolation is not one switch; the system offers four layers, composable and independently adjustable:
- Access isolation (always on). Keys reach only granted tenants inside granted clusters; unknown and denied resources answer identically to missing ones; new keys start with nothing. This is the everyday boundary, and for most multi-tenant products it is the only one needed.
- Database-enforced isolation (PostgreSQL clusters). The tenant grant is enforced twice: in the query predicate and again by row-level security inside the database, so an application-layer defect cannot widen reach. Choose PostgreSQL when the security review wants the boundary below the application.
- Vector-index isolation (promotion). A tenant's semantic vectors normally share pooled index capacity. Promoting a tenant gives it DEDICATED vector partitions: its nearest-neighbor graph holds only its own data, so a huge neighbor cannot degrade its recall or speed, wipes become instant truncations, and bulk re-embeds build into an empty graph. Large tenants are promoted automatically past a volume threshold; operators can promote or demote explicitly at any time, without re-embedding (see Storage Engines and Deployment).
- Physical isolation (separate clusters). Different storage engines, different database servers, different compliance domains: a cluster per environment (production and staging), or per data-residency requirement. Clusters never share anything, and a caller's cluster grants bound which ones it may even address.
Pick the lightest layer that satisfies the requirement; heavier layers are for requirements, not for comfort.
4Choosing your shape: three worked examples#
- A product with many customers. One cluster (the default). One tenant per customer: their isolation is grants, their settings follow their corpus. One key per service of yours, granted all customer tenants; or one key per customer integration, granted just theirs. Collections split each customer's datasets (contracts, tickets, wiki).
- An internal knowledge platform. One tenant per department when departments must not read each other; one shared tenant with per-source collections when they may. Metadata carries source, date, and document type for filtering and facets.
- A regulated deployment. A cluster per residency domain on PostgreSQL (row-level security below the app), tenants per business unit inside each, and keys granted strictly inside their domain: physical, database, and access isolation stacked.
5Stated plainly#
- Deployment, isolation, and dataset scoping are three independent decisions; the design's value is that changing one never forces a migration of another.
- The tenant is the unit of trust AND of configuration: grant by tenant, tune by tenant.
- Filters scope queries, grants scope access, clusters scope infrastructure; confusing those three is how search systems grow security holes.