Storage Engines and Deployment

The engine's storage layer adapts to the infrastructure you already run instead of demanding its own: the same API and the same access model ride on an embedded local store, on PostgreSQL, or on the enterprise databases your organization operates. This guide covers the engines, what each one gives up, and the mechanisms that keep the index healthy as volume grows.

1It works before you deploy anything#

A fresh installation auto-provisions a LOCAL cluster: an embedded store plus a local vector store, zero external dependencies. Every capability of the API (tenants, collections, hybrid search, grounded answers, evaluation) works against it immediately, which makes it the right place to develop, demo, and run modest corpora indefinitely. Deleting the last cluster from the Search section is remembered as a deliberate choice; the next restart does not re-provision behind the operator's back.

On a fleet, a node that carries a fleet signal never provisions the local cluster; section 7 states what the deployment declares instead.

2The engines#

A cluster's engine is fixed at creation, and engines never mix inside a cluster; different clusters on one server can use different engines side by side.

Two practical notes: on the app-level engines (Embedded, SQL Server, MySQL) the server itself owns tenancy, the text index, metadata filtering, and embedding at ingest, so behavior stays identical across engines; and clusters are created by the operator in the admin panel (or declared in the deployment's configuration on a managed fleet), never through the public API, so topology stays an operator decision.

3When to reach for PostgreSQL#

The embedded engine is not a toy, but three needs move a deployment to PostgreSQL:

• Defense in depth. Tenant grants are enforced by row-level security inside the database as well as in the query predicate, so the boundary holds even against an application-layer defect (see the isolation menu).
• Volume. Storage is hash-partitioned, reindexes run in the background while search keeps serving, and per-tenant vector partitions keep nearest-neighbor quality stable as individual tenants grow large.
• Operational fit. Backups, replication, and monitoring ride the PostgreSQL tooling your operations team already trusts.

Moving is a provisioning exercise, not a migration feature: create the new cluster, provision tenants, and re-index from your sources (ingestion is idempotent by content hash, so re-sending a corpus is safe and only real changes cost work).

4Scaling the semantic index#

Semantic search quality depends on the health of the nearest-neighbor index, and the engine manages it actively:

• Shared pool by default. Tenants share pooled vector capacity, which is right for most of them: small tenants get big-index quality without per-tenant overhead.
• Dedicated partitions for the whales. A tenant whose vector volume crosses a threshold (250k vectors by default) is PROMOTED automatically to dedicated partitions when that move is free; operators can promote explicitly before bulk-indexing a large tenant, which is the right order. A dedicated partition keeps the tenant's graph pure (its recall does not depend on neighbors), makes wipes instant truncations instead of million-row deletions, and lets re-embeds build into an empty graph. Demotion returns the tenant to the pool, vectors preserved, no re-embedding.
• Background embedding. Ingestion never waits on inference by default: documents become full-text searchable immediately and the embedding backlog drains in the background, so write throughput is bounded by storage, not by the embedding model. Callers that need a document semantically searchable the instant the call returns opt in per request. Embedding runs on its own inference pool, so bulk indexing never competes with chat for slots (see Inference Capacity).
• Live-tunable reindexing. Rebuilds (an embedding-model change, a normalization change) run in the background with a parallelism setting read live from the admin panel, and search keeps serving on the previous state until the swap completes. Progress is visible per collection and in Jobs.

5Ingestion bounds that protect the box#

Three limits, all live-tunable from the admin panel, bound what one upload can cost: a per-page character cap for paginated documents (generous; only pathological pages hit it), a re-pagination size for single-flow documents (plain text, Markdown, HTML are split on line boundaries into indexable pages, no content lost), and a maximum page count per document, which is the denial-of-service backstop bounding the total embedding and storage work a single request can generate.

6Operating it#

The Search section drills from clusters to one cluster to one tenant: storage and document counts, embedding backlogs, health, and database recommendations where the engine can see them. GET /lmkit/v1/search/status answers availability per caller for machines, and Observability covers the request, log, and telemetry story that Search shares with the rest of the server.

7Search on a fleet#

Search on a load-balanced fleet is a declaration, never a default. A node that carries a fleet signal (Farm mode, managed configuration through LMKIT_MANAGED_CONFIG, or a shared operator identity store under Admin:Identity) never provisions the local cluster, because an embedded cluster is one node's file and N nodes would serve N disjoint indexes under one cluster id. The cluster definitions themselves are domain objects: one document per cluster in the identity store, its connection strings and Qdrant key sealed under the farm's domain key, created, changed and deleted from any node's Search section and applied by every node sharing the store within its sync period (a node opens a cluster it has not seen before exactly as at startup). What the manifest still states, identically on every node, is the declaration the startup gate reads before the store opens:

• Search on: Search__Enabled=true. The clusters then come from the store; the first node to start against an empty store seeds them once from Search__Clusters__0__Id, Search__Clusters__0__IsDefault, Search__Clusters__0__FullTextStore, Search__Clusters__0__ConnectionString (supplied from the platform's secret store, as the container manifests do) plus the vector store settings the engine needs, and after that the panel is where they change.
• Search off deliberately: Search__Enabled=false.

A fleet-signal node that finds neither keeps Search disabled and names both remedies in the startup log and in the Search section. A farm node refuses to create an embedded cluster and quarantines one that reaches it through the store. An unmanaged node with an embedded identity store is a single node by definition: it provisions its private SQLite cluster exactly as section 1 describes (recorded in its own store), and two such nodes behind one balancer diverge silently.

On any external engine the content lives where every node reaches it, and the engines differ in how they coordinate the maintenance passes each node runs. PostgreSQL clusters coordinate them across nodes with advisory locks (the per-tenant vector pass, the per-collection full-text rebuild and the expiry sweep): every node's worker takes the lock for a pass and skips it while a sibling holds it. The backends are named so the database shows who holds what: every connection a node opens to the cluster carries the application name lmkit-one <tag> <boot> (the node's six-character tag and its boot id), and a lock-holding connection lmkit-reindex <tag> <boot>. A node's start terminates only the lock holders named after its OWN tag with a different boot id (its earlier incarnation, dead with the process) and never a sibling's live pass; a pass whose lock connection drops stops at its next probe instead of continuing unguarded. A node that crashes outright leaves its sessions to the database, so set a server-side idle_session_timeout (a few minutes) on the cluster's role or database to retire them, or the next start of the SAME node reclaims them. Two nodes that resolve to one name are one node to this rule and terminate each other's lock connections at start, so every fleet member sets a distinct LMKIT_NODE_NAME (Scaling Out section 6). MySQL and SQL Server clusters (with Qdrant holding the vectors) run their maintenance on every node unguarded: two nodes can run the same expiry purge or Qdrant deletion replay at once, so a multi-node farm on those engines accepts duplicate maintenance work or runs the clusters from one node. Their writes need no such choice: a store upserts by checking then inserting inside one transaction, and when two nodes write the same key at once (the same document indexed twice, the same collection created, the same setting written) the engine's unique constraint stops the second insert and that node runs its check again in a fresh transaction, finds the row and updates it, so no violation reaches the caller and no second row reaches the table. Budget connections per node: a PostgreSQL cluster's MaxConnections bounds ONE node's pools, so a fleet of N nodes opens up to N times that figure against the database; divide the engine's max_connections (minus its reserved slots and other clients) by the node count when setting it. The app-level engines open one connection per operation and ignore the setting, so their load follows each node's request concurrency. The reindex worker runs on every node and its parallelism setting is per node too: each node re-embeds the tenants whose lock it holds with that many documents in flight, so a fleet of N nodes drives up to N times the setting against the cluster and the embedding model; set it for the fleet you run.

A control-plane write on one node is observed by every other node within the sync period. Each node caches a tenant's settings, a collection's configuration and internal id, and the Qdrant collections it has ensured, so the ingestion and query paths spare the small admin pool; a write that changes any of them (a model swap, a normalization change, a collection created, reconfigured or deleted) drops the writing node's entries and advances the cluster's generation in the identity store, and the other nodes drop that cluster's entries when their sync brings the new generation (Deployment:SyncSeconds, five seconds by default on a farm), so a query on another node re-reads the row it needs within one sync period. Every entry also expires within thirty seconds on its own, which bounds the staleness when the store is unreachable or the sync lags; a query or an indexing call never serves a setting older than that from any node.

Four chores of a PostgreSQL cluster run once per fleet, under leases in the identity store. A cluster's schema migrations run at start under search-migration:<cluster>, so the nodes of one boot wave take turns instead of contending on the same DDL (the second starter waits, then finds everything applied). The database advisor evaluates every cluster on every node, so each node's Search section carries the findings, but raises the reload-scope settings (ALTER SYSTEM, checkpoint timeout and WAL size) only on the node holding search-database-advisor:<cluster>. And the telemetry panel's database health poll spreads itself: each node polls every three seconds multiplied by the live node count, so the database sees one poll per three seconds however many nodes watch it. And the sweep for dead vector-partition bloat runs every six hours on the node holding search-bloat-reconcile:<cluster>.

The maintenance signals a node used to keep to itself are documents of the identity store too, one per cluster and tenant. A deletion journals what it removed (search-deletions), so whichever node runs the cluster's orphan sweep drains what every node deleted, and a node that stops after deleting loses no reconciliation signal. A deletion in progress marks its tenant (search-preemptions), so the node running that tenant's re-embed pass cancels it within the sync period and skips the tenant until the deletion ends; a mark whose node is no longer alive in the roster is not honoured. A tenant-wide wipe records its generation and the partition rebuild it scheduled (search-partition-resets), so every node parks that tenant's re-embed until the rebuild ends and probes the promotion boundary the wipe opened.

Two consequences reach beyond the Search API. Vector stores and file_search of the Responses API live on the default cluster and follow its engine, so they are fleet-wide exactly when that cluster is external. And the affinity set is small: indexing and search calls are stateless on every node, grounded chat sessions (search/chat) are records every node serves (a node without the live conversation rebuilds it from the recorded history), and async jobs are records every node polls and cancels; only a job result too large for its record, or a record that cannot be read, sends a request back to the node whose tag opens the id with a 421 Misdirected Request naming it (Grounded Answers, Scaling Out).

8Stated plainly#

• Same API, same access model, four storage engines: the infrastructure decision never leaks into the application.
• Start on the auto-provisioned local cluster; move to PostgreSQL for row-level security, volume, or operational fit; use SQL Server or MySQL plus Qdrant where policy dictates.
• The index maintains itself in the background (embedding backlogs, reindexes, whale promotion); the admin panel shows it happening, and search keeps serving throughout.
• A fleet declares Search on or off in every node's environment; its clusters are domain documents created from any node's panel (or seeded once from the manifest), nothing embedded is provisioned or accepted on a farm node, and connection budgets are per node.