Scaling Out

How one server becomes a farm: identical nodes behind an ordinary load balancer, sharing an admin domain, an upload volume, and a model volume. Any node serves the stateless surfaces; a short list of surfaces is bound to the node that created their state, and every response names its node so a request that reaches the wrong one is told so instead of failing quietly; nodes join warm, leave draining, and the fleet grows or shrinks on a first-party signal. For the architect sizing past one box and the operator running the result, on Kubernetes or on plain VMs. This page owns the farm story; the container manifests are Running in Containers and the balancer directives are Behind a Reverse Proxy.

1Single node or farm: the decision#

One box keeps every convenience, and a farm trades three of them for capacity; this section is the price list, Reference Architectures places the fleet on the ladder that decides whether you need it, and Fleet Recipes writes out what each platform (Azure App Service, Azure Container Apps, Kubernetes, VM and Windows service farms, Compose and Swarm, Amazon ECS) needs from the contract below.

A single node keeps the embedded SQLite identity store under its state directory, writes every panel and wizard edit to its own appsettings.json, and provisions the embedded search cluster on first use. Nothing else is needed, and nothing below applies.

A farm is declared with one switch and needs three things decided before the second node starts:

Everything the identity database does not hold and no shared volume carries stays on the node that wrote it; section 5 is the complete list, and section 6 says which of those surfaces the balancer must pin.

2The model: scale by replication#

Every node is a complete engine with its own resident models, inference pools and device memory, and the fleet scales by adding nodes behind a load balancer, so size ONE node honestly with Inference Capacity and multiply.

Two things are deliberately NOT in the design, so nobody waits for them: cross-node tensor parallelism (one request never spans machines; a model too large for one node needs a bigger node, not more of them) and scale-to-zero (model loads are measured in minutes, so a zero-node fleet answers its first request unacceptably late; the floor is one warm node).

One thing IS in the design and has to be named: several surfaces depend on reaching the node that holds their state (a chat-with-document session, an MCP event stream, a running exchange, a job result too large for its record). Section 6 lists them with the contract the server emits for them and the pinning that keeps them on one node; nothing else on the data plane cares which node answers.

3What the fleet shares#

Three shared things turn independent servers into one domain, each explicit configuration a reviewer can read off the manifest, and the identity database among them is the availability dependency this section spells out (Operator Accounts covers the accounts inside it).

Search content is shared through the cluster's own external engine, never through the identity database, and every node runs the search maintenance worker against it: on PostgreSQL the passes coordinate through advisory locks, so one node runs a given pass while the others skip it, and a node's start reclaims only the locks its own earlier incarnation left behind (its backends are named after it) and never a sibling's live pass (Storage Engines and Deployment section 7).

Setting up the identity domain

Configure the store in the settings file or the environment, deliberately not in the admin form: the surface that edits configuration authenticates through this store, so its selection is bootstrap-level.

{
  "Admin": {
    "Identity": {
      "Store": "postgres",
      "ConnectionString": "Host=db;Database=lmkit_identity;Username=lmkit;Password=..."
    }
  }
}

• Identity gets a strictly dedicated database; the database server can be shared. Knowledge-base clusters store document content; operator credentials, sessions and lockout state never share a DATABASE with them, while one PostgreSQL, MySQL or SQL Server instance may host the identity database beside the cluster databases. Both directions are enforced: an identity store pointed at a content database refuses to start, and a knowledge-base cluster pointed at the identity database refuses to initialize, each naming the remedy. Create a small dedicated database (for example lmkit_identity) and grant the server's login rights on it alone.
• Seed the first operator before exposing the nodes. Admin__InitialPassword becomes the password of the admin account and is consumed only while no account exists: whichever node runs first creates the account, every other node finds it and ignores its copy of the seed, two nodes seeding at the same moment tolerate the duplicate, and a restart never resets the password. A shared identity store never opens the admin surface, with or without accounts, so the first operator arrives through this seed or through lmkit admin create-owner run on a node with that node's environment (The Command Line explains why the shell's environment matters). Admin__Password is not a third way in: it is the emergency reset, re-applied to the admin account on EVERY start it is present (the password overwritten, a disabled account re-enabled), and a managed node cannot remove it from the manifest, so it is set for one restart and removed afterwards; the log names the source still supplying it (Operator Accounts section 3).
• Caches are 30 seconds wide. Session validations and the API-key snapshot are cached per node for 30 seconds, so a revocation issued on one node is effective everywhere within that window, and immediately for a token another node has never seen: an unknown key is looked up in the store before it is refused, which is what makes a key minted seconds ago valid on every node at once. Remembered devices, web-area tickets and in-flight SSO sign-ins are domain-wide as well, because the signing secret and the PKCE challenges live in the store, so a sign-in that starts on one node completes on another.
• A node's pre-existing apikeys.json is adopted into the store on its first start and removed, so a single node promoted into a farm keeps the keys it minted alone.
• Name the balancer in Security:TrustedProxies (IP addresses or CIDR networks) so sign-in throttling, session records and the audit trail see the real client address from X-Forwarded-For instead of the balancer's. With the list empty, forwarded headers are ignored, which is the safe stance when nothing sits in front.
• Security events (sign-ins and failures, account lifecycle, credential and second-factor changes) are written to the store's central trail beside each node's own journal; the audit history endpoint serves that domain-wide view for the security source.

While the identity database is unreachable

Inference continues. API keys already in a node's 30-second snapshot keep authorizing for the whole outage, which includes a key revoked on another node after the snapshot was taken; a key not in the snapshot is refused with 401 until the database answers. The admin surface never opens: a node keeps the last answer it had about existing accounts, and on a shared store the surface is closed unconditionally, so an outage cannot turn a farm into an unprovisioned install. Admin sign-in, operator management and key management refuse until the store returns. The outage and its recovery are each recorded once in the security trail and the node log, not once per request. Readiness does not depend on the database, so the balancer keeps routing to a node whose store is down; that is deliberate, because the data plane still works.

Work done once per fleet

A shared subject is maintained by one node at a time. Each such job takes a lease in the identity store (its own name, three of its periods long) and runs only while it holds it; a node that stops renewing loses the lease and any sibling takes over on its next pass. The Farm panel lists every lease with its holder, and the Jobs page names the holder beside each of these jobs, whichever node you are pinned to; a manual run from another node names the holder instead of doubling its pass.

The database health poll of the Search telemetry needs no lease: each node polls every three seconds multiplied by the live node count, so the database sees one poll per three seconds however many nodes watch it, and every node's panel keeps a live headline.

The Search control plane's caches keep coherent the same way the shared objects do: a write on one node (a tenant setting, a collection created, configured or deleted) advances the cluster's generation in the identity database, every other node drops its cached entries for that cluster when its sync brings the new generation, and every entry expires within thirty seconds on its own (Storage Engines and Deployment section 7).

Every shared object the domain carries, by kind (generated from the domain's kind registry, so a kind added without its row fails the build): a configuration object is farm-wide through the manifest or the panel, a record kind is applied as it changes, a registry kind is replaced as a whole on every node.

4Configuration in each mode#

Every setting the panel or the API can change has one of three classes, declared once in the server and carried by the configuration payload as classes, so the panel, the writer and the fingerprint of section 3 agree on it:

A save carrying Node leaves on a node that does not own its file is refused before anything changes, always in one shape: HTTP 409 { "error": ..., "managed": true }, which the panel keys on. A save carrying Domain leaves alone lands in the store from any node, and one that mixes the two is refused as a whole, so nothing half-lands. The panel echoes the revision of each domain kind it loaded, and the store refuses a kind whose revision moved since (another node wrote it) with the reason, so a stale page never overwrites another operator's change.

Who owns the file. A Farm node never owns it: Farm mode makes Node settings read-only whatever LMKIT_MANAGED_CONFIG says. A single node owns it unless LMKIT_MANAGED_CONFIG=1 is set in the environment or a mounted settings file carries top-level ManagedConfiguration: true. Where the node does not own the file, settings come from double-underscore environment keys (Inference__SlotContextSize, Admin__Identity__Store, ...) or the mounted file, the environment winning where both state a value; nothing is seeded or migrated on disk, so a read-only ConfigMap and a read-only root filesystem both work, and secrets are read as written, never encrypted at rest by the node. An owned file is written by the panel and the setup wizard, round-trips through export and import, and seals its secrets with that node's keyring (enc:v1 values, readable only where they were written).

Who owns the domain. The identity store, unless the deployment declares Deployment__DomainSource=Manifest: then the shared objects are the manifest's too. At every start the first node of a boot wave takes a short lease and writes what its environment and mounted files declare over the store (unchanged documents are skipped, a sealed secret is compared by what it opens to), every other node applies that within its sync period, the domain editors render read-only with the manifest as the remedy, and a domain write answers the same 409 as a node setting. Without it, the manifest seeds an empty store once and the panel or the API changes the objects from then on.

What the Node rule refuses on a Farm or managed node: every plain panel setting, config import, the setup wizard's apply, the Search section's node settings (the dumps, the limits, the quality gate, reindex parallelism), the TLS editors and certificate upload and removal, the managed ffmpeg install, and the lmkit cert-acme verb. The panel renders those settings read-only, with a notice naming the mode, so a pod's resolved configuration can still be inspected.

What stays editable on every node: the Domain objects above, and everything that is state rather than configuration: model files (pull, load, unload, delete, with the fleet rules of section 5), custom-model import (which lands in the shared store), skills (authored on any node, materialized on every node), operators, sessions, API keys, uploads, knowledge-base content, memory facts, and browsing the skills library.

How a fleet gets its Domain objects. Author them once, on any node's panel or through the API, and every node sharing the identity store applies them. A manifest can seed them too, since every setting still binds from the environment or a mounted file (Agents__0__Name-style keys, the Agents, Connectors, Tools, Memory:Stores and Search:Clusters sections): the first node to start against an empty store writes the values it bound into it, once per kind, and from then on the store is the authority, so a later edit of those manifest keys is not re-read; change the objects through the panel or the API instead. Skills:Sources is the one Domain object with a per-node effect: each node installs at startup, and again whenever the sources change, the skill archives the list names, installing what is absent and leaving what is present untouched, so an edited skill ships as a new source (Agent Skills).

Secrets in Domain objects. The web-search key of the tools policy travels sealed inside its document: under the farm's LMKIT_DOMAIN_KEY every node opens it; on a single node without a key it is sealed under that node's keyring. Joining a farm re-seals the copied key for the domain and leaving seals it back for the node. A value sealed under a key this node does not hold reads as unset, with one log line, never as garbage.

Export honesty, one sentence each. The export carries the shared objects from the store in the file's own shape and masks every secret as ****, so a configuration exported from one node is a seed manifest whose secrets you supply, and an import routes those sections back to the store while the file keeps the node settings. An enc:v1 value decrypts only on the node whose keyring wrote it, so a copied settings file reads those values as empty elsewhere. A mounted managed file is read as written and never encrypted at rest, so a fleet's secrets travel by environment variable or secret mount (the secretKeyRef the container manifest uses), never inside the file.

5What stays per node, honestly#

The Farm page under Operate is the one farm-tier surface: every node's heartbeat row, drift against the node you browse, propagation lag (Farm Operations). Everything below is per node and says so.

Everything below lives on one node in this release; the table names where, what a fleet observes, whether a client can see the difference, and whether a setting relocates it (Where Data Lives has the full disk inventory).

6Affinity, honestly#

Affinity is never what makes a request correct: a request that reaches the wrong node is told so and told where to go. It is what keeps the node-bound surfaces of section 5 on one node so that never happens, and it is what keeps the operator's console reading one node's story. The server emits a contract a balancer or a client can act on, and the directives per proxy live in Behind a Reverse Proxy.

What the server emits.

• X-LMKit-Node on every response: the six-character node tag (lowercase letters and digits, a stable hash of the node's name) on every response, the full node name only on responses to an authenticated admin session. GET /health on a node's own address is the quickest way to read its tag.
• The node's name, which everything above derives from. A node is named by the first of LMKIT_NODE_NAME, WEBSITE_INSTANCE_ID, HOSTNAME and the machine name that is set, reduced to printable ASCII (any other character becomes a dash). The tag, the header, the cookie, the ownership of training journals and the names of the search connections all follow from it, so the name must be unique in the fleet and stable across restarts: two live nodes with one name share one tag and every protection on this page reads them as one node, and nothing detects that. In a container the default is the container or pod id, which changes on every recreation; set LMKIT_NODE_NAME per node (or a stable hostname:) wherever the platform does not keep the name for you (Running in Containers shows where).
• The owner's tag inside every node-bound id. A chat-with-document session_id, a grounded chat session_id, an Mcp-Session-Id, a job_id and a training job id open with the tag of the node that minted them, in an opaque string of the same length as before. Ids that are domain-wide (API keys, response ids, file ids, operator ids) carry no tag, because any node serves them.
• 421 Misdirected Request on a wrong-node request. A request that names a node-bound id whose tag is not the answering node's answers 421 with the body { "error": "not_on_this_node", "node": "<tag>", "hint": ... }, where node is the owner's tag (never a hostname) and hint tells the caller to reach that node. The node never proxies to the owner. An id the answering node minted itself and no longer holds keeps answering 404, the same "unknown or expired" as on a single node, and so does a malformed id.
• Location on every job 202. Every asynchronous acceptance minted by a task endpoint carries Location: /lmkit/v1/jobs/<job_id> beside the body's job_id, and X-LMKit-Node like every response.
• The lmk-node cookie on a fleet member. When the node carries a fleet signal (managed configuration, or a shared identity store), the responses that mint a node-bound id and the /admin and /playground page loads set lmk-node=<tag>: HttpOnly, SameSite=Lax, Secure when the request arrived over HTTPS, valid one day. A single node without a fleet signal never sets it, so nothing changes for the one-box install.

The tiers.

How a client keeps its pin. A browser keeps lmk-node and a balancer that reads application cookies (HAProxy with per-server cookie values, nginx Plus sticky route, AWS ALB application-cookie stickiness) routes on it directly; a balancer that mints its own affinity cookie (ingress-nginx, Traefik, IIS ARR, App Service, nginx Plus sticky cookie, Caddy) pins the same browser with its own cookie, and lmk-node then tells the operator which node a browser is on. A bearer API client keeps no cookies, so it is pinned by a consistent hash on the Authorization header: every request with the same key reaches the same node while the pool is stable, which is also the KV prefix-cache optimization (a client that resends long system prompts keeps its prefixes warm on one node). The rule the hash must satisfy is one sentence: the request that opens a session or starts a stream and the request that follows it must hash identically, which only a whole-surface hash guarantees. A session is opened by chat-with-document on whichever node answered, an MCP event stream on whichever node received the GET, and skip-thinking targets the exchange streaming on /lmkit/v1/chat, so hash EVERY request outside the browser set (/admin, /lmkit/v1/admin, /playground) by Authorization and pin the browser set by cookie or source address. A header hash falls back to round-robin on a request without the header, so a bearer client that omits its key loses its pin with its authorization. Where a balancer has no header hash (Traefik OSS, IIS ARR, an AWS ALB) a bearer client is unpinned; it can still follow the contract itself, by sending its follow-ups to the node X-LMKit-Node named, when it can reach nodes by address.

Reading a lost pin. A 421 is the symptom: the request reached a node that does not hold the id, and the body names the one that does. Compare X-LMKit-Node on the minting response with the tag inside the id and the value of lmk-node; when they agree and a later request still answers 421, the balancer is not honouring the pin, and the balancer guide says which directive to check.

Every identifier a client can present later, by kind (generated from the id registry, so a kind added without its row fails the build): a node-bound id opens with the owner's tag and a wrong-node request is told where to go; a domain-wide id carries no tag, because any node serves it.

7The load balancer#

Any TCP or HTTP balancer works once four requirements are met, and only the first one hides a trap (Behind a Reverse Proxy has the exact settings per proxy).

• Membership by readiness. GET /lmkit/v1/health/ready answers 200 only when the node can serve and 503 otherwise, with the status word alone: not_ready while it initializes, warming while its lineup loads, draining the moment a graceful stop begins. It is anonymous and exposes nothing else, so every prober gates on it, including the ones that cannot send a header (the IIS ARR health test, Azure Load Balancer, an AWS ALB target group, the App Service health check). Never probe /health for membership: it is liveness, 200 whenever the process is alive, warming and draining included, so a balancer on it routes to a node before its models are resident and keeps routing to a node that is stopping. GET /lmkit/v1/ready carries the detailed body behind a bearer key for probers that want it (Kubernetes httpHeaders, HAProxy http-check send hdr). Windows NLB has no application probe at all and cannot gate on anything.
• Streaming-safe forwarding. Disable response buffering and raise idle timeouts so streamed completions flow token by token.
• Real client addresses. List the balancer in Security:TrustedProxies so sign-in throttling and the audit trail record the caller, not the proxy. A platform front whose addresses cannot be listed (App Service, Container Apps) takes the * sentinel, which the server accepts on those two platforms and refuses everywhere else unless Security:ProxyFrontOnly=true states that the nodes are unreachable except through the front. When TLS ends at the front, set Security:PublicBaseUrl to the public address, so the SSO callback and the Secure flag on the server's cookies follow the front instead of the node's own plain listener.
• Affinity for the node-bound surfaces, per section 6.

8Node lifecycle: join warm, leave draining#

A node joins reporting warming until its configured lineup is resident and leaves reporting draining while it finishes its work, and the only setting that changes off Kubernetes is the advertise window (Running in Containers wires the same behaviour into the manifests).

Joining. Opt in with Inference:WarmupModels (model ids, or default for the default chat model): readiness stays 503 warming until the lineup is loaded, so the balancer holds traffic until the node can actually serve it. A model that cannot load parks the node visibly in warmup_failed instead of letting it serve refusals.

Leaving. The moment a graceful stop begins, readiness flips to 503 draining and the listener stays open for a short advertise window so requests routed before the balancer noticed still get served. DrainAdvertiseSeconds defaults to 5 when the server detects a platform that routes by readiness (Kubernetes, Azure App Service, Azure Container Apps, Amazon ECS) and to 0 everywhere else (clamped to 0..30), so a VM, Windows service or Compose fleet sets DrainAdvertiseSeconds=5 explicitly or its last requests are refused. Then the listener closes and in-flight work, streamed completions included, finishes within ShutdownDrainSeconds (default 100, clamped to 5..570). The supervisor's stop allowance must exceed advertise plus drain plus 15 seconds of teardown, 120 seconds with the defaults, and every supervisor names it differently: terminationGracePeriodSeconds in Kubernetes (30 by default), TimeoutStopSec under systemd (90 by default, BELOW the default drain, so the unit raises it to 130), stop_grace_period in Compose (10 by default), WEBSITES_CONTAINER_STOP_TIME_LIMIT on App Service (5 by default, 120 at most) and stopTimeout on ECS (30 by default, 120 at most). The server knows these defaults: at startup it logs one warning naming the setting to raise when the detected platform's allowance is below what the drain needs, and where a platform caps the allowance at 120 you lower ShutdownDrainSeconds to 90 instead. The Windows service asks the service control manager for the time it needs when the stop begins, so no setting exists there. Fleet Recipes has the line for each platform.

Because joins are warm and leaves are lossless, a rolling upgrade is the two combined: one node drains and finishes its streams while its replacement warms, and the fleet never serves a refusal it did not choose.

9The scale signal#

The gauge to act on is lmkit_inference_slots_saturation on /metrics, the busy share of the node's completion budget from 0 to 1, exported per node and averaged by the autoscaler (Observability covers the export path).

Pressure shows there BEFORE a queue forms, so a scaler acting on it adds capacity while the fleet still absorbs the load; lmkit_inference_queue_depth confirms when it did not. On Kubernetes, KEDA drives the replica count from avg(lmkit_inference_slots_saturation) (the ScaledObject, both trigger styles, and the fleet manifests are Running in Containers sections 7 and 9). On a VM fleet the same per-node gauge feeds whatever starts machines for you, or a person watching a graph; the contract is identical because the signal is.

10Without Kubernetes#

Nothing above requires an orchestrator: two VMs running the Linux package under systemd (or two Windows services, Windows Server Deployments), HAProxy or nginx in front, one PostgreSQL for the identity database, and an NFS or SMB share for uploads and models is a complete fleet, and this section writes out the exact environment it takes.

Choose the configuration owner first; the two variants differ in one key and in who may edit what:

In Farm mode the panel is read-only for configuration in both variants: the mode implies the managed rule, so the second column then describes a farm exactly.

Then set the same block on every box. This is 21 environment keys as written, and a real fleet sets fewer: the Search cluster row is needed only when Search is on and stands for one cluster's keys, and the two HTTPS rows are alternatives, never both.

Machine-wide delivery. Under systemd, Environment= lines in the unit (or an EnvironmentFile=), so the service sees them and an operator's interactive shell does not by default; on Windows, setx /M or Group Policy, because a service running as LocalSystem reads the SYSTEM environment, not any user's. A CLI verb run from a shell without these keys resolves a different identity store than the service does (The Command Line).

11Availability, restated#

A fleet is the live availability answer for the stateless surfaces and for nothing else, and Backup and Upgrades says what each copy set contains.

Losing a node loses its in-flight work AND everything section 5 lists as per node: its live chat-with-document sessions, the MCP event streams open on it, the jobs it was running (their records stay and close as lost), a training run in progress, and its operator history (audit, metrics, telemetry, forensics, logs). Keys, sessions, stored responses, uploads, memory facts, skills, custom-model records, job, MCP session and grounded chat session records and external search data live in the shared stores and volumes, which ride your database and storage operations. The single-box cold-standby story remains valid for single-box deployments; a fleet does not need it for the stateless data plane.

Where the requirement is regulatory separation (independent failure AND audit domains), separate fleets remain the answer: the same boundary One Server, Many Teams draws for tenancy, applied to infrastructure.

12Stated plainly#

• Scale by replication: identical nodes, a load balancer, one dedicated identity database, shared upload and model volumes, external search engines. Any node serves the stateless surfaces.
• A farm needs one identity database on PostgreSQL, MySQL or SQL Server, and one external engine per search cluster; the embedded engines are single-node only.
• Node settings are the node's file: Farm mode (and LMKIT_MANAGED_CONFIG=1 on a single node) makes them read-only and every such edit refuse with 409; without either, panel edits are node-local and drift silently. Domain objects (agents, connectors, memory store policies, the tools policy, the skills allowlist and sources, default models, per-model overrides, MCP ingest roots, the Claude Desktop lineup pins and serving shapes, the Search clusters and the Search on/off flag) live in the identity store and save from any node in both modes.
• Per node, honestly: training execution, chat-with-document sessions, the MCP event stream, the results of jobs too large for their record, skip-thinking and cancel, TLS and keyring, the operator plane, the throttle and budgets, embedded search. Every Operate section is labelled with its node and every record and export row carries the node name.
• The contract: X-LMKit-Node on every response, the owner's tag inside every node-bound id, 421 Misdirected Request with not_on_this_node and the owner's tag on a wrong-node request, Location on every job 202, and an lmk-node cookie on fleet members. A job's record lives in the shared store, so any node polls and cancels it; a session or a stream lives in the memory of the node that opened it, so the cookie or the Authorization hash is how a client keeps its follow-ups on the right node: hash every request outside the console and Playground paths by Authorization (the request that opens a session or starts a stream and the one that follows it must reach the same node) and pin the console with a cookie or the source address.
• Every prober gates on /lmkit/v1/health/ready, anonymous and status-only; /health is liveness and never membership. Set DrainAdvertiseSeconds=5 where no platform is detected and keep the supervisor's stop allowance above advertise plus drain plus 15 seconds.
• Kubernetes gets manifests and KEDA; plain VMs get the identical contract with systemd and any balancer. One request never spans machines, and the floor is one warm node.

The twelve questions the design answers, and where. Each has one owner among the fleet guides; the other guides link to it.

Validate your fleet. The probes the two-node harness runs on every build, to run once against yours:

• GET /lmkit/v1/health/ready on each node answers 200 and names farm as the mode; every response carries a different X-LMKit-Node per node, so two nodes that answer with one tag share a name and must be renamed.
• A request for a node-bound id sent to the other node answers 421 Misdirected Request with not_on_this_node and the owner's tag, and the lmk-node cookie is set on every fleet member's response.
• An agent saved through one node is listed by the other within the sync period (Deployment:SyncSeconds), and a node setting edited on either node refuses with 409.
• The Farm page's roster shows every node with zero domain lag and no unaccepted difference, and the security audit's Fleet checks all answer ok.
• The Farm page's work table shows each single-runner lease held by exactly one node, and a manual run of that job from another node answers 409 naming the holder.
• Stopping one node drops its readiness for the drain window while the other keeps serving, and the node rejoins under the same tag when it returns.