Scaling Out

How one server becomes a farm: identical nodes behind an ordinary load balancer, sharing an admin domain, an upload volume, and a model volume, with configuration owned by the deployment. Any node serves any API request; nodes join warm, leave draining, and the fleet grows or shrinks on a first-party signal. For the architect sizing past one box and the operator running the result, on Kubernetes or on plain VMs.

1The model: scale by replication#

Every node is a complete engine: its own resident models, its own inference pools, its own device memory. The fleet scales by adding nodes, and a load balancer in front spreads requests across them. Three things are deliberately NOT in the design, so nobody waits for them: cross-node tensor parallelism (one request never spans machines; a model too large for one node needs a bigger node, not more of them), scale-to-zero (model loads are measured in minutes, so a zero-node fleet answers its first request unacceptably late; the floor is one warm node), and sticky GPU sessions (no request depends on reaching the node that served the previous one).

The consequence worth stating first: capacity math is per node times node count. Size ONE node honestly with Inference Capacity, then multiply.

2What the fleet shares#

Four shared things turn independent servers into one domain. Each is explicit configuration, so a reviewer can read the fleet's shape off the manifest:

Identity and content never share a database: the identity store refuses to land in a knowledge-base database and the reverse, by design (Operator Accounts covers the store and the accounts in it). Search on a fleet runs its clusters on external engines (PostgreSQL and friends), which every node reaches with the same connection string (Storage Engines and Deployment).

3What stays per node, honestly#

• The operator plane. The dashboard, logs, request history, and telemetry panels are each node's own view; the admin footer names the node serving the page, so behind a load balancer you always know whose story you are reading. Fleet-wide observability is the export path: every node ships the same metrics to your Prometheus or OTLP stack (Observability).
• Async jobs. A job id from Prefer: respond-async polls on the node that minted it. Route bulk async callers by consuming system (one hostname per caller, or an affinity cookie at the balancer) rather than round-robin per request.
• Embedded search clusters. The zero-dependency SQLite engine is one node's file; a farm configures external engines instead.
• Per-owner budgets. Concurrency ceilings and per-owner job caps apply per node, so a fleet of N multiplies them by N; set them accordingly.

Everything else on the data plane is self-contained per request: the OpenAI, Anthropic, and Ollama surfaces carry their own history, and the Responses API rides the shared store.

4The load balancer#

Any TCP or HTTP balancer works; the requirements are short:

• Membership by readiness. Probe GET /lmkit/v1/ready (bearer key required; a literal header in the probe config) and route only to nodes answering 200. /health stays the anonymous liveness answer.
• Streaming-safe forwarding. Disable response buffering and raise idle timeouts so streamed completions flow token by token (Behind a Reverse Proxy has the exact settings per proxy).
• No stickiness required for correctness on the stateless surfaces. One optional performance trick: routing each API key to the same node when possible (consistent hashing on the Authorization header) keeps that caller's prompt prefixes in the node's cache, which measurably cuts time-to-first-token for clients that resend long system prompts.
• Real client addresses. List the balancer in Security:TrustedProxies so sign-in throttling and the audit trail record the caller, not the proxy.

5Node lifecycle: join warm, leave draining#

A node joining the fleet reports warming (503) on the readiness probe until its configured lineup is resident: opt in with Inference:WarmupModels (model ids, or default for the default chat model), and the balancer holds traffic until the node can actually serve it. A model that cannot load parks the node visibly in warmup_failed instead of letting it serve refusals.

A node leaving drains: the moment a graceful stop begins, readiness flips to 503 draining and the listener stays open for a short advertise window (default 5 seconds inside Kubernetes, detected automatically; set top-level DrainAdvertiseSeconds on VM fleets) so requests routed before the balancer noticed still get served. Then the listener closes and in-flight work, streamed completions included, finishes within ShutdownDrainSeconds (default 100). Give the supervisor a grace period above the sum: terminationGracePeriodSeconds in Kubernetes, TimeoutStopSec under systemd.

Because joins are warm and leaves are lossless, a rolling upgrade is just the two combined: one node drains and finishes its streams while its replacement warms, and the fleet never serves a refusal it did not choose.

6The scale signal#

The gauge to act on is lmkit_inference_slots_saturation on /metrics: the busy share of the completion budget, 0 to 1. Pressure shows there BEFORE a queue forms, so a scaler acting on it adds capacity while the fleet still absorbs the load; lmkit_inference_queue_depth confirms when it did not. On Kubernetes, KEDA drives the replica count from that gauge (the ScaledObject, both trigger styles, and the full fleet manifests are Running in Containers sections 7 and 9). On a VM fleet the same gauge feeds whatever starts machines for you, or a person watching a graph; the contract is identical because the signal is.

7Without Kubernetes#

Nothing above requires an orchestrator. Two VMs running the Linux package under systemd (or Windows services, Windows Server Deployments), HAProxy or nginx in front, one PostgreSQL for the identity database, and an NFS or SMB share for uploads and models is a complete fleet: set the same environment on every box (LMKIT_MANAGED_CONFIG=1, LMKIT_STATE_DIR on local disk, LMKIT_MODELS_DIR and FileManagement__UploadDirectory on the shares, Admin__Identity__* at the database, DrainAdvertiseSeconds=5, the warmup lineup) and the boxes behave exactly like the pods.

8Availability, restated#

The farm changes the availability answer. Any node serves any request, so losing a node loses only its in-flight work and its own operator history: keys, sessions, stored responses, uploads, and search data live in the shared stores and volumes, which ride your database and storage operations (Backup and Upgrades for what each copy set contains). The single-box cold-standby story remains valid for single-box deployments; a fleet simply does not need it for the data plane.

Where the requirement is regulatory separation (independent failure AND audit domains), separate fleets remain the answer: the same boundary One Server, Many Teams draws for tenancy, applied to infrastructure.

9Stated plainly#

• Scale by replication: identical nodes, a load balancer, shared admin domain, shared upload and model volumes, manifest-owned configuration. Any node serves any API request.
• Nodes join warm and leave draining, so scale-out, scale-in, and rolling upgrades never serve refusals; the saturation gauge is the signal that drives all three.
• Per node, honestly: the operator plane, async job polling, embedded search, and per-owner budgets; route async bulk callers with affinity and run fleet search on external engines.
• Kubernetes gets manifests and KEDA; plain VMs get the identical contract with systemd and any balancer. One request never spans machines, and the floor is one warm node.