LM-Kit OneDocs2026.9.7lm-kit.comEULA
Deployment/Scaling Out

Farm Runbooks

Scaling Out is the contract and Fleet Recipes the per-platform lines; this page is what an operator does, step by step, at each moment of a farm's life. Every procedure below is written against what this release does, and each ends with the check that tells you it worked.

What a farm shares in this release: the identity domain (operators, sessions, second factors, API keys with their counters, stored responses, the token signing secret, the security trail), the node roster (one heartbeat row per node, carrying the node's configuration fingerprint), the shared upload and model volumes, and the domain objects: agents, connectors, memory store policies, the tools policy, the skills allowlist and sources, default models, per-model overrides, MCP ingest roots, the Claude Desktop lineup pins and serving shapes, and the Search clusters, each a document every node applies within the sync period and any node's panel saves. Node settings (ports, TLS, the inference shape, security) still come from each node's environment and mounted files, which Farm mode makes read-only, identically on every node (Scaling Out section 4).

1Promote a single node to a farm#

You have one node on its embedded store and want a second one. The node's own domain (its operators, keys, responses and settings) moves into a shared database first, so nothing that signs in or authorizes today stops working.

  1. Create the identity database: a small, dedicated database on PostgreSQL, MySQL or SQL Server, with a login that owns it. Never a database that holds knowledge-base content; the server refuses one.

  2. Join from the panel or the shell. In the Access section, the identity domain card offers Join a farm to an Owner while the node is an unmanaged single node. Choose the engine, paste the connection string, Test it (the probe creates the schema and says whether the database already holds accounts), keep Copy this node's domain checked, and join. From a shell with the server stopped, the same is:

    lmkit farm join --store postgres --connection-string "Host=db;Database=lmkit_identity;Username=lmkit;Password=..."

    Either way the node writes Admin:Identity and Deployment:Mode=Farm into its own settings file, copies its domain into the database, and shows the domain key once together with the environment block every node needs and the prerequisites this node still has to meet (Scaling Out section 1 lists them). Copy the key and the block now; the key is not stored anywhere you can read it back from.

  3. Set the environment on this node: at least LMKIT_DOMAIN_KEY, and whatever the remaining-prerequisites list named (shared volumes for uploads and models, Search__Enabled, Security__NetworkAccess=Network). Put it where the service reads it: the unit file, the machine-wide variables, the manifest (The Command Line section 5).

  4. Restart the node. It starts as a farm member or refuses with the list of what is still missing; the refusal is the checklist.

  5. Check: the footer reads farm, lmkit farm status lists this node's heartbeat as farm ready live, and the operators who could sign in before still can.

Then add the second node (section 2).

2Add a node#

  1. Install the same version of LM-Kit One on the new machine or image.
  2. Give it the environment block of section 1, with two changes: its own LMKIT_NODE_NAME (or let the platform name it) and its own local LMKIT_STATE_DIR. Everything else is identical, the domain key included.
  3. Mount the upload and model volumes at the same paths as on every other node.
  4. Start it. It joins warm: readiness stays warming until its lineup is resident, and the balancer holds traffic until then (Scaling Out section 8).
  5. Add it to the balancer's pool with the readiness probe on /lmkit/v1/health/ready.
  6. Check: lmkit farm status on any node lists both heartbeats as live; an API key minted before authorizes on the new node; the operators sign in on it.

A node whose mode disagrees with the others refuses to start and names both modes. A node started with a domain key different from its siblings' is refused too: the first node of a farm seals a probe into the identity database, and every node must open it at startup. Keep the key in one secret store and hand it to every node from there.

3Remove a node#

  1. Take it out of the balancer's pool, or let the platform do it: a graceful stop flips readiness to draining and the listener stays open for the advertise window while the in-flight work finishes (Scaling Out section 8).
  2. Stop the service or delete the pod. On a graceful stop the node writes a last heartbeat that reads stopped.
  3. Nothing else is needed: its row is swept from the roster a day after its last beat, and the node held nothing the farm needed. Its state directory held only its own history (Where Data Lives), which you may keep or discard.
  4. Check: lmkit farm status shows the node as stopped and then, after a day, not at all.

4The identity database is down#

The contract is stated in Operator Accounts section 5 and holds on every node at once:

  • Inference keeps running: API keys already in a node's snapshot keep authorizing.
  • The admin surface stays closed; sign-in, operator management and unknown keys refuse until the database answers, and the security trail records the outage once.
  • Heartbeats and the domain sync pause and resume on their own; no node restarts, no node changes mode.

Do not restart nodes to "fix" it: a node that starts while the database is unreachable refuses to start (a shared store never falls back to an embedded one), which is one node fewer serving inference. Restore the database (section 5), then watch lmkit farm status on one node: the heartbeats resume within one sync period.

5Restore from backup#

The order matters, and it is always the same (Backup and Upgrades):

  1. The identity database first, from your database backup. A node started against an empty database creates a fresh schema with no accounts, which is a new domain, not yours.
  2. The shared volumes (uploads, models) next, at the same paths.
  3. The nodes last, each with its own state directory (its keyring is what decrypts its settings file; the two travel together).
  4. Check: operators sign in, an existing API key authorizes, lmkit farm status lists the nodes, and a file uploaded before the incident resolves through any node.

6Upgrade node by node#

The schema of the identity store grows additively, and each build records the version it knows: a node upgrades an older store in place at its first start, and a node refuses a store already written by a newer build, so a farm rolls forward and never back.

  1. Upgrade one node: drain it (section 3, step 1), install the new version, start it. Its first start upgrades the shared schema; every other node keeps running on the additive schema it already understands.
  2. Watch it join warm and take traffic, then repeat with the next node.
  3. A mixed-version window is expected while you roll. Do not start an old version against the upgraded store: it refuses, by design.
  4. Check: lmkit farm status shows every node on the new version; /health on each node reports it.

7Rotate the domain key#

Every node carries the same LMKIT_DOMAIN_KEY; the identity database holds a probe sealed under it (the domain-key-probe setting), which is how a node with another key is refused at startup, and the secrets the shared domain will carry are sealed under it as they land there. To rotate: set the new key on every node, delete the domain-key-probe row from the identity database's settings table (the first node to start seals a new one), and restart the nodes one at a time (section 6, steps 1 and 2). Keep the key in one secret store and treat a node that ever printed it as having disclosed it.

8Leave a farm#

A node returns to a single node on its own embedded store, with a copy of the shared domain if you want one:

lmkit farm leave

With the server stopped, the verb copies the shared domain into a fresh admin-identity.db under the node's state directory (--no-copy skips it), writes the settings file back to the embedded store as a single node, and lists the environment variables to remove before starting (LMKIT_DEPLOYMENT_MODE, LMKIT_DOMAIN_KEY, Admin__Identity__Store, Admin__Identity__ConnectionString), because the environment wins over the file. The shared database and the other nodes are untouched. Check: the footer reads single node and the operators sign in on the embedded store.

9Stated plainly#

  • Promote through the panel or lmkit farm join; the domain key is shown once and belongs in one secret store.
  • A new node is the same environment with its own name and state directory; it joins warm.
  • A removed node leaves nothing the farm needs; its row goes stale and is swept.
  • A database outage closes the admin surface and keeps inference running; restore the database first, the nodes last.
  • Upgrades roll forward one node at a time; an old build never starts against a newer store.
  • Leaving is the join in reverse, with the server stopped.