Farm Operations

Scaling Out is the contract and Farm Runbooks the procedures; this page is what the Farm page under Operate shows and how to read it. Everything on it comes from the shared identity store alone: each node writes one heartbeat row per sync period, and the node you are browsing derives the rest from those rows and its own state. No request ever crosses to another node.

1Two tiers, always labelled#

Every other Operate surface (dashboard, requests, logs, alerts, metrics, jobs) is one node's own, and its head says which node with a Node label; behind a load balancer the console is pinned to one node (Scaling Out section 6). The Farm page is the one farm-tier surface: it reads the shared tables and shows every node, this one included. A single node with the embedded store sees a roster of one, which is how the same page serves both modes.

2Roster#

One row per node: its name, state (ready, warming, draining, not_ready, or stopped when the node wrote a last row on its way out), build version, the age of its last heartbeat, saturation (busy completion slots over total, the same number /lmkit/v1/health/ready reports), busy and total slots, queue depth, requests per second and the error share of its last sixty seconds, and how many models are resident with how many of its warm-up lineup are still missing (the Inference section of each node lists that lineup with the node's own state per model, and readiness goes back to warming when a lineup model is evicted). A node that has missed three sync periods renders dim and reads stale (the page states the period in force); rows a day old are swept by the nodes themselves.

3Consistency#

Node settings are meant to be identical on every node, and each heartbeat carries a fingerprint of them: one digest per configuration section over the Node-class leaves, a secret entering as a keyed hash so a differing password shows without being shown. The page compares every node's digests to this node's and names the sections that differ (ports, security, inference, and so on); paths that legitimately differ per machine (the ffmpeg path, the context hibernation directory, the memory directory) never enter the fingerprint, and a path under the node's state directory is compared relative to it, so nodes installed at different paths with the same layout read as identical. Lag is how many revisions of the shared objects a node has not applied yet: current after every node's sync period, a count while a write travels. Versions lists every build in play; a rolling upgrade shows two until it finishes. Drift is observed and named here, never repaired: the remedy is the deployment that declares the differing setting (Scaling Out section 4). The Security section's audit asks a fleet node five more questions from the same facts: one shared store for the domain, no drift an Owner has not accepted, no stale row, no embedded search cluster, a balancer declared under Security:TrustedProxies; each answer names the section to fix.

4Forgetting a node#

A node that left for good keeps its last row until the daily sweep. An Owner can forget a stale row from the roster; a live row (a heartbeat within three sync periods) and this node's own row refuse, and a node that comes back writes a new row on its first heartbeat. Forgetting a row changes nothing else: no drain, no restart, no request to the other node.

5Accepting a difference#

A heterogeneous fleet has nodes whose hardware-bound settings legitimately differ: a smaller GPU runs a smaller slot pool. An Owner can accept such a difference per node and per section from the Consistency list (the Accept action beside the section; Accepted withdraws it). The acceptance is stored on that node's own roster row and survives every heartbeat; the section stays named, dimmed rather than flagged, and everyone else reads (accepted) beside it. An acceptance is a statement about one node: the same difference on another node flags until it is accepted there too, and a section that stops differing simply stops appearing.

6Shared stores#

What every node shares, as this node reads it. The identity database: its engine and redacted target, whether this node reaches it, and the live documents per kind of shared object with the head revision of each. The two shared volumes, uploads and models: every node probes its own mount on its heartbeat by writing one file, reopening it exclusively and removing it, and reports the path it mounts, so the page names a node whose path differs, a read-only mount, or a volume that is missing on one node; nothing is created or repaired by a probe. The search clusters: which engine holds the text and which the vectors, or that Search is off on this farm.

7Work#

Single-runner work runs on the node holding its lease in the identity store; a node that stops renewing loses the lease after three of its periods and any sibling takes over. The leases are the security trail retention, the job record retention, the upload volume's sweep, the stale roster row sweep, the skills sources reconcile, and per search cluster its schema migration and the raise of its server settings (Scaling Out section 3 lists them with their names). The list shows each lease, the holder, since when and until when. The Jobs page of every node names the same holder beside each single-runner job, so an operator pinned to one node still sees where the work runs, and a manual run from another node names the holder instead of doubling its pass.

8Domain#

Who owns the shared objects: the identity store, edited from any node's panel, or a deployment manifest re-imported at every start (Scaling Out section 4). The last change names the object, the operator and the node that wrote it. The security trail is domain-wide: sign-ins and failures, configuration and shared-object writes, nodes joining and leaving, acceptances and forgotten rows, one table every node appends to. Its newest lines are here; the full trail with its filters is on the Alerts page. The trail is bounded by the Security trail retention setting of the Access section (Security:TrailRetentionDays, 365 by default, 0 keeps every entry): once an hour the node holding the retention lease removes the entries older than that, on every engine alike.

9Stated plainly#

• The Farm page reads the shared identity store and nothing else; every other Operate surface is one node's own and says so.
• A row is stale after three missed sync periods; only stale rows can be forgotten, by an Owner.
• Drift names the differing sections of the Node class; per-machine paths never count; nothing is repaired from here.
• An accepted difference is per node and per section, stored on the node's row, and survives its heartbeats; it stops flagging, not being named.
• Lag is revisions not yet applied, per node; current is the steady state.
• Volume probes are each node's own view of its mount, written and removed by that node; a differing path is named, never changed.
• Single-runner work runs where its lease is held; the trail's retention is one such job.
• The security audit's fleet checks read the same rows and settings; an accepted difference satisfies the drift check, a forgotten row the stale check.