Observability and Operations

What the server tells you about itself, where, and how to feed it into the monitoring you already run. Everything here is on by default and local; exporting is the opt-in.

1The Operate sections#

The admin console's Operate group is the live view:

Every Operate section is THIS node's view: the registries and the stores behind them (request trail, metrics, telemetry, forensics, logs) are per node, and the admin footer names the node serving the page, so behind a load balancer you always know whose story you are reading; the fleet-wide picture is the export path in section 4 (Scaling Out).

Two quieter records complete the picture: the activity log notes every operator-relevant change (a key minted, a grant changed, a setting saved), and after an abnormal exit the dashboard shows an unexpected-restart report reconstructing what the dying process recorded, so a crash is a lead instead of a mystery. Alerts are evaluated on the node's own clock every few seconds, whether or not a dashboard is open; each new alert is also a line of the activity log (source alert) and, on a farm, part of the node's heartbeat summary.

2Attribution#

Requests are attributed to the API key that made them, so "who is generating this load" is a filter, not an investigation. Per-key use counts and last-used timestamps surface in the Access section beside each key; the anonymous local caller shows as such.

Statistics come in two tiers. Everything in the Operate sections is one node's own and its head says which node; on a farm each node also writes a sixty-second rollup (requests per second, error share, saturation, slots, queue depth, resident models, alerts) into its heartbeat row, and the Farm page reads every node's row from the shared identity store (Farm Operations). Two trails exist: the activity journal is this node's own, and the security trail is domain-wide, one table every node appends to, bounded by Security:TrailRetentionDays (365 by default, 0 keeps every entry) and swept once an hour by the one node holding the retention lease.

3Health for machines#

Three routes, one job each, all answering GET and HEAD:

• GET /health is liveness for uptime probes, without authentication ceremony: 200 while the process is alive, with the product version, the engine version and a boot id that changes on every start (so a client that asked for a restart can prove it happened). It stays 200 while the node warms or drains, so it never decides routing.
• GET /lmkit/v1/health/ready is readiness for any prober, anonymous and status-only: 200 with ready once the inference backend is initialized and every configured warmup model is loaded, 503 with not_ready until the backend is up, warming while the lineup loads, and draining the moment a graceful stop begins. No counts, no model names, no version: it exposes nothing a health check on a bare URL should not see, which is why a platform health check, a balancer probe or a HEAD-only monitor gates traffic on it.
• GET /lmkit/v1/ready is the same decision with the detailed body, behind a bearer token because it exposes runtime state: status (the same four words), backend_initialized, loaded_models (count), loaded_model_names, active_inferences, queue_depth, draining, node (this node's tag, the one X-LMKit-Node carries), version, mode (single_node, or farm when the node shares an identity database with its siblings), slots_saturation (section 4 defines it), fleet_signal (present only on a fleet member, naming why: Farm mode, managed configuration, a shared identity store) and, only while warmup gates readiness, warmup_pending and warmup_failed.

GET /lmkit/v1/search/status answers the same honestly for the Search feature, per caller (see the Access Model).

4Exporting to your stack#

The Telemetry export section wires the server into external monitoring; both paths carry the same measurements the built-in panels show:

• OpenTelemetry (OTLP): push traces and metrics to a configured endpoint (a collector, or any OTLP-speaking backend).
• Prometheus: enable the scrape endpoint and point your Prometheus at it.

The first-party gauges include the one an autoscaler acts on, lmkit_inference_slots_saturation: busy completion slots over total completion slots across the node's resident chat models (embedding pools excluded), from 0 to 1, falling back to active completions over the completion ceiling before any pool exists. It is one number with four readers, this gauge, the dashboard's slots tile, the slots_saturation field of /lmkit/v1/ready and the heartbeat row a farm reads, so they never disagree. Pressure shows there BEFORE a queue forms; lmkit_inference_queue_depth and lmkit_inference_active sit beside it. Three gauges exist for a fleet dashboard: lmkit_build_info (always 1, with version, engine and node labels, so version skew across nodes is one query), lmkit_ready_state (1 when the node answers ready, 0 otherwise, with the status word as a label) and lmkit_model_resident (one series per resident model, labelled model). Every gauge is per node: a fleet autoscaler averages the saturation across nodes (the KEDA example in Running in Containers does exactly that), and exporting is how the per-node panels become one fleet-wide view. Four queries give the farm view: avg(lmkit_inference_slots_saturation) for the load an autoscaler acts on, count by (version) (lmkit_build_info) for version skew, sum(lmkit_ready_state) for how many nodes take traffic, and count by (model) (lmkit_model_resident) for where each model lives. The one record that is already domain-wide without any export is the security source of the audit history: on a shared identity store, sign-ins, account changes and credential events from every node land in the database, and that source reads the whole domain (Operator Accounts).

Exporting is off by default: local observability costs nothing and sends nothing.

5Stated plainly#

• The console answers "what is happening" on this node without any external dependency; the footer names the node.
• Every request has an owner; every operator change has a record; every crash leaves a report.
• /health is liveness; /lmkit/v1/health/ready tells the truth about warming and draining to any prober, and /lmkit/v1/ready adds the detail behind a bearer.
• External monitoring is one section away, in the two formats the ecosystem actually speaks, and it is how a fleet reads itself as one.