Observability and Operations
What the server tells you about itself, where, and how to feed it into the monitoring you already run. Everything here is on by default and local; exporting is the opt-in.
1The Operate sections#
The admin console's Operate group is the live view:
| Section | What it answers |
|---|---|
| Dashboard | Is it healthy right now: active inferences, requests, jobs, queue depth, loaded models, storage, and server controls |
| Requests | Every API call, attributed to its key: route, status, timing; filter and drill into one request |
| Logs | The server log, live, with level filtering |
| Alerts | Conditions that deserve attention, stated in plain words |
| Metrics | Rates and latencies over time |
| Jobs | Long-running work (indexing, downloads, training) with progress and outcomes |
Two quieter records complete the picture: the activity log notes every operator-relevant change (a key minted, a grant changed, a setting saved), and after an abnormal exit the dashboard shows an unexpected-restart report reconstructing what the dying process recorded, so a crash is a lead instead of a mystery.
2Attribution#
Requests are attributed to the API key that made them, so "who is generating this load" is a filter, not an investigation. Per-key use counts and last-used timestamps surface in the Access section beside each key; the anonymous local caller shows as such.
3Health for machines#
GET /health answers liveness and version for load balancers and uptime probes without
authentication ceremony. GET /lmkit/v1/ready is the readiness probe: 200 once the inference
backend is initialized and 503 until then, with runtime state in the body (loaded model names
and count, active inferences, queue depth); because it exposes internal state it requires a
bearer token, so a readiness check sends an Authorization header. GET /lmkit/v1/search/status
answers the same honestly for the Search feature, per caller (see the
Access Model).
4Exporting to your stack#
The Telemetry export section wires the server into external monitoring; both paths carry the same measurements the built-in panels show:
- OpenTelemetry (OTLP): push traces and metrics to a configured endpoint (a collector, or any OTLP-speaking backend).
- Prometheus: enable the scrape endpoint and point your Prometheus at it.
The first-party gauges include the one an autoscaler acts on:
lmkit_inference_slots_saturation, the 0-to-1 busy share of the completion budget, where
pressure shows BEFORE a queue forms; lmkit_inference_queue_depth and
lmkit_inference_active sit beside it. On a load-balanced fleet, exporting is also how the
per-node panels become one fleet-wide view (Scaling Out).
Exporting is off by default: local observability costs nothing and sends nothing.
5Stated plainly#
- The console answers "what is happening" without any external dependency.
- Every request has an owner; every operator change has a record; every crash leaves a report.
- External monitoring is one section away, in the two formats the ecosystem actually speaks.