Running in Containers

How to run this server from the official image: one docker run on Linux, Windows or macOS, volumes for state and models, GPU passthrough, health-probe wiring, Compose, and Kubernetes in both honest shapes: one pod with node-local state, or an autoscaled fleet that shares its admin domain, uploads, models, and manifest-owned configuration. For self-hosters whose stack is Docker, and platform engineers whose infrastructure is a cluster.

1The official image#

The image is lmkitone/lm-kit-one on Docker Hub, mirrored digest for digest as ghcr.io/lm-kit/lm-kit-one on GitHub Container Registry. One tag serves both linux/amd64 and linux/arm64; Docker picks the variant for the machine it runs on. Tags are the product's own calendar versions (2026.9.0) plus latest, which moves only once a release has shipped every target, so latest never names a build that is not on the download page.

docker run -d --name lmkit -p 5189:5189 -p 7221:7221 \
  -v lmkit-state:/data/state -v lmkit-models:/data/models \
  lmkitone/lm-kit-one

With an NVIDIA GPU, one flag more (section 4 has the host prerequisite):

docker run -d --name lmkit --gpus all -p 5189:5189 -p 7221:7221 \
  -v lmkit-state:/data/state -v lmkit-models:/data/models \
  lmkitone/lm-kit-one

The API is at http://localhost:5189 and the admin panel at https://localhost:7221/admin. The panel travels over TLS only: the server accepts a plain-HTTP admin request from its own loopback alone, and through a published port every host client is a network peer, so a plain-HTTP page request is redirected to the HTTPS port, which serves the server's self-signed certificate and the browser asks once to trust it. On a trusted private network -e Security__RequireHttpsForAdmin=false lifts the rule (Going Live). The first sign-in is the admin account with a one-time password: a container is reachable from the network, so the server never leaves its panel open and instead creates that account at its first start and prints the password once, in the startup log (docker logs lmkit); the sign-in replaces it with a password of your own. To choose the first password yourself, seed it from the environment, -e Admin__InitialPassword=... (section 2). Every API call needs a key, minted in Access (Keys and Authentication); the health routes stay anonymous.

Windows and macOS hosts run the same image through Docker Desktop, which hosts Linux containers in its own virtual machine. On Windows the NVIDIA driver reaches the container through WSL2, so --gpus all works unchanged with nothing else installed. On macOS the container is CPU-only, because Metal never crosses into a container; a Mac that wants its GPU runs the native package instead. There is no Windows-native container image: those cannot use CUDA, need a host kernel matching the base image, and nothing in this space ships one. A Windows server that wants native performance takes the MSI and its service mode (Windows Server).

Upgrading is docker pull lmkitone/lm-kit-one, then remove and recreate the container with the same volumes: everything the server has written rides the state volume across it (Backup and Upgrades).

Stopping. docker stop gives a container 10 seconds by default, and a busy server drains in-flight completions for up to 100 (ShutdownDrainSeconds) plus a short teardown; the startup log says so on every start under Docker. docker stop -t 120 lmkit, or --stop-timeout 120 on the run line, gives a graceful stop the room it needs; Compose spells it stop_grace_period (section 6).

Verify what the container sees with docker exec lmkit lmkit devices, and the whole environment with docker exec lmkit lmkit doctor (The Command Line).

A container remains optional, not required. Where a cluster is not mandated, the service install in Going Live is a first-class path; containers buy orchestration, not portability the binary lacks.

2What the image decides#

Five decisions shape the image, each worth understanding before building on it:

• Base image. Ubuntu 24.04 LTS, the release the payload is built and packaged on, so the base is known compatible with the native stack. The binary targets glibc (the linux-x64 and linux-arm64 runtimes), so musl bases such as Alpine are out. What the image adds to the payload is small and deliberate: libicu74, without which the .NET runtime refuses to start; ca-certificates, so the server can pull catalog models over HTTPS; curl, for the health check; the CUDA 13 and CUDA 12 runtimes and the Vulkan loader with Mesa's drivers, so a GPU needs nothing but its host driver (section 4), with vulkaninfo beside them to read what the loader sees from inside the container; and ffmpeg, next.
• ffmpeg inside. Audio and video transcription and video frame extraction run ffmpeg as a separate program; a host install brings its own, or downloads the server-managed build from the admin panel. The image carries it already: the static build the server would download on a Linux host, pinned by release and checksum, on the PATH, so a recording in any container or codec (a video's audio track included) transcribes on the first request with nothing installed and nothing fetched, and the capabilities route advertises the full set (Transcription). An operator's own build still wins, through LMKIT_FFMPEG_PATH or the panel's FFmpeg Path. ffmpeg is GPL software, included unmodified; its license, version and build configuration are recorded in the image under /usr/share/doc/ffmpeg.
• Network posture. By default the server listens on loopback only (Going Live). Inside a container that makes it unreachable through published ports, which connect to the container's own interface, not its loopback. The image therefore sets Security__NetworkAccess=Network (environment variables override the settings file). The consequence is deliberate: in Network posture every API call must present a key, which is the right default for a server that is, by definition, on a network.
• Pinned paths. LMKIT_STATE_DIR=/data/state and LMKIT_MODELS_DIR=/data/models, both declared as volumes, so state never lands in the container's writable layer, where docker rm would delete it (section 3).
• Health check. The liveness route is GET /health, always anonymous (section 5); the image's HEALTHCHECK probes it at half-minute intervals after a one-minute start period.

The process runs as root, as most inference-server images do, so bind mounts work without a chown step. --user <uid> works when both mounted volumes are writable by that uid; lmkit doctor inside the container verifies paths, writability and devices in one pass.

The image exposes both ports: 5189 for the API over plain HTTP and 7221 for HTTPS with the self-managed certificate, which is where the admin panel lives from anywhere but the container's own loopback (section 1). In a cluster TLS normally terminates at the ingress or proxy in front of the container instead; list it in Security__TrustedProxies so its forwarded scheme counts as HTTPS for the admin rule, and take the buffering and forwarded-header specifics that keep streaming working from Behind a Reverse Proxy. A kubectl port-forward reaches the pod over its loopback and needs neither. The first operator account is created at the first start, one way or the other: seeded from the environment (Admin__InitialPassword becomes the password of the local admin account, applies only while no operator account exists yet, and is never stored in plain text), or, with nothing seeded, generated by the server, which prints a one-time admin password once in its startup log and makes the first sign-in replace it. Once any account exists the seed is ignored, restart after restart; Admin__Password is the emergency reset, re-applied on every start it is present, so it never belongs in a manifest for longer than one restart (Operator Accounts section 3). Then mint per-caller API keys in Access (Keys and Authentication).

Building on the image. A site that needs more than the environment can express (a corporate certificate authority, a model baked in, a binary an agent tool calls) extends the official image rather than starting from the archive, and rebuilds when a new version tag appears:

FROM lmkitone/lm-kit-one:2026.9.0

# A private certificate authority, for model mirrors and tools behind it.
COPY corp-root-ca.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

# A model baked into the image: `lmkit pull` needs no running server and
# writes to LMKIT_MODELS_DIR, which the image has already pinned.
RUN lmkit pull gemma4:12b

Everything the base image set (ports, volumes, health check, entrypoint) is inherited. A stage that changes LMKIT_MODELS_DIR must pull after the change, so the file lands where the runtime reads.

3State and models on volumes#

Two mounts, two lifetimes:

Pinning both explicitly matters most in a container: without LMKIT_STATE_DIR, the writability rule would resolve state next to the executable, inside the container's ephemeral layer, and the installation would silently die with the container.

Both mounts are read-write by default, and the model volume can be read-only when the deployment wants immutable models. Models download into the model volume on first use, and not only the chat models: the transcription, embedding, reranking and image segmentation models arrive the same way, on the first request that needs them, so a read-only volume must hold every model the deployment needs, pulled beforehand (lmkit pull <model> into the volume, or baked into the image, below). The server also writes its training jobs and imported models beside the models; ModelWorkDirectory moves those writes to a directory of their own, which is what a read-only deployment sets:

docker run -d --name lmkit -p 5189:5189 -p 7221:7221 \
  -v lmkit-state:/data/state -v lmkit-models:/data/models:ro \
  -e ModelWorkDirectory=/data/state/work \
  lmkitone/lm-kit-one

A request that needs a model the read-only volume lacks answers 503 model_store_read_only naming the model to pull; a training job on a read-only volume with no work directory is refused the same way, naming the setting. The startup log, the dashboard status and lmkit doctor report the posture on every start.

Mounting versus baking models. Mounting (above) is the default: the volume outlives image rebuilds and upgrades. The alternative is baking models into the image at build time, with lmkit pull in a stage on top of the official image exactly as section 2 shows (The Command Line). The image grows by the model file, but the result is a single immutable artifact that serves on first start with zero egress: the container form of the golden image (Air-Gapped Deployment). Pick one strategy per deployment; a baked image with a models volume mounted over /data/models hides its own baked files.

4GPU passthrough#

NVIDIA (CUDA) is the path that works predictably. On Linux, install the NVIDIA Container Toolkit on the host and run with --gpus all (or the Compose and Kubernetes equivalents below); on Windows, Docker Desktop reaches the host's NVIDIA driver through WSL2 and the same flag works with nothing else installed. The image ships the CUDA 13 runtime libraries (cudart, cublas, cublasLt) and, for a host whose driver is too old for CUDA 13, the CUDA 12 set, both registered with the container's dynamic linker: the container needs no CUDA installation of its own, the host contributes the driver, and nothing else. Verify what the runtime actually sees:

docker exec lmkit lmkit devices

If no GPU is listed, the server still serves, on CPU; the probe result is the fact to trust, not the run flags. Backend selection and driver details are GPUs, Drivers, and Backends.

Podman passes GPUs through the Container Device Interface (CDI): the device name nvidia.com/gpu=all must be described by a specification on the host, which the NVIDIA Container Toolkit writes and lists:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
nvidia-ctk cdi list

Then the run line names the device (--gpus all resolves to the same device):

podman run -d --name lmkit --security-opt=label=disable --device nvidia.com/gpu=all \
  -p 5189:5189 -p 7221:7221 \
  -v lmkit-state:/data/state -v lmkit-models:/data/models \
  lmkitone/lm-kit-one

--security-opt=label=disable is for hosts enforcing SELinux, which is every Fedora, RHEL and Rocky machine as installed: SELinux confines the container's processes away from the device nodes and driver libraries the specification injects, so the container starts and the runtime finds no GPU. The flag is a no-op on a host without SELinux, so the line is safe to keep as is. Regenerate the specification after every driver upgrade: it pins the driver's library versions. On Windows, Podman runs its containers inside a WSL2 machine, so the toolkit is installed and the specification generated in there: podman machine ssh, the toolkit package from NVIDIA's repository, then the same two commands; the Podman Desktop GPU page carries the package commands for each machine image.

When the GPU is not seen. The runtime's own reading is the fact to trust: lmkit devices inside the container (docker exec lmkit lmkit devices, podman exec lmkit lmkit devices) and the startup log (docker logs lmkit, podman logs lmkit), which names the reason a CUDA backend was set aside. In a container the CUDA runtime libraries are inside the image, so what the host's ldconfig lists never enters into it; the two questions are whether the GPU reached the container and whether the process may use it.

AMD and Intel (Vulkan) work on a Linux host with one flag: --device /dev/dri hands the GPU's render node to the container, and the image carries the rest, the Vulkan loader and Mesa's drivers (RADV for AMD, ANV for Intel). Mesa drives the GPU from user space over the kernel's stable interface, so nothing in the image has to match the host's driver version; the host needs only its kernel driver (amdgpu, i915 or xe), which every current distribution ships. Running with --user adds --group-add for the group that owns /dev/dri/renderD* (render or video). Mesa's software rasterizer is not in the image, so a CPU is never mistaken for a GPU. There is no Vulkan path on a Windows or macOS host: Docker Desktop passes NVIDIA GPUs only, through WSL2 and CUDA, and on those hosts an AMD or Intel GPU serves from the native package instead.

docker run -d --name lmkit --device /dev/dri -p 5189:5189 -p 7221:7221 \
  -v lmkit-state:/data/state -v lmkit-models:/data/models \
  lmkitone/lm-kit-one

Podman takes the same flag; on an SELinux host add --security-opt=label=disable, or the render node is visible but refused:

podman run -d --name lmkit --security-opt=label=disable --device /dev/dri -p 5189:5189 -p 7221:7221 \
  -v lmkit-state:/data/state -v lmkit-models:/data/models \
  lmkitone/lm-kit-one

Nothing has to be installed inside the container, and VK_DRIVER_FILES never needs setting: the loader reads the manifests the image ships. When the Hardware panel still reads RUNNING ON CPU, docker exec lmkit vulkaninfo --summary (or podman exec) is the loader's own account: a listed device with driverName = radv (AMD) or Intel open-source Mesa driver means the GPU reached the container and the SDK elects it on the next start; Failed to detect any valid GPUs means no render node reached the container (the --device flag is missing, or SELinux refused it); a device whose type is CPU would be a software rasterizer, which the image does not carry and the SDK never elects.

5Health probes#

Three routes, built for orchestration and wired in the manifests below; the full contract is in Observability:

The manifests probe the anonymous readiness route, so no key sits in a manifest and the readinessProbe gates on warming and draining for free. The bearer route exists for probers that want the body, through Kubernetes httpHeaders carrying a dedicated key granted nothing else (Keys and Authentication), and for KEDA's metrics-api scaler in section 9. Probers that cannot send a header (a cloud load balancer's health check, IIS ARR, the App Service health check) use the anonymous readiness route and keep the gate (Fleet Recipes has each platform's line). The image's HEALTHCHECK stays on /health: Docker and Swarm restart a container whose check fails, and a node that is warming or draining must not be restarted for it.

6Compose#

A complete single-host deployment: the official image, both volumes, the GPU reservation, a stop window sized for the drain, and a published port. This shape is equally the homelab answer: a VM on a hypervisor or a NAS runs it identically, with docker compose up -d.

services:
  lmkit:
    image: lmkitone/lm-kit-one:latest   # pin a version (lmkitone/lm-kit-one:2026.9.0) for reproducible rollouts
    container_name: lmkit
    restart: unless-stopped
    hostname: lmkit-1            # a stable node name; the default is the container id, new on every recreate
    stop_grace_period: 120s      # advertise + ShutdownDrainSeconds + teardown (section 1)
    ports:
      - "5189:5189"              # API, plain HTTP
      - "7221:7221"              # admin panel, HTTPS (section 1)
    environment:
      Admin__InitialPassword: "change-on-first-login"
    volumes:
      - lmkit-state:/data/state
      - lmkit-models:/data/models
    deploy:                      # NVIDIA. For an AMD or Intel GPU, replace this block with:
      resources:                 #   devices:
        reservations:            #     - /dev/dri:/dev/dri
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  lmkit-state:
  lmkit-models:

Other containers on the Compose network reach the server at http://lmkit:5189; host clients use the published port. Drop the deploy block on CPU-only hosts. The hostname: line (or LMKIT_NODE_NAME in the environment) gives the node a name that survives docker compose up recreating the container: without it the node is named after the container id, and a fine-tuning run journaled under the previous id is listed under a node that no longer exists until the heartbeat rule adopts it (Scaling Out section 6, Fine-tuning).

7Kubernetes: one pod, or a fleet#

Two honest shapes, and the difference between them is exactly what is shared:

One pod keeps everything node-local: state and models on ReadWriteOnce PVCs, panel edits persisted, Recreate strategy so a rollout never puts two writers on the same volumes. This remains the right shape for a single box's worth of load.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lmkit-one
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels: { app: lmkit-one }
  template:
    metadata:
      labels: { app: lmkit-one }
    spec:
      containers:
        - name: lmkit
          image: lmkitone/lm-kit-one:<version>   # a version tag, never latest, so a rollout is deliberate
          ports:
            - containerPort: 5189
          resources:
            limits:
              nvidia.com/gpu: 1
          livenessProbe:
            httpGet: { path: /health, port: 5189 }
            periodSeconds: 15
          readinessProbe:
            httpGet: { path: /lmkit/v1/health/ready, port: 5189 }   # anonymous, status only
            periodSeconds: 15
            failureThreshold: 4
          volumeMounts:
            - { name: state, mountPath: /data/state }
            - { name: models, mountPath: /data/models }
      volumes:
        - name: state
          persistentVolumeClaim: { claimName: lmkit-state }
        - name: models
          persistentVolumeClaim: { claimName: lmkit-models }

The nvidia.com/gpu resource requires the NVIDIA device plugin on the cluster; omit the limit on CPU nodes. Backups in this shape are snapshots of the state PVC, taken with the pod stopped for a clean copy, exactly as the runbook prescribes.

A fleet scales by replication: every pod is a complete engine with its own resident models, and what is shared is exactly three things plus the manifest: the admin-domain database, the upload volume, the model volume, and the configuration the manifest owns. The operator plane (dashboard, logs, request history) stays each pod's own view, and so do memory facts, panel-authored skills, custom-model records and the live sessions the pod created; the admin footer names the pod serving it. The platform-independent contract behind these manifests, the setup of the shared admin domain (Scaling Out section 3), managed configuration (section 4), what honestly stays per node (section 5) and which paths the ingress must pin (section 6) are all Scaling Out.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lmkit-one
spec:
  replicas: 2                     # KEDA drives this from here (section 9)
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }  # warm joins + drains keep rollouts lossless
  selector:
    matchLabels: { app: lmkit-one }
  template:
    metadata:
      labels: { app: lmkit-one }
      annotations:                # or a ServiceMonitor, where the Prometheus Operator runs
        prometheus.io/scrape: "true"
        prometheus.io/port: "5189"
        prometheus.io/path: "/metrics"
    spec:
      terminationGracePeriodSeconds: 120   # above DrainAdvertiseSeconds + ShutdownDrainSeconds
      containers:
        - name: lmkit
          image: lmkitone/lm-kit-one:<version>
          ports:
            - containerPort: 5189
          env:
            - name: LMKIT_MANAGED_CONFIG          # the manifest owns configuration (Scaling Out section 4)
              value: "1"
            - name: LMKIT_NODE_NAME               # the node name behind X-LMKit-Node and lmk-node (Scaling Out section 6);
              valueFrom:                          # a Deployment's pod name changes on every rollout, a StatefulSet's does not
                fieldRef:
                  fieldPath: metadata.name
            - name: LMKIT_STATE_DIR
              value: /data/state
            - name: LMKIT_MODELS_DIR
              value: /data/models
            - name: FileManagement__UploadDirectory
              value: /data/uploads
            - name: Security__NetworkAccess
              value: Network
            - name: Security__TrustedProxies__0   # the ingress controller's pods ALONE, never the cluster's pod CIDR:
              value: "10.244.7.0/24"              # every address here may set X-Forwarded-For (see below)
            - name: DefaultChatModel
              value: "qwen3.5:4b"
            - name: Inference__WarmupModels__0    # readiness waits for the lineup (section 9)
              value: "default"
            - name: Telemetry__Enabled
              value: "true"
            - name: Telemetry__EnablePrometheus
              value: "true"
            - name: Admin__Identity__Store        # the shared admin domain (Scaling Out section 3)
              value: postgres
            - name: Admin__Identity__ConnectionString
              valueFrom:
                secretKeyRef: { name: lmkit-identity, key: connection-string }
            - name: Search__Enabled               # a fleet member never provisions the embedded cluster;
              value: "false"                      # either switch Search off explicitly ...
            - name: Search__AutoProvisionLocalCluster
              value: "false"
            # ... or declare one external cluster every pod reaches (replace the two rows above):
            # - name: Search__Enabled
            #   value: "true"
            # - name: Search__Clusters__0__Id
            #   value: main
            # - name: Search__Clusters__0__IsDefault
            #   value: "true"
            # - name: Search__Clusters__0__FullTextStore
            #   value: PostgreSql
            # - name: Search__Clusters__0__ConnectionString
            #   valueFrom:
            #     secretKeyRef: { name: lmkit-search, key: connection-string }
          resources:
            limits:
              nvidia.com/gpu: 1
          livenessProbe:
            httpGet: { path: /health, port: 5189 }
            periodSeconds: 15
          readinessProbe:
            httpGet: { path: /lmkit/v1/health/ready, port: 5189 }   # anonymous, status only
            periodSeconds: 5      # the drain advertisement is 5s; probe inside it
            failureThreshold: 2
          volumeMounts:
            - { name: state, mountPath: /data/state }
            - { name: models, mountPath: /data/models }
            - { name: uploads, mountPath: /data/uploads }
      volumes:
        - name: state
          emptyDir: {}            # the per-pod operator history (request trail, logs, forensics);
                                  # memory facts, skills and custom-model records live in the shared admin database
                                  # (Scaling Out section 3), and custom-model blobs on the shared model volume.
        - name: models
          persistentVolumeClaim: { claimName: lmkit-models-shared }
        - name: uploads
          persistentVolumeClaim: { claimName: lmkit-uploads-shared }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lmkit-models-shared
spec:
  accessModes: [ReadWriteMany]    # every pod reads; a pod pulling a model writes once
  resources: { requests: { storage: 200Gi } }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lmkit-uploads-shared
spec:
  accessModes: [ReadWriteMany]
  resources: { requests: { storage: 50Gi } }
---
apiVersion: v1
kind: Service
metadata:
  name: lmkit-one
spec:
  selector: { app: lmkit-one }
  ports:
    - port: 5189
      targetPort: 5189

Front the Service with your ingress and terminate TLS there; streamed responses need the ingress-side buffering settings from Behind a Reverse Proxy, and the affinity is two rules the same guide gives per ingress: cookie affinity on /admin, /lmkit/v1/admin and /playground, and Authorization hashing on everything else, because a job id is minted by whichever endpoint was asked for async work and skip-thinking targets the exchange streaming on /lmkit/v1/chat, so a path list cannot pin them. Keep Security__TrustedProxies__0 to the ingress controller's own pods (its namespace's range or its pod selector's addresses) and add a NetworkPolicy that lets only the ingress reach the Service: every address in that list may set X-Forwarded-For, so a wider range lets any pod in it impersonate an allowlisted operator address or a loopback caller to the sign-in throttle, the admin IP allowlist and the trail. The KEDA ScaledObject that grows and shrinks replicas is in section 9, and the rest of the fleet's behavior is built in: a new pod reports warming until its lineup is resident, a stopping pod advertises draining while it finishes its streams, two pods asked for the same missing model pull it once (a cross-process lock on the shared model volume; the second pod waits, then loads the file the first one fetched), and every pod refuses configuration edits because the manifest owns them.

A pod carrying a fleet signal (managed configuration, or a shared identity store) never provisions the embedded search cluster: with neither Search__Enabled nor a cluster declared, Search stays disabled and the pod names the two remedies in its log and in the Search panel, declare Search__Clusters on an external engine every pod reaches, or set Search__Enabled=false to keep Search off deliberately. The manifest above takes the second remedy and shows the first in comments; the embedded engine would be one pod's file on the emptyDir state volume, diverging per pod (Storage Engines and Deployment).

8One admin domain across nodes#

The fleet manifest points every pod at one dedicated identity database, and that database is what makes operators, sessions, API keys and stored responses one domain; the rules behind it (the dedicated-database requirement, how the first operator is seeded, the cache windows, and what happens while the database is unreachable) are Scaling Out section 3. The recipe is the Admin__Identity__* rows in the section 7 manifest, or the same keys in a mounted settings file:

{
  "Admin": {
    "Identity": {
      "Store": "postgres",
      "ConnectionString": "Host=db;Database=lmkit_identity;Username=lmkit;Password=..."
    }
  }
}

9Autoscaling with KEDA#

The serving signal, the probes, and the drain behavior are built in; KEDA supplies the scaling loop. Scaling is by replication: every pod is a complete engine with its own resident models, and the fleet grows or shrinks on inference pressure.

The signal. With telemetry and the Prometheus endpoint enabled, /metrics exports the first-party gauges anonymously: lmkit_inference_slots_saturation (busy share of the concurrent-completion budget, 0 to 1: scale out as it approaches 1, because the queue only grows after it gets there), lmkit_inference_queue_depth, and lmkit_inference_active. On a Prometheus-less stack, /lmkit/v1/ready serves the same number as slots_saturation (with queue_depth beside it) for KEDA's metrics-api scaler, authenticated with an API key.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: lmkit-one
spec:
  scaleTargetRef: { name: lmkit-one }
  minReplicaCount: 1            # scale-to-zero is dishonest for minute-scale model loads
  cooldownPeriod: 600           # models are expensive to load; shed capacity slowly
  triggers:
    - type: prometheus
      metricType: Value         # the query is already a fleet average; AverageValue would divide it by the replica count again
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: avg(lmkit_inference_slots_saturation)
        threshold: "0.8"        # add a pod while the fleet still absorbs the load

The metrics-api variant for stacks without Prometheus:

  triggers:
    - type: metrics-api
      metricType: Value         # one pod's reading, sampled through the Service; Prometheus averages properly
      metadata:
        url: http://lmkit-one.default:5189/lmkit/v1/ready
        valueLocation: slots_saturation
        targetValue: "0.8"
      authenticationRef: { name: lmkit-api-key }   # Bearer via a TriggerAuthentication

Warmup gates readiness. A pod KEDA adds is useless until its models are resident, so opt in with Inference:WarmupModels (model ids, or default for the default chat model): both readiness routes answer 503 warming until the lineup is loaded, the load balancer holds traffic back, and a model that cannot load parks the pod visibly not-ready (warmup_failed names it) instead of letting it serve refusals. Pair it with the model delivery choice from section 3: image-baked or pre-warmed volumes make warmup fast; pull-on-demand makes the first pod slow exactly once.

Scale-down drains. The moment a pod receives its stop signal, /ready flips to 503 draining and the listener stays open for a short advertise window (default 5 seconds on Kubernetes and on the other platforms the server detects as routing by readiness; top-level DrainAdvertiseSeconds overrides it, 0 disables): endpoint deregistration races the stop signal, and the window serves the requests a load balancer routes before it observes the failed probe, so no preStop sleep hook is needed. The listener then closes while in-flight requests, streamed completions included, finish within ShutdownDrainSeconds (default 100); liveness stays healthy throughout, so nothing kills the pod mid-drain. Set terminationGracePeriodSeconds above advertise plus drain plus 15 seconds of teardown, 120 with the defaults; the server cannot read the pod's value, so its startup warning assumes the platform default.

10Managed configuration#

A fleet's configuration belongs to the manifest, not to any one pod: LMKIT_MANAGED_CONFIG=1 (or top-level ManagedConfiguration: true in a mounted settings file) makes the deployment the owner, and the exact split, refused versus editable, is Scaling Out section 4. What is container-specific: settings come from double-underscore environment keys or a ConfigMap-mounted file, the environment wins where both state a value, and the process never writes appsettings.json, so a read-only mount and a read-only root filesystem both work. Secrets belong in the environment or a secretKeyRef, never in the mounted file, because the file is read as written and an exported configuration masks them. Deployment__DomainSource=Manifest keeps the shared objects (agents, connectors, policies, clusters) in the manifest as well; without it they live in the identity store and are edited from any pod's panel (Scaling Out section 4).

env:
  - name: LMKIT_MANAGED_CONFIG
    value: "1"
  - name: LMKIT_STATE_DIR
    value: /state
  - name: Inference__WarmupModels__0
    value: "default"

11First-run refusals#

Five answers meet most first container runs. Each names its cause; this is where they are read together.

12Stated plainly#

• The official image is lmkitone/lm-kit-one on Docker Hub and ghcr.io/lm-kit/lm-kit-one, one tag for both architectures, Ubuntu 24.04 LTS underneath. Windows and macOS run it through Docker Desktop, and a site that needs more extends it with a FROM line instead of rebuilding from the archive.
• Inside a container the posture is Network, so every API call needs a key; /health stays anonymous for probes, /ready does not. The admin panel is HTTPS-only from any published port: reach it on 7221, or through a TLS-terminating ingress listed as a trusted proxy. The first sign-in is admin with the one-time password the startup log printed, unless Admin__InitialPassword seeded one.
• The image pins LMKIT_STATE_DIR and LMKIT_MODELS_DIR onto volumes; the state volume is the installation, the model volume is a bandwidth saver, and baking models instead is the air-gapped play. The model volume may be mounted read-only once every model is pulled beforehand and ModelWorkDirectory gives training jobs and imported models a writable home.
• ffmpeg rides inside the image, so transcription accepts video containers and every audio codec with nothing installed on the host, and the transcription capabilities route says so.
• CUDA passthrough via the NVIDIA Container Toolkit (or Docker Desktop's WSL2 on Windows) works with nothing installed beyond what the image ships: the CUDA 13 and CUDA 12 runtimes ride inside. Podman needs the toolkit's CDI specification generated first and, on an SELinux host, --security-opt=label=disable; section 4 reads the symptoms of a GPU that did not reach the container. AMD and Intel GPUs serve through Vulkan on a Linux host with --device /dev/dri: the loader and Mesa's drivers ride inside too.
• Operator identity scales across installations through a dedicated shared database (Admin:Identity); document content and identity never share one, and the server refuses a configuration that mixes them.
• Scaling is by replication: each pod is a complete engine with its own resident models. Sharing the admin-domain database and the upload volume makes any pod serve the stateless surfaces and the shared records (grounded chat sessions, MCP sessions, async jobs, memory facts, skills and custom-model records live in that database); chat-with-document sessions and the MCP event stream stay on the pod that opened them, so the ingress hashes everything outside the console paths by Authorization and cookie-pins the console, and the operator plane stays per pod (Scaling Out sections 5 and 6). KEDA grows the fleet on the saturation gauge (section 9), and the admin footer names the pod.
• A fleet member never provisions the embedded search cluster: declare an external cluster or set Search__Enabled=false with Search__AutoProvisionLocalCluster=false.
• A fleet runs LMKIT_MANAGED_CONFIG=1: the manifest owns configuration and every configuration write answers 409 (Scaling Out section 4); a single pod with a persistent state volume may keep panel edits instead.