Running in Containers

How to run this server in a container you build: a reference Dockerfile over the self-contained Linux archive, volumes for state and models, GPU passthrough, health-probe wiring, Compose, and Kubernetes in both honest shapes: one pod with node-local state, or an autoscaled fleet that shares its admin domain, uploads, models, and manifest-owned configuration. For platform engineers whose infrastructure is a cluster, and self-hosters whose stack is Compose.

1No official image today#

There is no official container image, registry path, or published Dockerfile. If you deploy in containers, you build the image, you pick and patch the base, and you rebuild when a new version ships. That is the support posture, stated up front.

What keeps that burden small: the Linux build is a self-contained, single-file executable with its native libraries beside it, so the image is one archive extracted onto one base, with no runtime to install and no dependency manifest to track across releases. The same property cuts the other way: a container is optional, not required. Where a cluster is not mandated, the service install in Going Live is a first-class path; containers buy orchestration, not portability the binary lacks. The artifact is the Linux x64 (or Arm64) archive from the download page, LM-Kit-One-<version>-linux-<arch>.tar.gz; it is rooted at the payload, so extraction yields the server files directly, with the lmkit executable at the top. An upgrade is a new archive, a rebuild, a redeploy; everything the server has written rides the state volume across it (Backup and Upgrades).

2The reference Dockerfile#

Four decisions shape it, each worth understanding first:

• Base image. The release payload is built and packaged on Ubuntu 24.04, so that base is known compatible with the native stack. The binary targets glibc (the linux-x64 runtime), so musl bases such as Alpine are out. Slimmer glibc bases can work, but verifying the native libraries' shared-library needs becomes your job; when in doubt, stay on the LTS base.
• Network posture. By default the server listens on loopback only (Going Live). Inside a container that makes it unreachable through published ports, which connect to the container's own interface, not its loopback. The image therefore sets Security__NetworkAccess=Network (environment variables override the settings file). The consequence is deliberate: in Network posture every API call must present a key, which is the right default for a server that is, by definition, on a network.
• Pinned paths. LMKIT_STATE_DIR and LMKIT_MODELS_DIR are set explicitly so state never lands in the container's writable layer, where docker rm would delete it (section 3).
• Health check. The liveness route is GET /health, always anonymous (section 5). curl exists in the image for exactly this; ca-certificates lets the server pull models from the catalog over HTTPS, since the Ubuntu base ships without CA roots.

FROM ubuntu:24.04

# curl serves the health check; ca-certificates lets the server
# pull catalog models over HTTPS.
RUN apt-get update \
 && apt-get install -y --no-install-recommends ca-certificates curl \
 && rm -rf /var/lib/apt/lists/*

# The Linux x64 archive from the download page, placed in the build
# context. ADD extracts it; the archive is rooted at the payload, so
# the server files land directly under /opt/lmkit.
ADD LM-Kit-One-<version>-linux-x64.tar.gz /opt/lmkit/
RUN chmod +x /opt/lmkit/lmkit

ENV LMKIT_STATE_DIR=/data/state \
    LMKIT_MODELS_DIR=/data/models \
    Security__NetworkAccess=Network

EXPOSE 5189

HEALTHCHECK --interval=30s --timeout=5s --start-period=60s \
  CMD curl -fsS http://localhost:5189/health || exit 1

ENTRYPOINT ["/opt/lmkit/lmkit"]

The default HTTP port is 5189. The server also serves HTTPS (default port 7221) with a self-managed certificate, but in a cluster TLS normally terminates at the ingress or proxy in front of the container; the buffering and forwarded-header specifics that keep streaming working are Behind a Reverse Proxy. Seed the first operator account through the environment (Admin__InitialPassword): it becomes the password of the local admin account, applies only while no operator account exists yet, and is never stored in plain text. Then mint per-caller API keys in Access (Keys and Authentication).

3State and models on volumes#

Two mounts, two lifetimes:

Pinning both explicitly matters most in a container: without LMKIT_STATE_DIR, the writability rule would resolve state next to the executable, inside the container's ephemeral layer, and the installation would silently die with the container.

Mounting versus baking models. Mounting (above) is the default: the volume outlives image rebuilds and upgrades. The alternative is baking models into the image at build time, which the CLI supports because lmkit pull needs no running server (The Command Line), placed after the ENV block so the pull lands in the same directory the runtime reads:

RUN /opt/lmkit/lmkit pull gemma4:12b

The image grows by the model file, but the result is a single immutable artifact that serves on first start with zero egress: the container form of the golden image (Air-Gapped Deployment). Pick one strategy per deployment; a baked image with a models volume mounted over /data/models hides its own baked files.

If the container runs as a non-root user, the mounted volumes must be writable by that uid; lmkit doctor inside the container verifies paths, writability, and devices in one pass.

4GPU passthrough#

NVIDIA (CUDA) is the path that works predictably. Install the NVIDIA Container Toolkit on the host and run with --gpus all (or the Compose and Kubernetes equivalents below). The archive ships the CUDA runtime libraries beside the binary, so the container needs no CUDA installation of its own: the host contributes the driver through the toolkit, and nothing else. Verify what the runtime actually sees:

docker exec <container> /opt/lmkit/lmkit devices

If no GPU is listed, the server still serves, on CPU; the probe result is the fact to trust, not the run flags. Backend selection and driver details are GPUs, Drivers, and Backends.

Vulkan in containers is fiddly, and honestly so. There is no equivalent of the NVIDIA toolkit: you must pass the render devices (/dev/dri) into the container and provide vendor ICD loader libraries inside it that match the host driver, a pairing that breaks on driver upgrades. On NVIDIA hardware, use the CUDA path. On other GPUs, weigh a CPU-only container against running the server on the host, where Vulkan support is straightforward.

5Health probes#

Two routes, built for orchestration and wired in the manifests below; the full contract is in Observability:

The readiness route requires a key because its body exposes runtime state. In Kubernetes the probe's httpHeaders carry the bearer token as a literal value in the manifest, so mint a dedicated key for probes and grant it nothing else (Keys and Authentication). Where a plaintext key in a manifest is unacceptable, probe /health for both and accept that scheduling will not wait for backend initialization.

6Compose#

A complete single-host deployment: the section 2 image, both volumes, the GPU reservation, and a published port. This shape is equally the homelab answer: a VM on a hypervisor or a NAS runs it identically.

services:
  lmkit:
    build: .
    restart: unless-stopped
    ports:
      - "5189:5189"
    environment:
      Admin__InitialPassword: "change-on-first-login"
    volumes:
      - lmkit-state:/data/state
      - lmkit-models:/data/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  lmkit-state:
  lmkit-models:

Other containers on the Compose network reach the server at http://lmkit:5189; host clients use the published port. Drop the deploy block on CPU-only hosts.

7Kubernetes: one pod, or a fleet#

Two honest shapes, and the difference between them is exactly what is shared:

One pod keeps everything node-local: state and models on ReadWriteOnce PVCs, panel edits persisted, Recreate strategy so a rollout never puts two writers on the same volumes. This remains the right shape for a single box's worth of load.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lmkit-one
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels: { app: lmkit-one }
  template:
    metadata:
      labels: { app: lmkit-one }
    spec:
      containers:
        - name: lmkit
          image: registry.example.com/lmkit-one:<version>  # the image you built
          ports:
            - containerPort: 5189
          resources:
            limits:
              nvidia.com/gpu: 1
          livenessProbe:
            httpGet: { path: /health, port: 5189 }
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /lmkit/v1/ready
              port: 5189
              httpHeaders:
                - name: Authorization
                  value: Bearer <probe-api-key>
            periodSeconds: 15
            failureThreshold: 4
          volumeMounts:
            - { name: state, mountPath: /data/state }
            - { name: models, mountPath: /data/models }
      volumes:
        - name: state
          persistentVolumeClaim: { claimName: lmkit-state }
        - name: models
          persistentVolumeClaim: { claimName: lmkit-models }

The nvidia.com/gpu resource requires the NVIDIA device plugin on the cluster; omit the limit on CPU nodes. Backups in this shape are snapshots of the state PVC, taken with the pod stopped for a clean copy, exactly as the runbook prescribes.

A fleet scales by replication: every pod is a complete engine with its own resident models, and what is shared is exactly what sections 8 and 10 make shareable: the admin-domain database, the upload volume, the model volume, and the manifest that owns configuration. The operator plane (dashboard, logs, request history) stays each pod's own view; the admin footer names the pod serving it. The platform-independent contract behind these manifests, including what honestly stays per node, is Scaling Out.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lmkit-one
spec:
  replicas: 2                     # KEDA drives this from here (section 9)
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxSurge: 1, maxUnavailable: 0 }  # warm joins + drains keep rollouts lossless
  selector:
    matchLabels: { app: lmkit-one }
  template:
    metadata:
      labels: { app: lmkit-one }
      annotations:                # or a ServiceMonitor, where the Prometheus Operator runs
        prometheus.io/scrape: "true"
        prometheus.io/port: "5189"
        prometheus.io/path: "/metrics"
    spec:
      terminationGracePeriodSeconds: 120   # above DrainAdvertiseSeconds + ShutdownDrainSeconds
      containers:
        - name: lmkit
          image: registry.example.com/lmkit-one:<version>
          ports:
            - containerPort: 5189
          env:
            - name: LMKIT_MANAGED_CONFIG          # the manifest owns configuration (section 10)
              value: "1"
            - name: LMKIT_STATE_DIR
              value: /data/state
            - name: LMKIT_MODELS_DIR
              value: /data/models
            - name: FileManagement__UploadDirectory
              value: /data/uploads
            - name: Security__NetworkAccess
              value: Network
            - name: DefaultChatModel
              value: "qwen3.5:4b"
            - name: Inference__WarmupModels__0    # readiness waits for the lineup (section 9)
              value: "default"
            - name: Telemetry__Enabled
              value: "true"
            - name: Telemetry__EnablePrometheus
              value: "true"
            - name: Admin__Identity__Store        # the shared admin domain (section 8)
              value: postgres
            - name: Admin__Identity__ConnectionString
              valueFrom:
                secretKeyRef: { name: lmkit-identity, key: connection-string }
          resources:
            limits:
              nvidia.com/gpu: 1
          livenessProbe:
            httpGet: { path: /health, port: 5189 }
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /lmkit/v1/ready
              port: 5189
              httpHeaders:
                - name: Authorization
                  value: Bearer <probe-api-key>
            periodSeconds: 5      # the drain advertisement is 5s; probe inside it
            failureThreshold: 2
          volumeMounts:
            - { name: state, mountPath: /data/state }
            - { name: models, mountPath: /data/models }
            - { name: uploads, mountPath: /data/uploads }
      volumes:
        - name: state
          emptyDir: {}            # per-pod operator plane; everything durable lives in the stores
        - name: models
          persistentVolumeClaim: { claimName: lmkit-models-shared }
        - name: uploads
          persistentVolumeClaim: { claimName: lmkit-uploads-shared }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lmkit-models-shared
spec:
  accessModes: [ReadWriteMany]    # every pod reads; a pod pulling a model writes once
  resources: { requests: { storage: 200Gi } }
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lmkit-uploads-shared
spec:
  accessModes: [ReadWriteMany]
  resources: { requests: { storage: 50Gi } }
---
apiVersion: v1
kind: Service
metadata:
  name: lmkit-one
spec:
  selector: { app: lmkit-one }
  ports:
    - port: 5189
      targetPort: 5189

Front the Service with your ingress and terminate TLS there; streamed responses need the ingress-side buffering settings from Behind a Reverse Proxy. The KEDA ScaledObject that grows and shrinks replicas is in section 9, and the rest of the fleet's behavior is built in: a new pod reports warming until its lineup is resident, a stopping pod advertises draining while it finishes its streams, two pods asked for the same missing model pull it once (a cross-process lock on the shared model volume; the second pod waits, then loads the file the first one fetched), and every pod refuses configuration edits because the manifest owns them.

8One admin domain across nodes#

Operator identity scales ahead of inference: accounts, sessions, second factors, and sign-in lockout budgets can live in one shared database that every node points at, so an operator signs in once and administers any node, a revocation reaches all of them, and an attacker rotating across nodes meets one failure budget instead of one per pod. Configure it in the file or environment (deliberately not in the admin form: the surface that edits configuration authenticates through this store):

{
  "Admin": {
    "Identity": {
      "Store": "postgres",
      "ConnectionString": "Host=db;Database=lmkit_identity;Username=lmkit;Password=..."
    }
  }
}

Store accepts postgres, mysql, or sqlserver; the default embedded keeps the single-node SQLite file under the state directory. The rules that matter:

• Identity gets a strictly dedicated database; the database server can be shared. Knowledge-base clusters store document content; operator credentials, sessions, and lockout state never share a DATABASE with them, while one PostgreSQL, MySQL, or SQL Server instance may happily host the identity database beside the cluster databases. Both directions are enforced: an identity store pointed at a content database refuses to start, and a knowledge-base cluster pointed at the identity database refuses to initialize, each naming the remedy. Create a small dedicated database (for example lmkit_identity) and grant the server's login rights on it alone.
• Seed the first operator before exposing the nodes. Set Admin__InitialPassword on every node (it becomes the admin account exactly once, on whichever node runs first), or create the account on one node before the others serve traffic: a node that booted against an empty store notices accounts created elsewhere within 30 seconds.
• Name your ingress in Security:TrustedProxies (IP addresses or CIDR networks) so sign-in throttling, session records, and the audit trail see the real client address from X-Forwarded-For instead of the load balancer's. With the list empty, forwarded headers are ignored, which is the safe stance when nothing sits in front.
• Session validations are cached per node for 30 seconds, so a revocation issued on one node is effective everywhere within that window, and immediately for tokens the other node has never seen. Remembered devices, web-area tickets, and in-flight SSO sign-ins are all domain-wide too: the signing secret and the PKCE challenges live in the shared store, so a sign-in that starts on one node completes on another.
• API keys and stored responses are domain-wide as well: a key minted on any node authorizes on every node the moment it exists (an unknown token is looked up in the store before it is refused), revocations land within the same 30-second window, per-key usage counters accumulate across nodes, and a Responses-API conversation chains on whichever node the next request reaches. A node's pre-existing apikeys.json is adopted into the store on its first start and removed.
• Point FileManagement:UploadDirectory at a shared volume so a file uploaded through one node serves requests on every node: storage is filesystem-authoritative (GUID-named files plus manifests, no in-memory index), which is exactly what a shared read-write volume wants. Per-owner concurrency budgets remain per node in this release, so size them accordingly.
• Security events (sign-ins and failures, account lifecycle, credential and second-factor changes) are written to the shared store's central trail beside each node's own journal; the audit history endpoint serves that domain-wide view for the security source.

9Autoscaling with KEDA#

The serving signal, the probes, and the drain behavior are built in; KEDA supplies the scaling loop. Scaling is by replication: every pod is a complete engine with its own resident models, and the fleet grows or shrinks on inference pressure.

The signal. With telemetry and the Prometheus endpoint enabled, /metrics exports the first-party gauges anonymously: lmkit_inference_slots_saturation (busy share of the concurrent-completion budget, 0 to 1: scale out as it approaches 1, because the queue only grows after it gets there), lmkit_inference_queue_depth, and lmkit_inference_active. On a Prometheus-less stack, /lmkit/v1/ready serves queue_depth as JSON for KEDA's metrics-api scaler, authenticated with an API key.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: lmkit-one
spec:
  scaleTargetRef: { name: lmkit-one }
  minReplicaCount: 1            # scale-to-zero is dishonest for minute-scale model loads
  cooldownPeriod: 600           # models are expensive to load; shed capacity slowly
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: avg(lmkit_inference_slots_saturation)
        threshold: "0.8"        # add a pod while the fleet still absorbs the load

The metrics-api variant for stacks without Prometheus:

  triggers:
    - type: metrics-api
      metadata:
        url: http://lmkit-one.default:5189/lmkit/v1/ready
        valueLocation: queue_depth
        targetValue: "4"
      authenticationRef: { name: lmkit-api-key }   # Bearer via a TriggerAuthentication

Warmup gates readiness. A pod KEDA adds is useless until its models are resident, so opt in with Inference:WarmupModels (model ids, or default for the default chat model): /lmkit/v1/ready answers 503 warming until the lineup is loaded, the load balancer holds traffic back, and a model that cannot load parks the pod visibly not-ready (warmup_failed names it) instead of letting it serve refusals. Pair it with the model delivery choice from section 3: image-baked or pre-warmed volumes make warmup fast; pull-on-demand makes the first pod slow exactly once.

Scale-down drains. The moment a pod receives its stop signal, /ready flips to 503 draining and the listener stays open for a short advertise window (default 5 seconds inside Kubernetes, detected via KUBERNETES_SERVICE_HOST; top-level DrainAdvertiseSeconds overrides it, 0 disables): endpoint deregistration races the stop signal, and the window serves the requests a load balancer routes before it observes the failed probe, so no preStop sleep hook is needed. The listener then closes while in-flight requests, streamed completions included, finish within ShutdownDrainSeconds (default 100); liveness stays healthy throughout, so nothing kills the pod mid-drain. Set terminationGracePeriodSeconds above the sum, for example 120.

10Managed configuration#

A fleet's configuration belongs to the manifest, not to any one pod: a panel edit that lands in a single pod's settings file silently diverges that pod from its siblings and evaporates on the next reschedule. Set LMKIT_MANAGED_CONFIG=1 (or top-level ManagedConfiguration: true in a mounted settings file) and the deployment becomes the configuration's owner:

• Every configuration write refuses, with the reason: the admin config form, the setup wizard's apply, config import, the search cluster editors, the TLS/ACME editor, activation provisioning, and the lmkit cert verb all answer that configuration is managed by the deployment. Nothing changes live state first, so a node never drifts from its manifest.
• The admin panel renders settings read-only: every value shown is the node's effective configuration (useful for inspecting what a pod actually resolved), with a notice naming the mode. Export still works and is the quickest way to turn a hand-tuned single node into a fleet manifest. State remains fully editable: models, knowledge-base content, operators, API keys, and uploads are data, not configuration, and the shared stores from section 8 keep them domain-wide.
• Nothing is seeded or migrated on disk: the process never writes appsettings.json, so a read-only ConfigMap mount and a read-only root filesystem both work.

Configuration then comes from environment variables (HttpPort, Inference__SlotContextSize, Admin__Identity__Store, ... using the __ separator) or a settings file mounted from a ConfigMap; the environment wins where both state a value. The admin footer names the node serving each page (Node), because behind a load balancer the dashboard, logs, and request history are that node's own view.

env:
  - name: LMKIT_MANAGED_CONFIG
    value: "1"
  - name: LMKIT_STATE_DIR
    value: /state
  - name: Inference__WarmupModels__0
    value: "default"

11Stated plainly#

• No official image exists; you own the Dockerfile, the base, and the rebuild cadence. The self-contained archive keeps that surface to one extraction on one LTS base.
• Inside a container the posture must be Network, so every API call needs a key; /health stays anonymous for probes, /ready does not.
• Pin LMKIT_STATE_DIR and LMKIT_MODELS_DIR onto volumes; the state volume is the installation, the model volume is a bandwidth saver, and baking models instead is the air-gapped play.
• CUDA passthrough via the NVIDIA Container Toolkit works with nothing installed in the image; Vulkan in containers is manual and brittle.
• Operator identity scales across installations through a dedicated shared database (Admin:Identity); document content and identity never share one, and the server refuses a configuration that mixes them.
• Scaling is by replication: each pod is a complete engine with its own resident models. Sharing the admin-domain database and the upload volume makes any node serve any API request (section 8); KEDA grows the fleet on the saturation gauge (section 9); the operator plane (dashboard, logs, request history) stays each node's own view, and the admin footer names the node.
• A fleet runs LMKIT_MANAGED_CONFIG=1: the manifest owns configuration, every panel or API write of it refuses, and the panel shows each node's effective settings read-only (section 10). A single pod with a persistent state volume may keep panel edits instead.