Configuration as Code

An infrastructure team keeps configuration in a repository, renders it per environment, and deploys it through the same pipeline as the software. This server is built for that: the manifest owns configuration when you say so, every setting has a stable key, secrets come from the environment, and drift is measured rather than guessed. Scaling Out section 4 defines the three setting classes and who may write each; this page is the workflow built on that definition, from the first export to a rollback.

1What the manifest is, here#

Configuration binds from two sources, in this order of precedence:

1. Environment variables, one per setting, the JSON path written with double underscores: Inference:SlotContextSize is Inference__SlotContextSize, an array element is Search__Clusters__0__ConnectionString. The environment wins where both sources state a value.
2. The settings file, appsettings.json under the state directory (LMKIT_STATE_DIR). When the executable's directory is not writable for the service account, a settings file beside the executable is read as well.

Declare an array in ONE source only. Sections such as Search:Clusters are merged element by element across sources, so a cluster deleted from the file survives as long as an environment key still names its index.

The deployment becomes the owner with LMKIT_MANAGED_CONFIG=1 in the environment, or a top-level "ManagedConfiguration": true in the file; a node in Farm mode (LMKIT_DEPLOYMENT_MODE=Farm) is owned whatever that flag says. Ownership means three things the pipeline can rely on: the process never writes the settings file, so a read-only mount and a read-only root filesystem both work; every Node-class write from the panel or the API answers 409 with { "error": ..., "managed": true } and the panel renders those settings read-only with a notice naming the mode; and secrets are read exactly as written, never sealed on disk by the node, which is why section 3 keeps them out of the file.

What goes where, class by class:

2Author the manifest from a configured node#

The fastest correct manifest is an export, not a hand-written file. Configure one node the way you want production to behave (a workstation or the dev node, unmanaged, through the panel and the setup wizard), then export from the console's General section or directly:

curl -s -H "X-Admin-Password: $ADMIN_SESSION" "$LMKIT_URL/lmkit/v1/admin/config/export" -o appsettings.json

$ADMIN_SESSION is the token POST /lmkit/v1/admin/login returns for an operator account; the console sends the same header. The export is the file in its own shape with two properties that make it a seed manifest: every secret is masked as ****, and the shared objects (agents, connectors, the tools policy, clusters) are included from the store. Before the first commit:

• Add "ManagedConfiguration": true at the top level, or plan LMKIT_MANAGED_CONFIG=1 in every environment.
• Move the NodeLocal paths (ffmpeg, the hibernation directory, the memory directory) into the per-environment overlay, or delete them so the defaults apply.
• Leave the **** values in place. They are the list of secrets the environment must supply, and the file is safe to review and to commit as it stands.
• Never copy a live appsettings.json instead of exporting: an owned file seals its secrets as enc:v1 values under that node's keyring, and those read as empty on any other machine.

Keep the file small where the environment is clearer: a fleet's whole Node class fits in the twenty-odd environment keys Scaling Out section 10 lists, and the file then carries only the structured sections (agents, connectors, tools) that are awkward as Agents__0__Name-style keys.

3Secrets from your vault#

These settings are secrets and never enter the repository or the mounted file: the identity database connection string (Admin__Identity__ConnectionString), every Search cluster connection string (Search__Clusters__<n>__ConnectionString), the certificate password (Https__CertificatePassword), the SSO client secret (Sso__ClientSecret), the web-search key of the tools policy (Tools__WebSearchApiKey), the first operator's password (Admin__InitialPassword) and, on a farm, the domain key (LMKIT_DOMAIN_KEY, at least 32 characters, identical on every node). The emergency reset Admin__Password is re-applied on every start it is present, so it never stays in a manifest beyond one restart (Operator Accounts section 3).

Deliver them as environment variables the service sees and nothing else does:

• systemd: an EnvironmentFile= rendered by your secret manager's agent (Vault Agent, the cloud CLI in a deploy step), mode 0600, owned by root, read by the unit alone. The unit keeps a second EnvironmentFile= for the non-secret keys, so the two files have different owners and different review paths.
• Kubernetes: a Secret synced from the vault by the External Secrets Operator or the Secrets Store CSI driver, referenced with secretKeyRef or envFrom on the container; the non-secret keys ride a ConfigMap the same way. The fleet manifest in Running in Containers section 7 already draws this line.
• Windows service: machine-scope variables set by the deployment tool, since a service running as LocalSystem reads the SYSTEM environment and no user's (Windows Server Deployments).

A rendered secrets file for a two-node farm has this shape, and no other content:

# /etc/lmkit-one/secrets.env  (0600 root:root, rendered by the vault agent, never committed)
LMKIT_DOMAIN_KEY=<32+ characters, identical on every node>
Admin__Identity__ConnectionString=Host=identity-db;Database=lmkit_identity;Username=lmkit;Password=<...>
Search__Clusters__0__ConnectionString=Host=search-db;Database=lmkit_search;Username=lmkit;Password=<...>
Sso__ClientSecret=<...>

An operator's shell does not see these files, and a lmkit verb run from such a shell resolves a different identity store than the service does: run verbs with the unit's environment loaded (The Command Line section 5).

4One base, one overlay per environment#

Everything that must be identical between environments is the base; everything that legitimately differs is the overlay, and the overlay is short:

A base plus an overlay under systemd is two EnvironmentFile= lines; on Kubernetes it is a kustomize overlay or a Helm values file per environment writing the same ConfigMap; on a Windows farm it is the deployment tool's variable set per environment. The base moves from development to staging to production as one commit, applied to each environment's overlay in turn, and nothing is edited by hand in between: on a managed node there is nowhere to edit.

Verify an environment after applying it, in this order:

1. lmkit doctor on a node, with the unit's environment: the state directory, the identity store the configuration selected (Identity store: embedded or shared), models, devices, the certificate, the service.
2. GET /lmkit/v1/admin/configuration with an operator session: the resolved configuration as the node reads it, with managed: true, mode (single_node or farm), domainSource (store or manifest) and classes naming the class of every leaf. The panel shows the same resolved values read-only.
3. On a farm, the Farm page's Consistency view (section 6).

5Applying a change#

Node settings are read at start. A changed environment key or mounted file takes effect on restart; on a farm that is a rolling restart, and the join-warm, leave-draining lifecycle makes it lossless (Farm Runbooks section 6 has the order of operations).

Domain objects depend on who owns the domain:

• The store owns it (the default). The manifest's Agents, Connectors, Tools, Memory:Stores and Search:Clusters sections seed an EMPTY store once, on the first start against it; from then on the store is the authority and a later edit of those keys in the manifest is not re-read. Change the objects through the panel or PUT /lmkit/v1/admin/configuration, which every node accepts for Domain leaves even when it is managed, and every other node applies within Deployment__SyncSeconds (5 by default). Choose this when application teams own agents and connectors and infrastructure owns the rest.
• The manifest owns it (Deployment__DomainSource=Manifest). At every start the first node of a boot wave takes a short lease and writes what its environment and files declare over the store (unchanged documents are skipped; a sealed secret is compared by the value it opens to), every other node applies that within its sync period, the domain editors render read-only with the manifest as the remedy, and a domain write answers the same 409. A change to an agent or a cluster is then a commit and a rolling restart, like any other setting. Choose this when the whole configuration must be reviewable in one place.

Skills:Sources is the one Domain object with a per-node effect: each node installs at start, and again when the sources change, the archives the list names, adding what is absent and leaving what is present untouched (Agent Skills).

6Drift, audit, rollback#

Drift. On a farm every heartbeat carries a fingerprint of the node's Node-class settings, one digest per section, with secrets entering as keyed hashes so a differing password shows without being shown. The Farm page compares every node to the one you browse and names the sections that differ (inference, security, ports); NodeLocal paths never enter the fingerprint, and a path under the state directory is compared relative to it. Lag counts the shared-object revisions a node has not applied yet. Drift is named, never repaired from the page: the remedy is the overlay that declares the differing key (Farm Operations section 3). The Security section's audit asks the fleet five more questions from the same facts, one shared store, no unaccepted drift, no stale row, no embedded search cluster, a declared balancer, each answer naming the section to fix.

Audit. Every configuration save lands in the node's activity log with the operator who made it, and a 409 refused by the managed rule is visible in the request trail like any other answer (Observability). On a managed node the activity log therefore records Domain edits alone, which is the point: the repository's history is the Node history.

Rollback. Redeploy the previous commit's overlay and restart; on a farm, node by node. Domain objects under the manifest roll back with it; under the store they are the last panel edit, so a team that keeps the store as the authority keeps an export beside each release (GET /lmkit/v1/admin/config/export) to know what the objects were. Configuration import (POST /lmkit/v1/admin/config/import, Owner role) is refused on a managed node, deliberately: the manifest is the import.

7Stated plainly#

• Two sources, environment over file, arrays in one source only; LMKIT_MANAGED_CONFIG=1 or Farm mode makes the deployment the owner, and every Node write then answers 409.
• Author the manifest by exporting a configured node: secrets arrive masked, shared objects included; add the managed flag, move per-machine paths to the overlay, commit.
• Secrets travel as environment variables rendered from your vault, never in the file, never in the repository; the domain key is the one secret every node must share.
• One base, one short overlay per environment, promoted as one commit; verify with lmkit doctor, the resolved configuration and, on a farm, the Consistency view.
• Node settings apply on restart; Domain objects apply live from the store, or with the manifest under Deployment__DomainSource=Manifest; drift is fingerprinted per section and named on the Farm page; rollback is the previous overlay.