Deployment

Backup and Upgrades

Everything this server persists lives under known paths with stated lifetimes; the inventory is the map. This page is the runbook built on it: the exact copy set for a backup, the order of operations for an upgrade or a rollback, and the move-to-a-new-host procedure, each with its verification step, because a backup nobody has restored is a hope, not a backup.

1What a full backup is#

The locations, in descending order of preciousness:

Copy	Contains	Notes
The state directory	Settings (`appsettings.json`) and the settings keyring; the embedded identity store (`admin-identity.db`: operators, sessions, second factors, API keys and stored responses, when no shared database is configured); certificates; uploads by default (`upload/`); memories (`memories/`); skills (`skills/`); the custom-model registry (`models-registry.json` and `models-registry.id`); setup state; embedded search databases; the request-trail, metrics and telemetry stores; forensics; logs; managed tools	The startup log names the directory and which rule chose it; `LMKIT_STATE_DIR` pins it. This copy IS the server's identity. Copy the WHOLE directory: secret settings are encrypted against its keyring, so `appsettings.json` alone restores with those values unreadable (re-enter them in the panel if that ever happens).
The admin identity database	Only when `Admin:Identity` points at a PostgreSQL, MySQL or SQL Server database: operators, sessions, second factors, lockout budgets, SSO challenges, the security trail, API keys with their counters, stored responses, the token signing secret, the node roster, the shared domain documents and the leases	Back it up with your database tooling. On restore, bring the database back BEFORE starting the nodes: a node refuses to start against a database it cannot reach, and one started against an empty database creates a fresh schema with no accounts (and applies any `Admin__InitialPassword` seed to it), which is a new domain, not yours. A node that is already running rides an outage on its 30-second session and key caches: the admin surface stays closed, sign-in, operator management and unknown keys refuse until the database answers, keys already in the snapshot keep authorizing.
The shared upload and model volumes	On a fleet: `FileManagement:UploadDirectory` and `LMKIT_MODELS_DIR` on shared mounts, the model volume holding catalog files, `blobs/` with its `refs` markers and `training/` (or the shared work directory `ModelWorkDirectory` names for those two)	Storage snapshots. Models re-pull from the catalog, but imported custom-model blobs and training artifacts are unique to you; the `blobs/refs` markers must travel with the blobs they mark.
The model directory (single node)	Downloaded model files (`LMKIT_MODELS_DIR` overrides the default), plus the same `blobs/` and `training/`, or the work directory `ModelWorkDirectory` names for them	Large and mostly replaceable: catalog models re-pull. Back it up to save bandwidth and air-gapped effort, and because custom-model blobs and training output do not re-pull from anywhere.
External search databases	Only when Search runs on a database you operate	Back up with YOUR database tooling; the server's copy set does not include what it does not host.

Deliberately NOT in the copy set: job records (memory-only, expire in an hour), the inference hibernation cache (meaningless to any other process), and anything under temp paths.

On a load-balanced fleet the copy set spreads: the shared admin database and any external search database ride your database backup practice, the shared upload and model volumes ride your storage snapshots, and each node's state directory is still worth its own copy. A node's state directory is NOT a disposable part of the node: replacing a node without its directory loses its memory facts (memories/), its panel-authored skills (skills/), its registry identity its blob markers carry (models-registry.id), its setup state and its operator history, and it loses the keyring, so a copied appsettings.json on a fresh node reads every encrypted setting as empty. What the identity database and the shared volumes hold survives the node; what the directory holds does not (Scaling Out lists the two sets).

Consistency: settings and sidecars are small files written atomically enough for practical purposes, but the embedded identity store and the embedded search databases are live SQLite files; the clean copy is taken with the server stopped, and a scheduled cold backup window is the honest default. If you must copy hot, treat search indexes as rebuildable (re-index from sources) rather than trusting a mid-write snapshot.

2Upgrading#

The product versions independently of the engine it embeds, both visible at /health and in the admin footer. The procedure:

Back up the state directory (section 1). Thirty seconds of copying buys a real rollback.
Stop the server: the service via the service manager, a desktop install by quitting the tray, a portable run by stopping the process. In-flight jobs do not survive a stop (the contract); drain or schedule accordingly.
Install the new version. The Windows installer keeps the install MODE (a machine running the service stays a service); portable deployments replace the directory, keeping the state directory intact, which is automatic when state lives outside it and is exactly why LMKIT_STATE_DIR is worth pinning on portable installs.
Start and verify: /health reports the new version; lmkit doctor passes; the startup log names the same state directory as before (a changed path here means the writability rule resolved differently: pin it and restart before anything else).
Run one real request per critical surface (a chat call, a search query if you use Search) before declaring the window closed.

Settings carry forward: configuration is read by key, unknown keys are ignored, and legacy keys keep working where names evolved, so an upgrade never demands a settings rewrite.

3Rolling back#

Rolling the BINARY back is the reverse of section 2. The state directory is the nuance:

Settings and keys roll back harmlessly: older versions ignore keys they do not know.
Embedded search databases migrate FORWARD on startup (schema migrations run once per version). An older binary pointed at a newer database is the one genuinely unsupported combination: restore the state-directory backup taken in step 1 alongside the binary rollback, or keep the newer state and re-run the newer version instead.
The identity store migrates forward too. The embedded store adds the columns a newer version needs when it opens; a shared database is created and grown by the first node of the new version that opens it (the creation is idempotent, so every other node finds it done). Treat a shared database as upgraded the moment any node of the new version has started against it, and do not roll a node back below that version while others run the new one: roll the fleet as one version, and restore the database backup with the binaries when the whole fleet goes back.

That is the whole reason step 1 is not optional: the pre-upgrade copy is what makes rollback a decision instead of a salvage operation.

4Moving to a new host#

Provision the new machine (same OS family keeps certificate and service stories simple) and install the same server version.
Stop the old server; copy the state directory and the model directory to the new host.
Pin LMKIT_STATE_DIR (and LMKIT_MODELS_DIR if you use it) at the new paths; start.
Re-point the world: DNS or client base URLs, and re-distribute trust if the certificate changed hosts; a hostname change means a new certificate with the new name (Going Live).
Verify as in section 2, then decommission the old host with the disposal story: remove its state directory once the new host is proven.

The embedded identity store moves with the directory, accounts, sessions and API keys included, so clients keep working without re-issuing. On a shared identity store only the state directory's per-node items move (settings, keyring, certificates, memories, skills, the custom-model registry, setup state and operator history); the accounts and keys stay in the database, which the new host reaches through the same Admin:Identity settings. Uploads mid-retention move too, though by default they expire in minutes and rarely matter to a planned move.

5What to rehearse, quarterly#

A restore, not just a backup: fresh machine, restore state and models, start, run one request. Time it; that number is your real recovery objective.
A rollback: previous binary plus the matching state copy, once, so the first real one is not a first.
The doctor: lmkit doctor on the restored host catches the classics (paths, certificates, service registration) in one pass (The Command Line).

6Stated plainly#

A backup is the state directory, plus the model directory, plus the admin identity database and your own search database when either runs external; cold copies are the honest default.
On a fleet the database and the shared volumes hold what survives a node, and each node's directory still holds memories, skills, the custom-model registry, setup state, the keyring and the operator history that do not.
Upgrades are stop, install, start, verify, with the pre-upgrade state copy as the rollback ticket; search databases and the identity store migrate forward and are the reasons rollbacks want that copy.
Moves are the same copy plus re-pointed names and re-distributed trust; keys travel with the identity store, clients notice nothing.
Rehearse the restore; the backup that has never been restored is not yet a backup.