LM-Kit OneDocs2026.8.10lm-kit.com
Deployment

Backup and Upgrades

Everything this server persists lives under known paths with stated lifetimes; the inventory is the map. This page is the runbook built on it: the exact copy set for a backup, the order of operations for an upgrade or a rollback, and the move-to-a-new-host procedure, each with its verification step, because a backup nobody has restored is a hope, not a backup.

1What a full backup is#

Three locations, in descending order of preciousness:

Copy Contains Notes
The state directory Settings (appsettings.json), the settings keyring, API keys (hashed sidecar), certificates, uploads, memories, skills, embedded search databases, the request-trail and telemetry stores, logs The startup log names the directory and which rule chose it; LMKIT_STATE_DIR pins it. This copy IS the server's identity. Copy the WHOLE directory: secret settings are encrypted against its keyring, so appsettings.json alone restores with those values unreadable (re-enter them in the panel if that ever happens).
The model directory Downloaded model files (LMKIT_MODELS_DIR overrides the default) Large but replaceable: models re-pull from the catalog. Back it up to save bandwidth and air-gapped effort, not because it is unique.
External search databases Only when Search runs on a database you operate Back up with YOUR database tooling; the server's copy set does not include what it does not host.

Deliberately NOT in the copy set: job records (memory-only, expire in an hour), inference hibernation files (meaningless to any other process), and anything under temp paths.

On a load-balanced fleet the copy set moves with the data: the shared admin database (operators, API keys, stored responses) and any external search database ride your database backup practice, the shared upload and model volumes ride your storage snapshots, and each node's state directory holds only that node's operator history, replaceable with the node itself (Scaling Out).

Consistency: settings and sidecars are small files written atomically enough for practical purposes, but the embedded search databases are live SQLite files; the clean copy is taken with the server stopped, and a scheduled cold backup window is the honest default. If you must copy hot, treat search indexes as rebuildable (re-index from sources) rather than trusting a mid-write snapshot.

2Upgrading#

The product versions independently of the engine it embeds, both visible at /health and in the admin footer. The procedure:

  1. Back up the state directory (section 1). Thirty seconds of copying buys a real rollback.
  2. Stop the server: the service via the service manager, a desktop install by quitting the tray, a portable run by stopping the process. In-flight jobs do not survive a stop (the contract); drain or schedule accordingly.
  3. Install the new version. The Windows installer keeps the install MODE (a machine running the service stays a service); portable deployments replace the directory, keeping the state directory intact, which is automatic when state lives outside it and is exactly why LMKIT_STATE_DIR is worth pinning on portable installs.
  4. Start and verify: /health reports the new version; lmkit doctor passes; the startup log names the same state directory as before (a changed path here means the writability rule resolved differently: pin it and restart before anything else).
  5. Run one real request per critical surface (a chat call, a search query if you use Search) before declaring the window closed.

Settings carry forward: configuration is read by key, unknown keys are ignored, and legacy keys keep working where names evolved, so an upgrade never demands a settings rewrite.

3Rolling back#

Rolling the BINARY back is the reverse of section 2. The state directory is the nuance:

  • Settings and keys roll back harmlessly: older versions ignore keys they do not know.
  • Embedded search databases migrate FORWARD on startup (schema migrations run once per version). An older binary pointed at a newer database is the one genuinely unsupported combination: restore the state-directory backup taken in step 1 alongside the binary rollback, or keep the newer state and re-run the newer version instead.

That is the whole reason step 1 is not optional: the pre-upgrade copy is what makes rollback a decision instead of a salvage operation.

4Moving to a new host#

  1. Provision the new machine (same OS family keeps certificate and service stories simple) and install the same server version.
  2. Stop the old server; copy the state directory and the model directory to the new host.
  3. Pin LMKIT_STATE_DIR (and LMKIT_MODELS_DIR if you use it) at the new paths; start.
  4. Re-point the world: DNS or client base URLs, and re-distribute trust if the certificate changed hosts; a hostname change means a new certificate with the new name (Going Live).
  5. Verify as in section 2, then decommission the old host with the disposal story: remove its state directory once the new host is proven.

API keys move with the state directory (they are server-side records; clients keep working without re-issuing). Uploads mid-retention move too, though by default they expire in minutes and rarely matter to a planned move.

5What to rehearse, quarterly#

  • A restore, not just a backup: fresh machine, restore state and models, start, run one request. Time it; that number is your real recovery objective.
  • A rollback: previous binary plus the matching state copy, once, so the first real one is not a first.
  • The doctor: lmkit doctor on the restored host catches the classics (paths, certificates, service registration) in one pass (The Command Line).

6Stated plainly#

  • A backup is the state directory, plus the model directory for convenience, plus your own database when Search runs external; cold copies are the honest default.
  • Upgrades are stop, install, start, verify, with the pre-upgrade state copy as the rollback ticket; search databases migrate forward and are the one reason rollbacks want that copy.
  • Moves are the same copy plus re-pointed names and re-distributed trust; keys travel with the state, clients notice nothing.
  • Rehearse the restore; the backup that has never been restored is not yet a backup.