Choosing and Operating Models

How models get onto the server, how the server decides which one serves a request, and how they behave against real memory limits.

1Getting models#

Two paths, both from the admin panel's Models section:

• Pull from the built-in catalog: curated models with known capabilities, sizes, and context lengths. Downloads run in the background with live progress and survive the page.
• Import your own: point the server at a model file (local path or URL). The import probes the file first and refuses what cannot serve, rather than failing later on a request. The operator can disable custom models entirely, pinning the server to the catalog.

Models are stored under the model directory shown in the same section (overridable with LMKIT_MODELS_DIR), separate from the server's state, so the two can live on different volumes and be backed up on different schedules. Disk usage per model is listed beside it.

Clients speaking the Ollama dialect can also pull catalog models over /api/pull (see API Compatibility). A model can also be MADE rather than pulled: fine-tuning produces artifacts that import like any custom model. Which model to pick for a given machine and task, quantization included, is its own guide: The Right Model for Your Machine.

2The default-model slots#

Requests that do not name a model are served by the default for their capability, configured in the Models section. Each slot is one decision:

A named model in the request always wins over the slot. Search tenants carry their OWN embedding and reranking choices (see Indexing Well), so changing a global slot never silently re-embeds a tenant.

3Loading, unloading, and memory#

• Models load on first use and stay resident to serve the next request without the load cost; the Models section shows what is resident and lets you load, unload, or hibernate explicitly.
• Embedding and reranking run as their own roles: loading an embedder never evicts the chat model, and vice versa.
• Size against the device before pulling: a model's file size approximates its memory footprint at rest; context length adds working memory per active request on top. The Hardware section shows what the devices offer, and sizing for concurrent load is its own discipline: Inference Capacity.
• When device memory runs short, the server degrades rather than dies: it reduces what it keeps resident and falls back instead of taking the process down with an out-of-memory failure.

4Stated plainly#

• Nothing serves until it is on disk and probed; imports that cannot serve are refused up front.
• Defaults are per capability, overridable per request, and never override a Search tenant's own model choices.
• Residency is observable and controllable from the Models section; nothing loads or unloads behind the operator's back except first-use loading itself.