GPUs and Backends

The server picks one compute backend at startup: CUDA for NVIDIA, Vulkan for AMD and Intel, Metal on Apple silicon, the CPU otherwise. This page covers what each platform needs installed, how to confirm a model actually sits on the GPU, and what runs well when there is no GPU at all. Read it in the first hour on real hardware, or whenever generation is slower than the card suggests.

1How the backend is chosen#

At startup the runtime probes the backends present in the installation in a fixed order and settles on the first one whose host requirements are met:

Three rules govern the choice:

• One backend per process, decided at startup. Every setting in the Hardware section applies on the next restart; the section says so.
• There is no pin-by-name setting. Steering is done with allow switches: the CUDA Acceleration toggle in the Hardware section (with it off, an NVIDIA machine serves through Vulkan instead), and the EnableVulkan key in the settings file's Hardware section, the toggle's file-only sibling. Both are non-operative on macOS, where Metal is the backend.
• A CUDA backend that cannot be used is skipped with a warning in Logs. A host whose Vulkan probe finds no usable device is skipped without a default-visible log line, and CPU candidates are walked quietly. The Hardware section's active-backend line is therefore the one check that always answers.

2What each platform needs#

• NVIDIA on Windows. A current NVIDIA driver, nothing else: the CUDA runtime libraries ship beside the backend and resolve from there.
• NVIDIA on Linux, headless included. The driver supplies only libcuda.so.1. The CUDA backend also links the runtime libraries (libcudart, libcublas, libcublasLt), which come from the CUDA toolkit packages or from copies deployed beside the backend. Check with ldconfig -p | grep libcublas: if it prints nothing, install the CUDA runtime libraries and restart the server. No display server is required; a headless box with the driver and those libraries serves CUDA.
• AMD and Intel GPUs. Served through Vulkan. The Vulkan runtime ships with the GPU driver: the current vendor driver on Windows, Mesa or the vendor stack on Linux. The runtime probes for a working Vulkan installation and skips the backend when the probe finds no device.
• Apple silicon. Metal is part of macOS: nothing to install. Weights and inference state live in unified memory, so the machine's memory is the budget (Hardware Sizing).

Driver guidance that stays true: keep the current driver from your GPU vendor rather than chasing a specific version number. The one version-shaped failure, a driver too old for the CUDA generation, has its own signature in section 5.

3Verify where inference landed#

Do not assume; the check takes a minute:

1. The startup log prints one line: Backend initialized using Cuda13 host. (or Cuda12, Vulkan, Metal, Avx2, and so on). It is the single authoritative answer.
2. The Hardware section shows the resolved backend above the device list: Cuda12, Cuda13, Vulkan, and Metal run inference on the GPU; Avx2, Avx, and CPU run it on the processor. Each detected device is listed with its index and memory.
3. lmkit devices prints the same device table from the command line: number, name, description, type, and total memory.
4. The dashboard's GPU activity cards (Dashboard) track per-device memory and compute over time. Loading a model should move the memory figure by roughly the model's file size; generation should move the compute figure where the driver reports one. Where it does not, judge by the memory figure and by generation speed itself.

4Partial offload: when the model does not fit whole#

Two different shortfalls get conflated; the server handles both without refusing the load outright, and differently:

• The weights do not fit the device. Before loading, the server reclaims idle state (cached contexts, idle pool memory) on the target device, and on multi-GPU machines an auto-placed model may be moved to a roomier card. If the weights still do not fit whole, the load falls back to a partial offload: some layers stay on the GPU, the rest compute on the CPU. The load succeeds and every request works, but generation is markedly slower, and the log states the fallback plainly. A reduction taken under transient memory pressure lasts only until the model is reloaded: once the device has room again the server logs an advisory saying so, and unloading and reloading the model restores the full placement. For a model genuinely too big for the card, the durable fixes are a smaller quantization of the same model, a smaller model, or more memory (Models, The Right Model).
• The inference pool does not fit. The weights fit but slots times window does not: the admission gate fits the pool down (slots first, then window) rather than refusing. Sizing that shape deliberately is Inference Capacity's subject.

A partial offload is a degraded success, not an error. Whenever a machine serves slower than its card suggests, look for its log line before touching anything else.

5The silent CPU fallback#

The symptom: everything works, nothing errors, and generation is an order of magnitude slower than the hardware should deliver while the GPU sits idle. This is the classic first-hour evaluation killer, and it is diagnosable in a minute: the Hardware section shows a RUNNING ON CPU callout when CUDA is enabled but the runtime settled on a processor backend. The causes:

• Missing CUDA runtime libraries (Linux). The log names the exact library that could not be resolved and what satisfies it. Install the CUDA runtime libraries, restart.
• Driver too old for the CUDA generation. The backend loads but reports no usable devices; the log says exactly that and points at the driver. Update it.
• No AVX2 on an x64 host. The GPU bundles ship an AVX2-built CPU companion, so on an x64 machine without AVX2 they are skipped by design and the log states it. That machine serves CPU-only regardless of settings.
• No working Vulkan. The probe finds no device, typically a driver installed without its Vulkan component, and the skip leaves no log line at the default capture level: the Hardware section's active backend is the diagnostic here. Reinstall the current GPU driver; vulkaninfo, where available, confirms what the host exposes.

After any fix: restart, then reread the startup line from section 3. Broader first-run diagnosis lives in Troubleshooting First Runs.

6CPU-only, honestly#

A machine with no usable GPU is a real deployment target for the right workloads. On a CPU backend the weights and inference state live in system RAM, so the machine's memory is the budget (Hardware Sizing). Inference threads default to the physical core count, and Linux ARM64 is a supported platform.

• Works well: compact chat models (qwen3.5:0.8b, qwen3.5:2b, qwen3.5:4b, gemma4:e2b, gemma4:e4b; the catalog in Models is the living list), embeddings and reranking (embeddinggemma-300m, qwen3-embedding:0.6b, harrier-oss:0.6b, bge-m3, bge-m3-reranker), transcription (the whisper family, whisper-tiny through whisper-large-turbo3), the OCR models (glm-ocr, paddleocr-vl-1.6:0.9b), and the document pipeline, much of which (the PDF toolbox) needs no model at all. Batch text analysis rides the same rule: work that runs in the background tolerates CPU latency.
• Where it stops: interactive chat on large models, where first-token latency and generation rate stop being acceptable. Judge with your own prompts on your own box, not folklore: Measuring What Matters.

A laptop, mini-PC, or ARM64 box serving embeddings, transcription, and document processing is a legitimate installation, not a compromise.

7Several GPUs#

• Device selection. GPU Selection in the Hardware section pins a model host device; the default auto-selects the best fit. Auto placement never trades a discrete card for an integrated GPU, whose reported memory is shared system RAM that its driver cannot honor at model scale.
• Distributed inference. The toggle splits tensor computation across all GPUs, which is how models larger than any single card's memory become servable. It requires at least two devices (the control disables itself otherwise), and the interconnect, not the silicon, sets the ceiling (Hardware Sizing).

8Stated plainly#

• One backend per process, resolved at startup in a fixed order (CUDA, then Vulkan, then CPU; Metal on macOS); every change here applies after a restart.
• NVIDIA on Windows needs the driver alone; on Linux it also needs the CUDA runtime libraries; AMD and Intel ride the driver's Vulkan; Apple silicon needs nothing installed.
• Verify placement, never assume it: the startup log line, the Hardware section's active backend, lmkit devices.
• A partial offload is a logged, degraded success; the silent CPU fallback is diagnosable from the Hardware callout, with warning log lines for the CUDA causes.
• CPU-only serves compact chat, embeddings, transcription, and document work well; interactive chat on large models is where it stops.