Memory, Speed, and Hardware
Training runs on the same hardware that serves inference, GPU-accelerated where a GPU exists. Because LoRA freezes the base model, the memory story is different from full fine-tuning, and better: this guide covers what actually consumes memory, the ladder to walk when a run does not fit, and the levers that make runs fast.
1Where the memory goes#
Four consumers, in the order they usually matter:
| Consumer | Scales with | Notes |
|---|---|---|
| Frozen base weights | Model size and quantization, exactly as loaded for inference | The floor; training adds nothing to it |
| Activations and gradients | The micro-batch (tokens per backward slice), linearly | The BIG adjustable; this is what the micro-batch lever controls |
| Adapter + optimizer state | rank x target modules x adapted layers | AdamW keeps two moments per trained weight; still small next to the base |
| Compute buffer | Graph shape | Reported per run as peak compute bytes in the training report |
The takeaway: the base model's footprint is fixed by your model choice, and nearly all the ADJUSTABLE memory is activations, which the micro-batch controls linearly.
2The out-of-memory ladder#
When a run fails to allocate, walk down this ladder and stop at the first rung that fits; each step trades less than the one below it:
- Lower
micro_batch(to 256, then 128). Linear memory relief, zero effect on what is learned, only on speed. This solves most cases, and a run that leavesmicro_batchunset takes this rung on its own: it halves the micro-batch until the graph fits before refusing. - Restrict the layer range to the last third of the blocks. Adapter memory and backward compute shrink proportionally, and most task behavior lives there anyway.
- Narrow the target modules (attention_and_feedforward back to attention).
- Lower the rank (16 to 8). Only matters if the task genuinely needed the capacity.
- Lower the window (
cutoff_length), accepting that longer samples are skipped; check the dataset digest's skip count before accepting this trade (see Preparing a Training Dataset).
On a machine with several GPUs there is a rung that trades nothing in quality: spread the run over them (section 4).
A merged-model output additionally needs disk for the merged GGUF, not device memory; adapter output is always small.
3What makes runs fast#
- GPU offload. Training a fully offloaded model is orders of magnitude faster than CPU training; the run's report names the device it actually used, and the workbench shows GPU, VRAM, and RAM live. CPU-only training works and is fine for small experiments, just budget minutes-to-hours instead of seconds-to-minutes per epoch.
- Sequence packing. A dataset of short samples wastes most of every window unpacked; packing fills windows with isolated samples and multiplies throughput accordingly (text-only; see Choosing Training Parameters).
- A bigger micro-batch. The same lever from the ladder, in reverse: when memory is not tight, larger backward slices run faster.
- A smaller base. The most honest speed lever of all: prove the behavior on a small model first, then decide whether the task truly needs a bigger one.
Runs execute one at a time, and training competes with inference for the same device memory: on a box that also serves traffic, schedule heavy runs off-peak, or train on a second machine and ship the artifact (it is one file).
4Training on several GPUs#
The devices field of a run (the GPUs picker in the workbench, shown on machines
with two GPUs or more) decides where it trains:
| Value | Placement |
|---|---|
auto (default) |
Where the model already serves, or where the server would load it |
1 |
GPU 1 alone, leaving the others free to serve |
0,1 |
Exactly those GPUs, the layers spread between them |
all |
Every GPU, the layers spread between them |
A spread run splits the model by layers: each GPU holds its share of the base weights, of the adapter and of the activations, and computes the layers it holds. A copy loaded for the run divides the layers by each GPU's free memory, so the GPU with more room takes more. Admission prices every GPU for its own share and refuses the run when any one of them cannot hold it, naming that GPU; the plan in the workbench shows the same figures per GPU before anything downloads.
What it buys and what it costs:
- Capacity, not speed. A spread fits a base and its training state that no single GPU holds. A step visits the GPUs one after another, so a run that fits one GPU trains at least as fast there.
- Isolation. Pinning a run to a GPU the serving traffic does not use keeps inference latency untouched while it trains.
- A private copy when needed. A run whose placement matches where the model already serves trains on that copy. Any other placement loads a private copy of the base laid over the chosen GPUs, released when the run ends, so the model clients are served from never moves.
- One GPU for the vision projector. Training the projector (
train_vision_projector) needs the vision weights and the decoder on the same GPU, so it takes a single-GPU placement.
The run's event log names its placement, the detail panel shows every GPU's load and memory
live, and devices on the job and Devices on the SDK's TrainingReport record where it ran.
5Training over quantized bases#
Adapters train in float over the FROZEN quantized weights, which is why fine-tuning a quantization-compressed model fits on consumer hardware at all; the approach is a close cousin of QLoRA. Two consequences:
- An adapter belongs to the exact base it trained on. Apply it to the same quantized model you trained over; deployment details in Deploying and Evaluating.
- For maximum-fidelity merges, the
full_precisionoption trains from the model's published F16/BF16 variant instead (downloaded on first use, when the model's repository publishes one). It costs the full-precision footprint; adapter-over-quantized is the right default and full precision the deliberate exception.
6Stated plainly#
- Almost all adjustable training memory is activations: lower the micro-batch first, always.
- Several GPUs add room, not speed: spread a run that does not fit one GPU, pin one that must stay away from serving traffic.
- The report's device line and peak compute bytes are the measured truth; capacity-plan from them, not from guesses.
- Train small and prove the behavior before paying for a bigger base; the artifact is one portable file either way.