Memory, Speed, and Hardware

Training runs on the same hardware that serves inference, GPU-accelerated where a GPU exists. Because LoRA freezes the base model, the memory story is different from full fine-tuning, and better: this guide covers what actually consumes memory, the ladder to walk when a run does not fit, and the levers that make runs fast.

1Where the memory goes#

Four consumers, in the order they usually matter:

The takeaway: the base model's footprint is fixed by your model choice, and nearly all the ADJUSTABLE memory is activations, which the micro-batch controls linearly.

2The out-of-memory ladder#

When a run fails to allocate, walk down this ladder and stop at the first rung that fits; each step trades less than the one below it:

1. Lower micro_batch (to 256, then 128). Linear memory relief, zero effect on what is learned, only on speed. This solves most cases.
2. Restrict the layer range to the last third of the blocks. Adapter memory and backward compute shrink proportionally, and most task behavior lives there anyway.
3. Narrow the target modules (attention_and_feedforward back to attention).
4. Lower the rank (16 to 8). Only matters if the task genuinely needed the capacity.
5. Lower the window (cutoff_length), accepting that longer samples are skipped; check the dataset digest's skip count before accepting this trade (see Preparing a Training Dataset).

A merged-model output additionally needs disk for the merged GGUF, not device memory; adapter output is always small.

3What makes runs fast#

• GPU offload. Training a fully offloaded model is orders of magnitude faster than CPU training; the run's report names the device it actually used, and the workbench shows GPU, VRAM, and RAM live. CPU-only training works and is fine for small experiments, just budget minutes-to-hours instead of seconds-to-minutes per epoch.
• Sequence packing. A dataset of short samples wastes most of every window unpacked; packing fills windows with isolated samples and multiplies throughput accordingly (text-only; see Choosing Training Parameters).
• A bigger micro-batch. The same lever from the ladder, in reverse: when memory is not tight, larger backward slices run faster.
• A smaller base. The most honest speed lever of all: prove the behavior on a small model first, then decide whether the task truly needs a bigger one.

Runs execute one at a time, and training competes with inference for the same device memory: on a box that also serves traffic, schedule heavy runs off-peak, or train on a second machine and ship the artifact (it is one file).

4Training over quantized bases#

Adapters train in float over the FROZEN quantized weights, which is why fine-tuning a quantization-compressed model fits on consumer hardware at all; the approach is a close cousin of QLoRA. Two consequences:

• An adapter belongs to the exact base it trained on. Apply it to the same quantized model you trained over; deployment details in Deploying and Evaluating.
• For maximum-fidelity merges, the full_precision option trains from the model's published F16/BF16 variant instead (downloaded on first use, when the model's repository publishes one). It costs the full-precision footprint; adapter-over-quantized is the right default and full precision the deliberate exception.

5Stated plainly#

• Almost all adjustable training memory is activations: lower the micro-batch first, always.
• The report's device line and peak compute bytes are the measured truth; capacity-plan from them, not from guesses.
• Train small and prove the behavior before paying for a bigger base; the artifact is one portable file either way.