Choosing Training Parameters
Every parameter in the workbench maps to one lever with a physical meaning. The useful mental model has three axes: capacity (how much the adapter can absorb: rank, target modules, layer range), exposure (how hard the data pushes: epochs, learning rate, schedule), and stability (what keeps the push controlled: warmup, clipping, decay, batch shape). This guide explains each lever and ends with starting recipes.
1Capacity: rank, alpha, and target modules#
Rank is the adapter's inner dimension: its learning capacity and its size. Style, tone, and format adaptation live comfortably at rank 4 to 8 (the default is 8); absorbing new associations and domain knowledge wants 16 to 32. Raising rank past what the task needs mostly buys slower training and a bigger artifact, not quality.
Alpha scales the adapter's effect: the applied scale is alpha / rank, and the common rule
is alpha = 2 x rank (the defaults, 16 over 8, follow it). Treat rank as the knob and keep the
ratio; tuning both independently multiplies the search space for little gain. At rank 32 and
above, enable rank-stabilized LoRA (use_rslora), which scales by alpha / sqrt(rank) so
high ranks keep a usable effective scale; the scaling folds into the saved artifact, so nothing
downstream changes. Original papers: LoRA and
rsLoRA.
Target modules pick which weight matrices receive adapters:
| Choice | What it adapts | When |
|---|---|---|
attention (default) |
The four attention projections | Style, tone, format, instruction following |
attention_and_feedforward |
Attention plus the dense MLP | Tasks that must absorb new content, not just shape |
all |
Everything, including MoE expert projections | Maximum capacity; expert flags match nothing on dense models |
Layer range (first_layer / last_layer) restricts adapters to a block range. Adapter
memory and backward compute shrink proportionally, and the last third of the blocks carries most
task-specific behavior, so a last-third range is the honest middle ground when memory is tight
(see Memory, Speed, and Hardware).
2Exposure: epochs, learning rate, schedule#
The optimizer is AdamW, starting at learning_rate
(default 1e-4). Small datasets tolerate more: 2e-4 over 3 epochs is a solid small-run baseline.
The tells are directional: a run whose loss barely moves is under-exposed (raise the rate or the
epochs); a run whose loss jumps around or spikes upward is over-exposed (lower the rate, add
warmup).
Schedules shape the rate over the run:
| Schedule | Curve | When |
|---|---|---|
cosine (default) |
Smooth decay toward min_learning_rate |
The safe default, matches the SGDR family |
constant |
Flat | Short sanity runs, overfit checks |
linear |
Straight-line decay | Interchangeable with cosine in practice |
cosine_with_restarts |
Cosine that restarts each epoch | Multi-epoch runs on small sets, each epoch re-explores |
polynomial |
Power-2 decay | Front-loads learning, quietest finish |
Warmup (warmup_ratio) spends the first fraction of steps climbing from zero to the peak
rate. It costs nothing and prevents the first steps from lurching while the optimizer's moments
are still empty; 0.05 to 0.1 is plenty for runs beyond a quick experiment.
Epochs multiply exposure. Three (the default) suits datasets of hundreds of samples; tiny curated sets often want 5 to 8 with a validation split watching for the overfitting turn (covered in Reading a Training Run).
3Stability: batch shape and the regularizers#
Gradient accumulation combines the gradients of N samples into one optimizer step: a larger effective batch, smoother steps, no extra memory. Raise it (4 to 8) when the loss curve is noisy because samples differ a lot from each other.
Sequence packing fills each training window with several short samples instead of one padded sample. Packed samples are fully isolated (each gets its own attention sequence, positions restart at zero, loss never crosses a boundary), so it changes throughput, not semantics: a dataset of short samples trains several times faster packed. Image samples never pack, and some architectures always train unpacked.
Micro-batch (micro_batch) is how many tokens are evaluated per backward slice. It is a
speed and memory lever, not a quality lever: the first thing to lower when a run does not fit
device memory, the first to raise when it does.
The regularizers:
- Weight decay (default 0.01): standard AdamW shrinkage, rarely worth touching.
- Gradient clipping (
max_grad_norm, default 1.0): rescales only outlier steps; ordinary steps pass through untouched. Keep it on. - NEFTune (
neftune_alpha, typical 5 to 15): adds scaled noise to the input embeddings during training (validation stays clean). Its measured niche is small instruction datasets, where it counters memorization; try it when a small set overfits before it generalizes. - LoRA+ (
lora_plus_ratio, common value 16): trains the adapter's zero-initialized half at a multiple of the base rate, speeding convergence at unchanged memory. A free trial when runs converge slowly.
Seed: set one to make adapter initialization reproducible; 0 draws randomly. Reruns for comparison should pin it.
4Starting recipes#
Three configurations that converge cleanly, as starting points rather than gospel:
| Task shape | Recipe |
|---|---|
| Identity, style, or fixed format (map many phrasings to one consistent behavior) | rank 8, alpha 16, attention, 3 epochs, lr 2e-4, cosine |
| Domain knowledge plus format (hundreds of samples with short factual answers) | rank 16, alpha 32, attention_and_feedforward, 8 epochs, lr 1e-4, warmup 0.1, cosine, packing on, validation split 0.1 |
| Vision labeling (teach a vision model your visual domain) | rank 8, alpha 16, attention, 3 epochs, lr 2e-4; see Fine-tuning on Images |
Change one lever at a time from a recipe, and let the curves arbitrate: the loop of adjusting against evidence is Reading a Training Run.
5Stated plainly#
- Rank and modules set what CAN be learned; epochs and learning rate set how hard the data pushes; everything else keeps the push stable.
- The defaults are a real baseline, not a placeholder: change them in response to a curve, not preemptively.
- Keep alpha at 2 x rank, switch to rsLoRA at rank 32+, and pin the seed for any run you intend to compare.