Reading a Training Run
The workbench charts training loss, validation loss, next-token accuracy, learning rate, step rate, and an ETA while a run executes, with live device telemetry beside them. Those curves are the run's diagnosis: this guide is how to read them, and what to do about each pathology.
1The two losses#
Training loss measures how well the model predicts the supervised tokens of the samples it is currently learning from. Validation loss measures the same thing on the held-out split (default 5%) that training never touches, once per epoch. The relationship between the two curves is the entire story:
| Pattern | Meaning | Response |
|---|---|---|
| Both fall, validation slightly above training | Healthy: learning that generalizes | Let it finish |
| Steep early drop, then a long flatten | Normal shape; the flatten is convergence | Stop adding epochs once validation stops improving |
| Training falls, validation rises | Overfitting: memorizing, not generalizing | See section 2 |
| Both flat at a high value | The run is not learning | See section 3 |
| Loss jumps around or spikes upward | Steps too aggressive | Lower the learning rate, add warmup 0.1 |
Accuracy is next-token accuracy over the supervised tokens, in 0 to 1. It is a coarser, more intuitive companion to loss: for exact-format tasks it should climb toward 1.0 on a tiny overfit check, and its ceiling on real runs is a sanity signal, not a target to chase.
2Overfitting, and the two switches built for it#
Overfitting is the natural endpoint of every small-dataset run; the question is only where you stop. Two run options handle it without babysitting:
early_stopping_patience: stop after N validation passes without improvement, keeping everything trained so far. Patience 2 with a validation split is the set-and-forget choice for multi-epoch runs.artifact_from_best: produce the artifact from the weights at the BEST validation loss rather than the last step; the run snapshots the adapter whenever validation improves. This makes even a run that overfits at the end deliver its best self.
Both require a validation split. Beyond them: fewer epochs, more (or more varied) data, or NEFTune noise for small instruction sets (see Choosing Training Parameters).
3Runs that learn nothing#
A loss curve that never moves has a short list of causes, in the order worth checking:
- Masking mismatch. With assistant-only loss on, a non-zero unmasked-sample count means the chat template did not align and samples degraded to full-sequence loss; the dataset digest surfaces it. Fix the dataset roles (see Preparing a Training Dataset).
- Learning rate too low for the dataset size. Double it and rerun the overfit check.
- Capacity too small for the task. A rank-4 attention adapter cannot absorb a knowledge corpus; raise rank and widen target modules.
- The window skipped the dataset. Samples longer than the training window are skipped; check the digest's skip count.
The five-minute triage for all of these: train on four or five samples with generous epochs. A correct setup overfits a tiny set to near-zero loss; a setup that cannot is broken somewhere above, and no amount of data will fix it.
4Checkpoints, resume, and stopping#
- Checkpoints (
checkpoint_steps> 0) save the adapter weights plus optimizer state every N optimizer steps. They appear in the run's checkpoint list in the workbench. - Resume starts a NEW job that continues a finished job from a chosen checkpoint: same hyperparameters and dataset (restored from the job's journaled snapshot), optimizer moments and adapter restored, step counter continuing. Use it to extend a run that stopped too early without retraining from scratch.
- Cancel is cooperative: the run halts after the current batch and the adapter trained so far is saved, so cancelling late in a good run still yields a usable artifact.
- Restarts are survivable. Jobs journal every state change beside their artifact. A run interrupted by a server stop reports as failed after restart, and its partial artifact, when one was written, stays downloadable.
Runs execute one at a time; queued jobs start when the runner frees. Finished and running jobs are visible in Jobs as well as the workbench.
5The report#
Every completed run ends with a measured report: the device it ran on, steps completed, total wall time, mean seconds per optimizer step, first and final loss, final accuracy, validation loss, and the peak compute-buffer bytes. Two uses worth building a habit on:
- Regression triage. A run that suddenly takes 20x longer per step almost always landed on the CPU; the report's device line answers it in one glance.
- Capacity planning. The peak compute bytes number is the ground truth for what a configuration actually needs, which feeds directly into Memory, Speed, and Hardware.
6Stated plainly#
- Validation loss is the number that matters; training loss only says the optimizer is working.
- Set early stopping and artifact-from-best on any run longer than an experiment; they convert overfitting from a failure into a stopping rule.
- When nothing learns, overfit five samples first: it isolates setup problems from data problems in minutes.