Preparing a Training Dataset

The dataset IS the specification of the behavior you are training: the model will learn exactly what the examples demonstrate, including their inconsistencies. This guide covers the formats the engine loads, how samples become training windows, and the preparation habits that separate a clean run from a puzzling one.

1Accepted formats#

Upload one file to the workbench (or reference it by dataset_file_id in the API); the format is detected from the extension and content:

The API also accepts conversations inline in the request body (dataset field) and raw text (raw_text), so small datasets need no upload step at all.

2Two stages, two supervision rules#

• sft (the default): samples are chat conversations, rendered through the model's own chat template. With assistant-only loss (the default), the model is graded ONLY on the assistant turns; system and user tokens are context, not targets. This is the standard setup for instruction tuning, and it is why examples must carry the exact output you want reproduced.
• pretrain: samples are raw text and every token is supervised. Use it to soak a model in domain prose (internal wikis, code, reports) before or instead of instruction tuning. Masking does not apply.

One trap worth knowing: when the engine cannot locate the assistant span inside a rendered conversation (a chat-template mismatch), that sample falls back to full-sequence supervision. The run surfaces this as an unmasked-sample count; a non-zero value with assistant-only loss enabled means some of your samples are training on the QUESTION text too, which weakens instruction tuning. Re-check the dataset's roles when you see it.

3What a good sample looks like#

• Exactly the output you want, every time. If half the examples answer in JSON and half in prose, the model learns to be inconsistent. Decide the format first, then make every sample demonstrate it byte for byte.
• Vary the inputs, not the rule. Cover the phrasings, lengths, and edge cases the model will actually meet; keep the demonstrated behavior identical across them. Teaching one behavior per dataset keeps runs debuggable.
• Include the boundaries. If the model should refuse or say "not in scope" for some inputs, those refusals must be examples too; a dataset with only happy paths trains a model that never declines.
• A few hundred sharp examples beat thousands of noisy ones. Small, curated sets converge faster, overfit more visibly (which makes problems diagnosable), and are cheap to iterate on.
• System prompts count. If deployment will use a system prompt, train with the same one in the samples; if not, leave it out. Mismatched framing between training and serving costs quality for free.

4Windows, skipped samples, and the validation split#

Every sample must fit the training window (cutoff_length; 0 sizes the window to the longest sample). Samples longer than the window are SKIPPED, not truncated, so a low cutoff silently shrinks the dataset: the workbench's dataset digest reports how many samples a given window leaves out. Check it before starting rather than discovering a half-trained behavior after.

The validation split (default 5%) holds samples out of training to measure generalization after each epoch. It is drawn from the same file, so it guards against overfitting but not against a biased dataset. Keep a second, untouched set of examples outside the dataset entirely and judge the final artifact on those; that separation is what Deploying and Evaluating builds on.

5Datasets with images#

Vision datasets attach images to individual messages. Three reference styles resolve:

• Data URIs or raw base64 inline in the JSON: work anywhere, no packaging needed.
• Relative paths ("images": ["scans/inv-001.png"]): resolve beside the dataset file on disk, or inside the same ZIP archive. Ship dataset plus images as one ZIP and the references just work.
• <image> markers in the message text place each image at an exact position; without markers, images lead the message.

Resolution is checked up front: a dataset whose image references ALL fail to resolve refuses to start (it would otherwise train text-only without telling you); partial failures warn and train the affected samples without their images. The rest of the vision story, including what actually trains, is Fine-tuning on Images.

6Version datasets like code#

The artifact is reproducible only if its inputs are: same dataset, same parameters, same seed. Keep datasets in version control, name them with a version, and record which dataset version produced which artifact. Finished jobs journal their full request for exactly this reason, and chat templating differences between models are one more argument for re-validating a dataset when you change the base model.

7Stated plainly#

• The model learns what the examples DO, not what you meant: consistency in outputs is the single highest-leverage property of a dataset.
• Watch two counters before training: samples skipped by the window, and samples that lost their assistant mask. Both quietly change what the run learns.
• Hold out real evaluation examples outside the dataset; the validation split alone is not an evaluation.