Fine-tuning on Images

A vision fine-tune teaches a vision-capable model to describe, label, or extract from YOUR kind of images: meter displays, invoice layouts, product photos, forms. This guide covers what actually trains, how the dataset is shaped, and the constraints specific to image runs.

1What actually trains#

The base model must already have vision weights (a vision-capable entry in Models). During training, the vision encoder stays FROZEN: images pass through it exactly as at inference, and the adapters train the language side to interpret its output for your task. Practical consequences:

• The model cannot learn to SEE new things; it learns to say the right words about what the encoder already perceives. If a human cannot resolve the detail in the image at the model's input resolution, training will not conjure it.
• Everything from the text chapters carries over unchanged: the same dataset formats, the same parameters, the same curves.

2Dataset shape#

Each training message can carry images (see datasets with images for the reference styles and the ZIP packaging). A typical sample:

{"messages": [
  {"role": "user",
   "content": "What does this meter read? <image>",
   "images": ["meters/unit-042.png"]},
  {"role": "assistant",
   "content": "00432.7 kWh"}
]}

The <image> marker places the image at that exact spot in the text; without markers, images lead the message. As always, the assistant turn is the supervised target: make it EXACTLY the output you want, in the exact format, every time.

3Hard constraints#

• The Output module is not trainable on image runs. A run whose dataset carries images refuses the all-style target selection that includes the output head; use attention or attention_and_feedforward. The engine rejects the combination up front rather than training something subtly wrong.
• Image samples never pack. Sequence packing applies to text-only samples; expect image runs to spend one window per sample, which makes them slower per sample than text runs.
• Resolution is respected, not improved. Keep source images at their natural capture resolution. Upscaling or redrawing image content at larger canvas sizes does not add information and has been observed to hurt fine-grained tasks like digit reading.

4Practical guidance#

• A narrow behavior moves with surprisingly few images. Forty to sixty well-labeled images can visibly shift a specific behavior (one display type, one document layout). Broad visual competence is not trainable this way; narrow reliability is.
• Balance the labels. If 90% of examples read "OK" and 10% read "FAULT", the model learns that "OK" is usually safe to say. Balance classes, or oversample the rare ones.
• Keep capture conditions honest. Train on images that look like production inputs (same camera angles, lighting, crops). A dataset of clean scans trains a model that stumbles on phone photos.
• Evaluate on held-out images. Accuracy on training images proves memorization, nothing more. Keep a slice of images the run never saw and grade the artifact on exact-match output, per the protocol in Deploying and Evaluating.

5Stated plainly#

• Vision fine-tuning adapts the language side over a frozen eye: it sharpens interpretation, never perception.
• Use attention or attention-plus-feedforward targets, expect unpacked (slower) windows, and never inflate source images.
• Narrow task, consistent labels, held-out image evaluation: that combination is what ships.