Table of Contents

Fine-Tune Vision Models on Image Data

LMKit.Finetuning.LoraFinetuning trains on conversations that carry images, using the same API as text fine-tuning. Images are encoded through the model's vision tower and laid out in the training window exactly as inference lays them out, so what the adapter learns is what inference runs. The vision tower stays frozen; the adapter trains on the language side.

This turns a general vision-language model into a specialist for YOUR images: instrument displays, product photos, form layouts, defect patterns, document types. Everything runs locally, which matters precisely where such images are sensitive.


Why Fine-Tune a Vision Model

  1. Generic VLMs misread specialized visuals. Segment displays, gauges, dense labels, and domain-specific layouts are underrepresented in pretraining data. A small set of labeled examples teaches the exact reading skill needed, on a model small enough for edge hardware.
  2. Consistent structured answers about images. "Respond with the value only", "classify into these six defect types": vision fine-tuning locks the output contract the same way text fine-tuning does.

Prerequisites

Requirement Minimum
.NET SDK 8.0+
Base model A vision-capable model (HasVision is true), for example qwen3.5:0.8b, qwen3.5:2b, gemma4:e2b
Training data Labeled images: a question and the expected answer per image
Hardware CUDA GPU recommended; image samples cost more per step than text

Two constraints specific to image training:

  • LoraTargetModules.Output is not supported when samples carry images; use Attention (the default) or AttentionAndFeedForward.
  • Image samples are never packed by SequencePacking.

Step 1: Control the Cost of Every Image

Each image consumes vision tokens in the training window, and that cost follows the image detail setting. Bind it once at startup:

using LMKit.Global;
using LMKit.Inference.Vision;

// Every image an inference or training pipeline touches follows this budget.
Configuration.DefaultImageDetail = ImageDetail.Minimal;

Minimal keeps small or synthetic images cheap; leave the default High for photos where fine detail carries the answer. The training window is sized to the data automatically, and image samples get a higher window ceiling than text, but the detail level is the lever that decides how many tokens each image occupies.


Step 2: Build Image Training Data in Code

Each training sample is a conversation whose user turn carries an image attachment and whose assistant turn is the expected answer:

using LMKit.Data;
using LMKit.Finetuning;
using LMKit.Model;
using LMKit.TextGeneration.Chat;

LMKit.Licensing.LicenseManager.SetLicenseKey("");

using LM model = LM.LoadFromModelID("qwen3.5:0.8b");

using var finetuning = new LoraFinetuning(model);
finetuning.Parameters.Rank = 8;
finetuning.Parameters.Alpha = 16;
finetuning.Parameters.TargetModules = LoraTargetModules.Attention;
finetuning.Parameters.Epochs = 3;
finetuning.Parameters.LearningRate = 2e-4f;
finetuning.Parameters.Seed = 42;

var labeled = new (string ImagePath, string Answer)[]
{
    ("photos/panel-0042.jpg", "The display reads 47.3."),
    ("photos/panel-0043.jpg", "The display reads 8.1."),
    // ... one entry per labeled image
};

var data = new ChatHistory(model);
for (int i = 0; i < labeled.Length; i++)
{
    if (i > 0)
    {
        data.AddMessage(AuthorRole.BeginOfNewConversation, string.Empty);
    }
    var attachment = new Attachment(labeled[i].ImagePath);
    data.AddMessage(new ChatHistory.Message("What value does the display show?", attachment));
    data.AddMessage(AuthorRole.Assistant, labeled[i].Answer);
}

int samples = finetuning.AddTrainingData(data);
Console.WriteLine($"{samples} labeled images loaded");

finetuning.FinetuningProgress += (sender, e) =>
{
    if (!e.IsValidation)
    {
        Console.Write($"\repoch {e.Epoch + 1}/{e.TotalEpochs}  step {e.Step}/{e.TotalSteps}  loss {e.Loss:F4}   ");
    }
};

finetuning.TrainToAdapter("display-reader.gguf");

Attachment also accepts raw bytes (new Attachment(byte[] data, string name)), a stream, or a URI, so images can come from anywhere.


Step 3: Or Load an Image Dataset File

Dataset files declare images per item, and <image> markers in message content place them:

[
  {"id":"s1","images":["images/panel-0042.jpg"],"messages":[
    {"role":"user","content":"What value does the display show?\n\n<image>"},
    {"role":"assistant","content":"The display reads 47.3."}]},
  {"id":"s2","images":["images/panel-0043.jpg"],"messages":[
    {"role":"user","content":"What value does the display show?\n\n<image>"},
    {"role":"assistant","content":"The display reads 8.1."}]}
]

int added = finetuning.AddDatasetFile("dataset.zip");

Image resolution rules:

  • Relative paths resolve beside the dataset file, or inside the ZIP archive when the dataset ships as one. Packaging the JSON together with its images/ folder in a single ZIP is the recommended distribution shape, and it is exactly what TrainingDataset.ExportAsSharegpt and ShareGptExporter produce.
  • Inline data: URIs and base64 images resolve anywhere.
  • A dataset whose declared images all fail to resolve is refused with a FinetuningException instead of silently training text-only. Partially resolved datasets train and log a warning with the affected count.

TrainingDataset.Load(model, path) exposes ResolvedImageCount, UnresolvedImageCount, and UnresolvedImageRefs to check a dataset before training.


Step 4: Evaluate on Held-Out Images

using LMKit.TextGeneration;
using LMKit.TextGeneration.Sampling;

model.ApplyLoraAdapter("display-reader.gguf");

using var chat = new MultiTurnConversation(model)
{
    SamplingMode = new GreedyDecoding(),
    MaximumCompletionTokens = 24
};

var probe = new Attachment("photos/held-out-0100.jpg");
string reply = chat.Submit(new ChatHistory.Message("What value does the display show?", probe)).Completion;
Console.WriteLine(reply);

Hold out images the training never saw and compare base against tuned replies, exactly as for text fine-tunes. The vision display reader sample runs this full loop with generated images and a measurable before/after score.


Common Issues

Problem Cause Fix
FinetuningException: training data contains images but the model has no vision weights The base model is text-only Pick a vision-capable model (model.HasVision)
FinetuningException about the Output target module Image training does not support LoraTargetModules.Output Use Attention or AttentionAndFeedForward
FinetuningException: dataset references N images, none could be resolved Loose JSON with missing image files, or wrong relative paths Ship the dataset as a ZIP with its images folder, or fix the paths; check UnresolvedImageRefs
Out of device memory Image samples carry hundreds to thousands of encoder cells Lower Configuration.DefaultImageDetail (or the attachment's ImageDetail); lower MicroBatchSize; reduce Rank
Slow steps Large images at high detail ImageDetail.Minimal or Standard is usually enough for reading tasks; reserve high detail for fine-grained visuals

Next Steps

Share