Table of Contents

👉 Try the demo: https://github.com/LM-Kit/lm-kit-net-samples/tree/main/console_net/model-optimization/fine-tuning/vision_display_fine_tuning

Vision Fine-Tuning for C# .NET Applications (Seven-Segment Display Reader)


🎯 Purpose of the Demo

This demo fine-tunes a small vision-language model with LoRA on labeled images, entirely on the local machine. It teaches the model to read seven-segment displays (the value panels on meters, scales, and industrial instruments), measures reading accuracy on held-out values before and after training, and saves a 2 MB GGUF adapter. The workflow is the template for any labeled image task: swap the generated displays for your own photos.

👥 Who Should Use This Demo (Target Audience)

  • .NET developers building vision features for equipment, documents, or products that generic models misread.
  • Teams deploying image understanding on edge hardware where images must not leave the device.

🚀 What Problem It Solves

Segment displays, gauges, and domain-specific visuals are underrepresented in pretraining data: a general VLM drops digits and confuses segments. A few dozen labeled examples teach the exact reading skill needed, on a model small enough for edge deployment. Training runs through the model's own vision path (frozen vision tower, adapters on the language side), so what the adapter learns is what inference runs.

💻 Demo Application Overview

The sample renders deterministic seven-segment displays as BMPs (no image library), builds one training conversation per labeled image (user turn carries the image, assistant turn is the expected reading), binds the image pixel budget once with Configuration.DefaultImageDetail, trains attention adapters for three epochs, then re-evaluates held-out displays with the adapter applied.

✨ Key Features

  • Vision LoRA training with the same API as text: AddTrainingData with image attachments, the same LoraTrainingParameters, the same GGUF output.
  • Controlled image cost: ImageDetail decides the vision-token budget of every training sample.
  • A measurable before/after: exact-value reading accuracy on displays the training never saw.
  • Self-contained: the demo generates its own labeled images deterministically.

Example Output

Before fine-tuning:
  [err]  87.3 <- 8.3
  [err]  91.4 <- 8.1.4
BASE: displays read correctly 3/10

Training on 48 labeled displays (rank 8, 3 epochs)...
Adapter saved: display-reader.gguf (2115 KB)

After fine-tuning:
  [ok ]  70.7 <- The display reads 70.7.
  [ok ]  91.4 <- The display reads 91.4.
TUNED: displays read correctly 7/10

🏗️ Architecture

RenderDisplay(value)  -->  BMP bytes  -->  Attachment
        |
        v
ChatHistory: [user: "What value does the display show?" + image]
             [assistant: "The display reads 47.3."]
        |
        v
LoraFinetuning (Attention adapters, frozen vision tower)
        |
        v
display-reader.gguf  -->  ApplyLoraAdapter  -->  held-out evaluation

⚙️ Getting Started

Prerequisites

  • .NET 8.0 SDK or later.
  • First run downloads qwen3.5:0.8b (about 600 MB, vision-capable).
  • A CUDA GPU is recommended; image samples cost more per training step than text.

Download

Clone the samples repository and open the demo folder console_net/model-optimization/fine-tuning/vision_display_fine_tuning.

Run

dotnet run -c Release

🔧 Troubleshooting

  • Out of memory: lower Configuration.DefaultImageDetail, lower MicroBatchSize, or reduce the rank.
  • Confused digits after training: raise ImageDetail so segment shapes survive encoding, or add more labeled examples per digit.
  • FinetuningException about the Output module: image training supports Attention and AttentionAndFeedForward target modules only.

🚀 Extend the Demo

  • Replace the generated BMPs with photos of your real displays and readings; new Attachment(path) per image is all that changes.
  • Package your dataset as a ShareGPT ZIP (JSON plus images folder) and load it with AddDatasetFile.
  • Apply the same recipe to document layouts, product photos, or defect classification.

📚 Additional Resources

Share