Table of Contents

Fine-Tune a Model with LoRA

LMKit.Finetuning.LoraFinetuning trains a LoRA (Low-Rank Adaptation) adapter on your own data, entirely on the local machine, in-process, with no Python toolchain. Training runs on the same GGUF model files used for inference, quantized bases included, and produces GGUF artifacts: a small adapter applied at load time, or a merged standalone model. When a CUDA device is present, the full training step runs on the GPU; otherwise training falls back to the CPU.

This guide covers the complete training loop: data loading, hyperparameters, live metrics, checkpointing, and producing a deployable artifact.


Why Fine-Tune

Fine-tuning fixes problems that prompting cannot:

  1. The model does not know your domain. Your database schema, product names, error codes, or internal terminology are not in any base model. Pasting them into every prompt costs tokens and latency on every request; a fine-tune bakes them into the weights once.
  2. The output format drifts. A model asked to reply with exactly one word, one SQL statement, or one JSON shape complies most of the time. Fine-tuning on a few dozen examples turns "most of the time" into consistent behavior, on models small enough to run anywhere.

If the knowledge changes daily (support tickets, news, inventory), retrieval is the better tool: see Should I Use RAG or Fine-Tuning?. The two combine well: fine-tune for format and tone, retrieve for facts.


Prerequisites

Requirement Minimum
.NET SDK 8.0+
Base model Any text-generation GGUF or LM-Kit model (embedding models cannot be fine-tuned)
Training data A few dozen examples upward; a dataset file or in-code conversations
Hardware CUDA GPU recommended (an order of magnitude faster); CPU works

Training roughly doubles the model's memory footprint compared to inference, so prefer a compact base such as qwen3.5:0.8b or qwen3.5:2b for the first runs.


Step 1: Create the Project

dotnet new console -n FineTuneLora
cd FineTuneLora
dotnet add package LM-Kit.NET

Step 2: Load the Base Model

using LMKit.Model;

LMKit.Licensing.LicenseManager.SetLicenseKey("");

using LM model = LM.LoadFromModelID("qwen3.5:0.8b");

Any model from the catalog or a custom GGUF path works. Two choices matter here:

  • Quantized base (the default). Training on the Q4 catalog artifact works and keeps downloads small. The adapter learns against the quantized weights it will run on.
  • Full-precision base (maximum fidelity). Many catalog models also publish F16 weights. Training against them gives the adapter and any later merge the most accurate gradient signal, at the cost of a larger download:
using LMKit.Model;

ModelCard card = ModelCard.GetPredefinedModelCardByModelID("qwen3.5:0.8b");

using LM model = card.FullPrecisionUri != null
    ? new LM(card.FullPrecisionUri)
    : LM.LoadFromModelID("qwen3.5:0.8b");

Step 3: Load Training Data

LoraFinetuning accepts data through three routes, freely combined:

using LMKit.Finetuning;
using LMKit.TextGeneration.Chat;

using var finetuning = new LoraFinetuning(model);

// 1. A dataset file: chat JSONL, ShareGPT JSON, Alpaca JSON, plain text,
//    or a ZIP archive mixing them. The format is auto-detected.
int added = finetuning.AddDatasetFile("data/support-conversations.jsonl");

// 2. Conversations built in code. BeginOfNewConversation markers pack
//    several training conversations into one history.
var history = new ChatHistory(model);
history.AddMessage(AuthorRole.System, "You are the Contoso support assistant.");
history.AddMessage(AuthorRole.User, "How do I rotate an API key?");
history.AddMessage(AuthorRole.Assistant, "Open Settings, select API Keys, choose the key, then select Rotate.");
finetuning.AddTrainingData(history);

// 3. Raw text for continued-pretraining-style data: every token is supervised.
finetuning.AddRawText("Contoso error E-4102 means the sync token expired and a full re-sync is required.");

For chat data, only the assistant turns are supervised by default (AssistantLossOnly = true): the model learns to produce the answers, not to imitate the questions. See Prepare Training Datasets for LoRA Fine-Tuning for the dataset formats in detail.

Inspect what was loaded before training:

Console.WriteLine($"{finetuning.SampleCount} samples, {finetuning.SampleMinLength} to {finetuning.SampleMaxLength} tokens");

// A non-zero value signals a chat-template mismatch: those samples fell back
// to full-sequence loss, which weakens instruction tuning.
if (finetuning.UnmaskedSampleCount > 0)
{
    Console.WriteLine($"Warning: {finetuning.UnmaskedSampleCount} samples could not be masked.");
}

// Samples longer than the training window are skipped; check the cost of a
// window before forcing one with ContextSize.
Console.WriteLine($"{finetuning.CountSamplesLongerThan(2048)} samples exceed 2048 tokens");

Step 4: Configure Hyperparameters

All hyperparameters live on finetuning.Parameters and have working defaults. A typical instruction fine-tune:

finetuning.Parameters.Rank = 8;
finetuning.Parameters.Alpha = 16;
finetuning.Parameters.TargetModules = LoraTargetModules.Attention;
finetuning.Parameters.Epochs = 3;
finetuning.Parameters.LearningRate = 2e-4f;
finetuning.Parameters.Schedule = LearningRateSchedule.Cosine;
finetuning.Parameters.WarmupRatio = 0.05f;
finetuning.Parameters.ValidationSplit = 0.1f;
finetuning.Parameters.Seed = 42;

What to change, and when:

Parameter Default Raise / change it when
Rank 8 The task must absorb real knowledge (16 to 32); keep low for style and format tasks. Adapter size and memory grow with it
Alpha 16 Keep at 2 x rank as a starting rule
TargetModules Attention Use AttentionAndFeedForward for knowledge-heavy tasks; All includes MoE expert projections on MoE models
Epochs 3 Small datasets often need an extra pass or two; watch validation loss for the point of diminishing returns
LearningRate 1e-4 2e-4 is a common instruction-tuning value; lower it if loss oscillates
Schedule Cosine The default decays to MinLearningRate over the run; Constant, CosineWithRestarts, Linear, Polynomial are available
WarmupRatio 0 A short warmup (0.03 to 0.1) stabilizes the first steps at higher learning rates
ValidationSplit 0.05 Fraction of samples held out and reported per epoch; set 0 to train on everything
AssistantLossOnly true Leave on for chat data; irrelevant for raw text
SequencePacking false Turn on for datasets of short samples: several samples share one training window, multiplying throughput
GradientAccumulation 1 Combine gradients from N samples per optimizer step: a larger effective batch without more memory
UseRsLora false Rank-stabilized scaling (alpha divided by the square root of the rank); keeps ranks of 32+ training at a usable effective scale
LoraPlusRatio 0 LoRA+: trains the B matrices at this multiple of the learning rate (16 is the published value); speeds convergence at no memory cost
NeftuneAlpha 0 NEFTune embedding noise (5 to 15): a regularizer against overfitting on small datasets
FirstLayer / LastLayer 0 / 0 Restrict adapters to a block range; the last third of the blocks often keeps most of the quality at a fraction of the memory
Seed 0 Set for reproducible adapter initialization

Two more levers live on the engine itself:

  • finetuning.ContextSize: the training window. By default it is sized to the longest loaded sample (capped by the model's trained window), so short-sample datasets train fast automatically.
  • finetuning.MicroBatchSize: tokens evaluated per backward pass. This is the first lever when a run does not fit device memory: halve it. Larger values run faster when memory allows.

Step 5: Train with Live Metrics

finetuning.FinetuningProgress += (sender, e) =>
{
    if (e.IsValidation)
    {
        Console.WriteLine($"epoch {e.Epoch + 1}: validation loss {e.Loss:F4}, accuracy {e.Accuracy:P1}");
    }
    else
    {
        Console.Write($"\repoch {e.Epoch + 1}/{e.TotalEpochs}  step {e.Step}/{e.TotalSteps}  loss {e.Loss:F4}  lr {e.LearningRate:E1}   ");
    }
};

finetuning.TrainToAdapter("contoso-support.gguf");

The event fires after every optimizer step and every validation batch. Three control patterns build on it:

// Stop early from inside the handler; the adapter trained so far is saved.
finetuning.FinetuningProgress += (sender, e) =>
{
    if (!e.IsValidation && e.Loss < 0.05)
    {
        e.Stop = true;
    }
};

// Keep the best validation checkpoint while training continues.
double bestLoss = double.MaxValue;
finetuning.FinetuningProgress += (sender, e) =>
{
    if (e.IsValidation && e.Step == e.TotalSteps && e.Loss < bestLoss)
    {
        bestLoss = e.Loss;
        finetuning.TrySaveAdapterSnapshot("contoso-support-best.gguf");
    }
};

// Stop from another thread (a cancel button, a timeout).
finetuning.RequestStop();

TrySaveAdapterSnapshot is valid only from inside a FinetuningProgress handler, which is exactly where best-checkpoint policies live.


Step 6: Checkpoint and Resume

Long runs survive interruptions by writing step checkpoints, then resuming from the optimizer state:

// Write a checkpoint (adapter + optimizer state + metadata) every 200 steps.
finetuning.CheckpointDirectory = "checkpoints";
finetuning.CheckpointSaveSteps = 200;

finetuning.TrainToAdapter("contoso-support.gguf");

To resume after an interruption:

using var resumed = new LoraFinetuning(model);
resumed.Parameters.Rank = 8;               // must match the checkpointed run
resumed.Parameters.Alpha = 16;
resumed.Parameters.TargetModules = LoraTargetModules.Attention;

resumed.AddDatasetFile("data/support-conversations.jsonl");

resumed.ResumeOptimizerPath = "checkpoints/optimizer-step-400.bin";
resumed.ResumeStep = 400;

resumed.TrainToAdapter("contoso-support.gguf");

The runtime restores the optimizer moments and the adapter weights, and validates that the adapter identity (rank, alpha, target modules) matches the run being resumed.


Step 7: Ship an Adapter or a Merged Model

Two output shapes, one training run:

// A: a small adapter file, applied at load time. Keep one base model and
// swap adapters per tenant, per task, or per experiment.
finetuning.TrainToAdapter("contoso-support.gguf");
model.ApplyLoraAdapter("contoso-support.gguf", scale: 1.0f);

// B: one standalone GGUF model with the adapter merged in. Nothing to manage
// at runtime; loads like any other model.
finetuning.TrainToModel("contoso-support-merged.gguf");

An adapter is typically a few MB to a few tens of MB; the merged model is the size of the base. For multi-adapter blending, scale control, and merging existing adapters (including into encrypted bases), see Load and Merge LoRA Adapters at Inference Time.


Step 8: Measure Before and After

A fine-tune without a held-out measurement is a guess. Keep a small set of inputs the training never saw, and compare:

using LMKit.TextGeneration;
using LMKit.TextGeneration.Sampling;

int Evaluate(LM lm, (string Input, string Expected)[] heldOut)
{
    int correct = 0;
    foreach (var (input, expected) in heldOut)
    {
        var chat = new SingleTurnConversation(lm)
        {
            SystemPrompt = "You are the Contoso support assistant.",
            SamplingMode = new GreedyDecoding(),
            MaximumCompletionTokens = 64
        };
        if (chat.Submit(input).Completion.Contains(expected, StringComparison.OrdinalIgnoreCase))
        {
            correct++;
        }
    }
    return correct;
}

int before = Evaluate(model, heldOut);
model.ApplyLoraAdapter("contoso-support.gguf");
int after = Evaluate(model, heldOut);
Console.WriteLine($"held-out: {before}/{heldOut.Length} -> {after}/{heldOut.Length}");

Greedy decoding makes the comparison deterministic. The text-to-SQL sample shows this pattern end to end with a measurable task.


Common Issues

Problem Cause Fix
UnmaskedSampleCount is not zero Assistant spans could not be located in the rendered chat template Check the dataset roles; those samples train with full-sequence loss instead of assistant-only loss
"No usable training samples after packing (N skipped)" Samples longer than the training window Raise ContextSize, or shorten samples; CountSamplesLongerThan shows the impact before training
Out of device memory during training Backward-pass activations exceed free memory Lower MicroBatchSize (halve it); reduce Rank; narrow TargetModules; restrict FirstLayer/LastLayer
Loss decreases but held-out quality does not improve Overfitting on a small set, or the task needs more capacity Add data variety; try NeftuneAlpha = 5; for knowledge tasks widen TargetModules to AttentionAndFeedForward and raise Rank
Loss oscillates or diverges Learning rate too high for the dataset Lower LearningRate; add WarmupRatio = 0.05f; keep MaxGradNorm at its default 1.0
FinetuningException about images and the Output module Image training does not support LoraTargetModules.Output Use Attention or AttentionAndFeedForward; see the vision fine-tuning guide
Training is slow Running on CPU Training is GPU-accelerated on CUDA; on CPU, enable SequencePacking for short samples and keep the model small

API Reference

Member Description
LoraFinetuning(LM model) / LoraFinetuning(string modelPath) Create the training engine over a loaded model or a model file
AddDatasetFile(path) / AddDataset(dataset) Load a dataset file (auto-detected format) or a TrainingDataset
AddTrainingData(ChatHistory) / AddRawText(text) Add conversations or raw-text samples in code
Parameters LoraTrainingParameters: rank, alpha, target modules, epochs, learning rate, schedule, regularizers
FinetuningProgress Per-step and per-validation-batch metrics; set Stop to end the run
TrySaveAdapterSnapshot(path) Save the current adapter weights mid-run (inside the progress handler)
RequestStop() Cooperative stop; the adapter trained so far is saved
CheckpointDirectory / CheckpointSaveSteps Step checkpointing
ResumeOptimizerPath / ResumeStep Resume a checkpointed run
ContextSize / MicroBatchSize Training window and memory lever
SampleCount / SampleMinLength / SampleMaxLength / UnmaskedSampleCount / CountSamplesLongerThan(n) Dataset diagnostics
TrainToAdapter(path) Train and save a GGUF adapter
TrainToModel(path, scale) Train, merge into the base, and save a standalone GGUF model

Next Steps

Share