Table of Contents

Prepare Training Datasets for LoRA Fine-Tuning

The quality of a fine-tune depends entirely on the training data. LM-Kit.NET covers the whole dataset lifecycle: load the standard industry formats (chat JSONL, ShareGPT, Alpaca, plain text, ZIP archives, all auto-detected), build datasets programmatically from ChatHistory objects, export to ShareGPT JSON for review and version control, and inspect what the trainer will actually see before spending compute on it.


Why Dataset Preparation Matters

  1. Your data already exists in a standard format. Teams accumulate training pairs as OpenAI-style JSONL, ShareGPT exports, or Alpaca files. LoraFinetuning.AddDatasetFile reads them directly, so no conversion scripts sit between your data and a training run.
  2. Reviewable, versionable data beats opaque pipelines. Exporting to ShareGPT JSON lets you inspect, filter, and version-control training data before committing to a run, and re-export after every refinement without touching model code.

Prerequisites

Requirement Minimum
.NET SDK 8.0+
Base model Any text-generation model (used for tokenization and template rendering)
Disk Space for model + training data + output adapter

Step 1: Create the Project

dotnet new console -n DatasetPrep
cd DatasetPrep
dotnet add package LM-Kit.NET

Step 2: Understand the Dataset Pipeline

   files (auto-detected)                built in code
┌──────────────────────────┐      ┌───────────────────┐
│ chat JSONL / ShareGPT /  │      │  ChatHistory      │
│ Alpaca / text / ZIP      │      │  (User/Assistant) │
└────────────┬─────────────┘      └─────────┬─────────┘
             │                              │
             ▼                              ▼
      ┌──────────────────────────────────────────┐
      │             TrainingDataset              │
      │  (ChatTrainingSample + raw-text samples) │
      └───────┬──────────────────────────┬───────┘
              │                          │
              ▼                          ▼
      ┌────────────────┐        ┌─────────────────┐
      │ LoraFinetuning │        │ ShareGptExporter│
      │ (training run) │        │ (.json + images)│
      └────────────────┘        └─────────────────┘
Class Purpose
TrainingDataset Loads dataset files (format auto-detected) and holds chat plus raw-text samples
ChatHistory A sequence of role-tagged messages (system, user, assistant), optionally with image attachments
ChatTrainingSample Wraps a ChatHistory for training, with a target modality
ShareGptExporter / DatasetBuilderOptions Streams samples to a ShareGPT JSON file with an images folder
LoraFinetuning The training engine; accepts dataset files, TrainingDataset, ChatHistory, and raw text

Step 3: Load Existing Dataset Files

AddDatasetFile (and TrainingDataset.Load) auto-detect the format from extension and content:

using LMKit.Finetuning;
using LMKit.Model;

LMKit.Licensing.LicenseManager.SetLicenseKey("");

using LM model = LM.LoadFromModelID("qwen3.5:2b");
using var finetuning = new LoraFinetuning(model);

// Any of these work; formats can be mixed across calls.
finetuning.AddDatasetFile("conversations.jsonl");  // chat JSONL (the OpenAI shape)
finetuning.AddDatasetFile("sharegpt.json");        // ShareGPT
finetuning.AddDatasetFile("alpaca.json");          // Alpaca
finetuning.AddDatasetFile("corpus.txt");           // plain text
finetuning.AddDatasetFile("dataset.zip");          // ZIP bundling any mix, plus image assets

The accepted shapes:

Chat JSONL: one JSON object per line; lines starting with # are skipped.

{"messages":[{"role":"system","content":"You are the Contoso assistant."},{"role":"user","content":"How do I rotate an API key?"},{"role":"assistant","content":"Open Settings, select API Keys, then Rotate."}]}

ShareGPT: a JSON array of {"conversations":[{"from","value"}]} items. Images sit at the item level and <image> markers in turn text consume them in order.

Alpaca: a JSON array of {"instruction","input","output"}; instruction and input merge into the user turn.

Plain text: a corpus split into raw-text samples on the <SFT> delimiter when present, otherwise on blank-line paragraph boundaries. Every token is supervised (continued-pretraining style).

<SFT>
Contoso error E-4102 means the sync token expired and a full re-sync is required.
<SFT>
Contoso error E-2210 means the device certificate was rejected.

ZIP archive: every .jsonl, .json, and .txt entry inside is loaded and merged; other entries are treated as image assets referenced by the datasets. This is the recommended shape for image datasets, and it is what the exporter produces.

To inspect before feeding a run:

var dataset = TrainingDataset.Load(model, "dataset.zip");
Console.WriteLine($"{dataset.Count} samples ({dataset.Samples.Count} chat, {dataset.RawTextSamples.Count} raw text)");
Console.WriteLine($"images: {dataset.ResolvedImageCount} resolved, {dataset.UnresolvedImageCount} unresolved");

finetuning.AddDataset(dataset);

A dataset whose declared images all fail to resolve is refused with a FinetuningException rather than silently training text-only. For the image-specific rules, see Fine-Tune Vision Models on Image Data.


Step 4: Build Training Samples from ChatHistory

using System.Text;
using LMKit.Finetuning;
using LMKit.Model;
using LMKit.TextGeneration.Chat;

LMKit.Licensing.LicenseManager.SetLicenseKey("");

Console.OutputEncoding = Encoding.UTF8;

using LM model = LM.LoadFromModelID("qwen3.5:2b",
    loadingProgress: p => { Console.Write($"\r  Loading: {p * 100:F0}%   "); return true; });

var dataset = new TrainingDataset();

// Sample 1: single-turn Q&A
var chat1 = new ChatHistory(model);
chat1.AddMessage(AuthorRole.System, "You are a customer support agent for Acme Corp.");
chat1.AddMessage(AuthorRole.User, "How do I reset my password?");
chat1.AddMessage(AuthorRole.Assistant,
    "To reset your password, go to Settings > Account > Change Password. " +
    "Enter your current password, then type your new password twice. " +
    "Click Save to confirm the change.");
dataset.AddSample(new ChatTrainingSample(chat1));

// Sample 2: multi-turn conversation; every assistant turn is supervised
var chat2 = new ChatHistory(model);
chat2.AddMessage(AuthorRole.System, "You are a customer support agent for Acme Corp.");
chat2.AddMessage(AuthorRole.User, "What are your business hours?");
chat2.AddMessage(AuthorRole.Assistant,
    "Our support team is available Monday through Friday, 9 AM to 6 PM Eastern Time.");
chat2.AddMessage(AuthorRole.User, "Do you have weekend support?");
chat2.AddMessage(AuthorRole.Assistant,
    "We offer limited weekend support via email only. " +
    "Emails received on weekends are answered by Monday noon.");
dataset.AddSample(new ChatTrainingSample(chat2));

Console.WriteLine($"Created {dataset.Samples.Count} training samples.");

// Export for review, filtering, and version control
dataset.ExportAsSharegpt("training_data.json", overwrite: true);

Step 5: Advanced Export with Options and Progress

For larger datasets, ShareGptExporter streams to disk with progress and fine-grained control:

using System.Text;
using LMKit.Finetuning;
using LMKit.Finetuning.Export;
using LMKit.Inference;
using LMKit.Model;
using LMKit.TextGeneration.Chat;

LMKit.Licensing.LicenseManager.SetLicenseKey("");

Console.OutputEncoding = Encoding.UTF8;

using LM model = LM.LoadFromModelID("qwen3.5:2b");

var samples = new List<ChatTrainingSample>();

string[,] qaData = {
    { "What is your return policy?", "Items can be returned within 30 days of purchase with receipt." },
    { "How do I track my order?", "Log into your account and visit Orders > Track Shipment." },
    { "Do you offer international shipping?", "Yes, we ship to 40+ countries. Rates vary by destination." },
    { "How do I cancel my subscription?", "Go to Account > Subscriptions > Cancel. Effective at billing cycle end." }
};

for (int i = 0; i < qaData.GetLength(0); i++)
{
    var chat = new ChatHistory(model);
    chat.AddMessage(AuthorRole.System, "You are a helpful customer support agent.");
    chat.AddMessage(AuthorRole.User, qaData[i, 0]);
    chat.AddMessage(AuthorRole.Assistant, qaData[i, 1]);
    samples.Add(new ChatTrainingSample(chat, InferenceModality.Text));
}

var options = new DatasetBuilderOptions
{
    Overwrite = true,
    IndentedJson = true,
    ImagePrefix = "sample",
    ImageFolderName = "images",
    RoleMappingPolicy = RoleMappingPolicy.Strict,
    ContinueOnError = false,
    ExpectedCount = samples.Count
};

var progress = new Progress<ExportProgress>(p =>
{
    Console.Write($"\r  Exporting: {p.Completed}/{p.Total} ({p.Percent:F0}%)   ");
});

ExportResult result = await ShareGptExporter.ExportAsync(
    samples,
    "customer_support_dataset.json",
    options,
    progress);

Console.WriteLine($"\n  Samples written: {result.SamplesWritten}");
Console.WriteLine($"  JSON path:       {result.JsonPath}");
Console.WriteLine($"  Images folder:   {result.ImagesFolder}");
Console.WriteLine($"  Skipped:         {result.SkippedSamples}");

Step 6: Inspect What the Trainer Will See

The trainer renders every conversation through the model's chat template and masks the non-assistant tokens. Check the result before training:

using var finetuning = new LoraFinetuning(model);
int added = finetuning.AddDatasetFile("customer_support_dataset.json");

Console.WriteLine($"{added} samples, {finetuning.SampleMinLength} to {finetuning.SampleMaxLength} tokens");

// Non-zero means those samples could not be masked to assistant-only loss:
// a chat-template mismatch worth fixing before training.
if (finetuning.UnmaskedSampleCount > 0)
{
    Console.WriteLine($"Warning: {finetuning.UnmaskedSampleCount} unmasked samples");
}

// Samples longer than the training window are skipped at training time.
int tooLong = finetuning.CountSamplesLongerThan(2048);
if (tooLong > 0)
{
    Console.WriteLine($"{tooLong} samples exceed 2048 tokens; raise ContextSize or shorten them");
}

From here, the training run itself (hyperparameters, live metrics, checkpointing, artifacts) is covered in Fine-Tune a Model with LoRA.


ShareGPT JSON Output Format

The exported JSON follows the ShareGPT schema, compatible with many fine-tuning tools, and loads straight back through AddDatasetFile:

[
  {
    "id": "sample001",
    "images": [],
    "messages": [
      { "role": "system", "content": "You are a helpful customer support agent." },
      { "role": "user", "content": "What is your return policy?" },
      { "role": "assistant", "content": "Items can be returned within 30 days..." }
    ]
  },
  {
    "id": "sample002",
    "images": ["images/sample002_1.png"],
    "messages": [
      { "role": "user", "content": "What does this image show?\n\n<image>" },
      { "role": "assistant", "content": "The image shows a product diagram..." }
    ]
  }
]

Relative image paths resolve beside the JSON file or inside its ZIP archive; packaging the JSON with its images/ folder in one ZIP keeps the dataset portable.


Role Mapping Policies

When source data contains non-standard roles, RoleMappingPolicy controls export behavior:

Policy Behavior Use Case
Strict (default) Roles are left as-is. Export fails on unrecognized roles Clean, validated data
CoerceUnknownToUser Unknown roles are mapped to "user" Data from external sources
DropUnknown Messages with unknown roles are silently dropped Noisy data with metadata messages

Dataset Quality Checklist

Check Why
Consistent system prompts The system prompt used in training should match the one used at inference
Balanced turn lengths Extremely long or short assistant responses skew training
No duplicate samples Duplicates cause overfitting to specific examples
Representative distribution Include edge cases and paraphrases, not just the most common phrasing
Correct role ordering System first, then alternating user/assistant
UnmaskedSampleCount == 0 Every chat sample trains with assistant-only loss

Common Issues

Problem Cause Fix
ExportAsync throws on the first sample Empty ChatHistory Ensure each sample has at least one user and one assistant message
Low quality after fine-tuning Too few or too uniform samples Aim for 50+ diverse examples; add paraphrases of the same intent
AddDatasetFile returns fewer samples than expected Malformed JSON lines are skipped; unknown array shapes are ignored Validate the JSON; one object per line for JSONL
FinetuningException about unresolved images Loose JSON whose image files are missing Ship the dataset as a ZIP with its images, or fix the relative paths
Samples skipped at training time Longer than the training window Check CountSamplesLongerThan; raise ContextSize or shorten samples

Next Steps

Share