Class Quantizer

Namespace: LMKit.Quantization

Assembly: LM-Kit.NET.dll

Provides functionality for quantizing LM models.

public sealed class Quantizer

Inheritance: object

Quantizer

Inherited Members: object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Examples

Example: Quantize a model to 4-bit precision

using LMKit.Quantization;
using LMKit.Model;
using System;

// Create quantizer with source model path
var quantizer = new Quantizer("models/llama-3.2-3b-f16.gguf");

// Set thread count for faster quantization
quantizer.ThreadCount = Environment.ProcessorCount;

// Quantize to 4-bit (Q4_K_M offers good balance of size and quality)
quantizer.Quantize(
    dstFileName: "models/llama-3.2-3b-q4km.gguf",
    modelPrecision: LM.Precision.MOSTLY_Q4_K_M);

Console.WriteLine("Quantization complete!");

// Load and use the quantized model
LM model = LM.LoadFromPath("models/llama-3.2-3b-q4km.gguf");
Console.WriteLine($"Model loaded with {model.Precision} precision");

Example: Quantize to different precision levels

using LMKit.Quantization;
using LMKit.Model;
using System;

string sourceModel = "models/large-model-f16.gguf";

// Create multiple quantized versions
var quantizer = new Quantizer(sourceModel);

// 8-bit for highest quality
quantizer.Quantize("models/large-model-q8.gguf", LM.Precision.MOSTLY_Q8_0);

// 5-bit for balanced size/quality
quantizer.Quantize("models/large-model-q5km.gguf", LM.Precision.MOSTLY_Q5_K_M);

// 4-bit for smallest size
quantizer.Quantize("models/large-model-q4km.gguf", LM.Precision.MOSTLY_Q4_K_M);

Console.WriteLine("All quantization variants created");

Remarks

Quantization reduces model size and memory requirements by converting weights from higher precision (e.g., FP32, FP16) to lower precision formats (e.g., 4-bit, 8-bit). This enables running larger models on hardware with limited resources while maintaining acceptable quality.

Supported Precision Modes

Q4_0, Q4_1, Q4_K_S, Q4_K_M - 4-bit quantization (smallest size)
Q5_0, Q5_1, Q5_K_S, Q5_K_M - 5-bit quantization (balanced)
Q6_K - 6-bit quantization (good quality)
Q8_0 - 8-bit quantization (high quality)
F16, F32 - Full precision (highest quality)

Constructors

Quantizer(string): Initializes a new instance of the Quantizer class with the specified model path.

Properties

ThreadCount: Gets or sets the number of threads to be used for processing.
Ensures that the thread count is always at least 1 to prevent invalid configurations.

Methods