Table of Contents

Method Quantize

Namespace
LMKit.Quantization
Assembly
LM-Kit.NET.dll

Quantize(string, Precision, bool, MetadataCollection)

Quantizes the model specified by Quantizer(string) and saves the quantized model to the specified destination file.

public void Quantize(string dstFileName, LLM.Precision modelPrecision = Precision.MOSTLY_Q4_K_M, bool quantizeOutputTensor = true, LLM.MetadataCollection metadataOverrides = null)

Parameters

dstFileName string

The file path where the quantized model will be saved.

modelPrecision LLM.Precision

The desired precision mode for the quantized model.
Only the following enumeration members are accepted:

Defaults to MOSTLY_Q4_K_M.
quantizeOutputTensor bool

Indicates whether the output tensor should be quantized. Defaults to true.

metadataOverrides LLM.MetadataCollection

A collection of metadata overrides to apply during the quantization process. Defaults to null.

Exceptions

QuantizationException

Thrown when the quantization process fails.