Table of Contents

Property TensorDistribution

Namespace
LMKit.Model
Assembly
LM-Kit.NET.dll

TensorDistribution

Gets or sets how the model's tensors are distributed across GPU devices.

public LM.TensorDistribution TensorDistribution { get; set; }

Property Value

LM.TensorDistribution

Examples

var config = new LM.DeviceConfiguration
{
    // 70% of the layers on the first device, 30% on the second.
    TensorDistribution = new LM.TensorDistribution(new[] { 0.7f, 0.3f })
};

LM model = new LM(modelUri, deviceConfiguration: config);

Remarks

The default (no explicit proportions, layer mode) lets the runtime decide: the model lives on its main device alone, unless FavorDistributedInference is enabled, in which case its layers are spread over every eligible GPU in proportion to free memory. Supply explicit proportions to control that spread directly; entry i maps to the i-th device of the runtime's device order (discrete GPUs first; integrated GPUs participate only when no discrete device exists).

Layer mode (the default) assigns whole layers to devices: it pools their memory, and a token's forward pass visits the devices in sequence. Row mode splits the weight matrices themselves so the devices compute each layer together; it needs backend support for tensor-parallel execution and fast interconnect to pay off.

Share