Field FavorDistributedInference

Namespace

LMKit.Global

Assembly

LM-Kit.NET.dll

Determines whether models load spread across every eligible GPU instead of living on their main device alone.

public static bool FavorDistributedInference

Returns

bool

true to spread models across all eligible GPUs; false to place each model on one device. Default is false.

Remarks

When true, a model with no explicit LM.TensorDistribution proportions has its layers distributed over all eligible GPUs in proportion to their free memory (integrated GPUs join only when no discrete device exists). This pools the devices' memory, so models larger than any single card load fully on GPU, and lets large prompt batches overlap across devices when every layer is offloaded.

Layer distribution does not make single-request token generation faster: the devices hold different layers, so a token's forward pass visits them one after another. For a model that already fits one GPU comfortably, single-device placement avoids the inter-device hop entirely.

The value is captured when a model loads; changing it affects subsequent loads, never models already resident.