Table of Contents

Method GetOptimalContextSize

Namespace
LMKit.Hardware
Assembly
LM-Kit.NET.dll

GetOptimalContextSize()

Determines the optimal GPU context size based on the currently available GPU device's free memory.

public static int GetOptimalContextSize()

Returns

int

An int that represents an optimal context size for the detected GPU device. Typical values range from 2,048 to 32,768, depending on the available GPU memory.

Examples

using LMKit.Hardware;
using System;

int contextSize = DeviceConfiguration.GetOptimalContextSize();
Console.WriteLine($"Optimal context: {contextSize} tokens");

Remarks

The method identifies the GPU device with the best performance characteristics and then calculates its free memory. Based on predefined memory thresholds, it returns an integer representing an optimal "context size" that can be used for GPU-accelerated operations. The context size scales with available memory, allowing for more complex or larger data sets to be processed efficiently.

If no suitable device is found or if the device's free memory does not meet any threshold, a default value is returned.

GetOptimalContextSize(LM)

Determines the optimal GPU context size based on the currently available GPU device's free memory, with an option to cap this size according to the specified language model.

public static int GetOptimalContextSize(LM model)

Parameters

model LM

An optional LM instance whose maximum allowable context length will be used to cap the computed size.

Returns

int

An int representing the final optimal context size, bounded by the device's memory budget and the model's ContextLength.

Remarks

When InferenceScheduling is SharedSlotPool and the model can host a slot pool, the answer is the pool's per-slot window: the live engine's fitted window when one is attached, otherwise the window an attach would open with. Every pooled request decodes inside a slot of exactly that size, so it is the one context length components can size prompts against without ever disagreeing with the pool that must serve them.

Under per-request scheduling, the method sizes against the model's device: it measures the model's real context-memory curve through the native simulation and returns the largest window whose measured footprint fits the standing inference budget, capped by the model's trained window.

If no suitable GPU device is found or the device's free memory does not meet any threshold, a default size is returned. The final recommended value ensures efficient GPU-accelerated operations while respecting the model's capabilities.

Share