Property InferenceMemoryRatio
InferenceMemoryRatio
Gets or sets the fraction of each device's total memory available to standing inference state: the shared parallel decoder's slots and the cached (kept-warm) inference contexts together.
public static double InferenceMemoryRatio { get; set; }Property Value
- double
The default value is
0.35(35% of total device memory). Values are clamped to the range [0.0, 0.9].
Remarks
The budget is arbitrated at runtime, live work first: the parallel
decoder may claim up to the whole budget for its slots (idle cached
contexts yield to an attaching or growing decoder), and the cached
context pool is bounded by whatever the decoder is not currently
reserving on that device. The ratio is applied per device: each
GPU's budget is TotalVRAM * InferenceMemoryRatio, and the
CPU budget is TotalSystemRAM * InferenceMemoryRatio.
Set to 0 to keep no standing inference memory at all:
context caching is disabled and no parallel decoder auto-attaches.
Engines created explicitly through
GetOrCreate(LM, int, int)
use the caller's sizing and bypass this budget.