Property IdleDeviceMemoryRelease
IdleDeviceMemoryRelease
Gets or sets how long a resident slot pool must stay idle before the device memory it holds only transiently returns to the driver: the context its attachment prompts are prefilled on, the backend scratch its decodes grew (temporaries kept at the largest size ever requested), and the vision encoder's compute buffers (kept at the size the largest image required). The pool itself, its slots and every retained conversation stay resident; the released memory is rebuilt by the next request that needs it, at a cost of milliseconds for scratch and compute buffers and a context creation for the attachment prefill context.
public static TimeSpan IdleDeviceMemoryRelease { get; set; }Property Value
- TimeSpan
Default is 5 minutes.
Remarks
Read live by every pool. Zero (or any negative value) never releases on idle; the memory is then released only under device memory pressure or by ReleaseIdleDeviceMemory().