Method ReleaseIdleDeviceMemory
ReleaseIdleDeviceMemory()
Returns to the device the memory the runtime holds only between requests, across every loaded model: the attachment prefill contexts of the slot pools, the backend scratch of idle contexts (temporaries kept at the largest size any decode requested), and the compute buffers of the vision encoders (kept at the size the largest image required).
public static DeviceMemoryReleaseResult ReleaseIdleDeviceMemory()Returns
- DeviceMemoryReleaseResult
What was released, by kind.
Remarks
Safe on a busy process: memory a request in flight uses is never touched (a busy pool keeps its scratch, a prefill in progress keeps its context), and nothing released changes what a later request computes. Each piece is rebuilt by the next request that needs it, at a cost of milliseconds, or of one context creation for an attachment prefill context. Resident pools, their slots, retained conversations and model weights stay as they are. The runtime also does this on its own when a slot pool has been idle for IdleDeviceMemoryRelease, and before it retires anything under device memory pressure.