Field EnableSessionStateOffload
Gets or sets a value indicating whether an idle conversation's cached state may be moved OUT of device memory instead of being discarded, so that resuming it restores the state rather than reprocessing the whole history.
public static bool EnableSessionStateOffloadReturns
- bool
trueif idle conversation state may be offloaded; otherwise,false. Default istrue.
Remarks
The shared decoder can only keep a few dozen conversations resident at once: sequence slots are a fixed resource, and retained state widens the attention window that every decode pays for. With this enabled, a conversation pushed out of that resident set keeps its state in host memory, and on disk behind that, neither of which the decoder competes for. Thousands of conversations can then stay warm, where previously everything past the resident set reprocessed its full history on every turn.
State is only kept when doing so is expected to cost less than reprocessing, judged from measured transfer and prompt-processing rates, so enabling this never trades throughput for capacity. The budgets adapt to the host; the disk tier is placed under ContextHibernationDirectory and is reclaimed when the decoder that produced it goes away.