Class SharedSlotPoolOptions
Shape and admission policy of the resident slot pool used when InferenceScheduling is SharedSlotPool.
public sealed class SharedSlotPoolOptions- Inheritance
-
SharedSlotPoolOptions
- Inherited Members
Remarks
A pool is allocated once at the configured shape and is never resized or retired while the model is loaded, so its memory footprint is known before the first request. Changing a value here marks resident pools stale; each is rebuilt at the new shape once its in-flight requests complete.
Two scopes exist. SharedSlotPool is the
process-wide shape every model uses by default. Each model also exposes
its own instance through SharedSlotPool, which
INHERITS every value it does not state: reading a property there answers
with the process-wide value until that property is overridden, and
follows later changes to it. The *Override properties are the
tri-state view of the same values, where null means
"inherit"; assigning null to one restores inheritance.
Constructors
- SharedSlotPoolOptions()
Creates a standalone shape whose unstated values fall back to the SDK defaults. Governs nothing on its own: only the instances SharedSlotPool and SharedSlotPool expose drive live pools.
Properties
- Allocation
When the pool claims its memory. Defaults to OnFirstCompletion, so a process that loads a model and never runs a completion reserves nothing.
- SlotContextSize
Token window of a single slot: the longest prompt plus completion one request may occupy. Defaults to 8192. Values below 256 are raised to 256; the model's own trained window caps it at attach.
- SlotContextSizeOverride
SlotContextSize as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- SlotCount
Requests the pool decodes together. Defaults to 4, clamped to [1, 64].
- SlotCountOverride
SlotCount as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- TotalContextSize
Tokens the pool's context holds in total, derived from the slot shape. Excludes the runtime's own staging reserve.
- WhenAllSlotsBusy
What a request does when every slot is busy. Defaults to Queue, which makes a burst wider than the pool wait rather than fail.