Class SharedSlotPoolOptions
Shape and admission policy of the resident slot pool used when InferenceScheduling is SharedSlotPool.
public sealed class SharedSlotPoolOptions- Inheritance
-
SharedSlotPoolOptions
- Inherited Members
Remarks
A pool is allocated once at the configured shape and is never resized or retired while the model is loaded, so its memory footprint is known before the first request. Changing a value here marks resident pools stale; each is rebuilt at the new shape once its in-flight requests complete.
Two scopes exist. SharedSlotPool is the
process-wide shape every model uses by default. Each model also exposes
its own instance through SharedSlotPool, which
INHERITS every value it does not state: reading a property there answers
with the process-wide value until that property is overridden, and
follows later changes to it. The *Override properties are the
tri-state view of the same values, where null means
"inherit"; assigning null to one restores inheritance.
Constructors
- SharedSlotPoolOptions()
Creates a standalone shape whose unstated values fall back to the SDK defaults. Governs nothing on its own: only the instances SharedSlotPool and SharedSlotPool expose drive live pools.
Properties
- Allocation
When the pool claims its memory. Defaults to OnFirstCompletion, so a process that loads a model and never runs a completion reserves nothing.
- BatchInvariantDecoding
Whether the pool decodes batch-invariantly. Defaults to true. Every request's logits are then bit-identical whatever other requests share its decode passes, however many rows it contributes to a pass and wherever its cache cells sit, so identical greedy requests answer identically from one run to the next. Reading answers with the inherited value while BatchInvariantDecodingOverride is null; assigning sets the override.
- BatchInvariantDecodingOverride
BatchInvariantDecoding as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- LookupDraftEqualization
Chain-length rule for lookup-drafted passes that carry several slots. The default is ExpectedValue on every backend: what a verified row costs depends on the device (its kernels, and on CUDA, where the rows are decided on the device and the pass shape is kept fixed, the host link and the host scans), so only a rule that prices the rows from the pass costs measured on the device itself drafts where it pays. Drafting changes how fast a completion arrives, never what it emits. Read on every pass, so a change applies to the next pass without a rebuild. Reading answers with the inherited value while LookupDraftEqualizationOverride is null; assigning sets the override.
- LookupDraftEqualizationOverride
LookupDraftEqualization as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance. A rule change needs no rebuild.
- LookupDrafts
Whether a request that asks to be drafted by lookup (a reference sequence the completion is expected to reproduce, such as a page's text layer for a transcription) is drafted on the pool. Defaults to true. Reading answers with the inherited value while LookupDraftsOverride is null; assigning sets the override.
- LookupDraftsOverride
LookupDrafts as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- SlotContextSize
Token window of a single slot: the longest prompt plus completion one request may occupy. Defaults to 8192. Values below 256 are raised to 256; the model's own trained window caps it at attach.
- SlotContextSizeOverride
SlotContextSize as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- SlotCount
Requests the pool decodes together. Defaults to 4, clamped to [1, 64].
- SlotCountOverride
SlotCount as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- TotalContextSize
Tokens the pool's context holds in total, derived from the slot shape. Excludes the runtime's own staging reserve.
- WhenAllSlotsBusy
What a request does when every slot is busy. Defaults to Queue, which makes a burst wider than the pool wait rather than fail.