Class SharedSlotPoolOptions
Shape and admission policy of the resident slot pool used when InferenceScheduling is SharedSlotPool.
public sealed class SharedSlotPoolOptions- Inheritance
-
SharedSlotPoolOptions
- Inherited Members
Remarks
A pool is allocated once at the configured shape and is never resized or retired while the model is loaded, so its memory footprint is known before the first request. Changing a value here marks resident pools stale; each is rebuilt at the new shape once its in-flight requests complete.
Two scopes exist. SharedSlotPool is the
process-wide shape every model uses by default. Each model also exposes
its own instance through SharedSlotPool, which
INHERITS every value it does not state: reading a property there answers
with the process-wide value until that property is overridden, and
follows later changes to it. The *Override properties are the
tri-state view of the same values, where null means
"inherit"; assigning null to one restores inheritance.
Constructors
- SharedSlotPoolOptions()
Creates a standalone shape whose unstated values fall back to the SDK defaults. Governs nothing on its own: only the instances SharedSlotPool and SharedSlotPool expose drive live pools.
Properties
- Allocation
When the pool claims its memory. Defaults to OnFirstCompletion, so a process that loads a model and never runs a completion reserves nothing.
- LookupDraftEqualization
Chain-length rule for lookup-drafted passes that carry several slots. Defaults to Longest: every drafting slot verifies the whole depth, a decision that depends on the proposals alone, so a run's drafting is the same from one run to the next. The expected-value rule prices each pass against timings it measures, and a device shared with other engines makes those timings move; measured on a document workload at four slots the two rules cost the same and the priced rule declined a tenth of the passes for nothing. Read on every pass, so a change applies to the next pass without a rebuild. Reading answers with the inherited value while LookupDraftEqualizationOverride is null; assigning sets the override.
- LookupDraftEqualizationOverride
LookupDraftEqualization as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance. A rule change needs no rebuild.
- LookupDrafts
Whether a request that asks to be drafted by lookup (a reference sequence the completion is expected to reproduce, such as a page's text layer for a transcription) is drafted on the pool. Defaults to true. Reading answers with the inherited value while LookupDraftsOverride is null; assigning sets the override.
- LookupDraftsOverride
LookupDrafts as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- SlotContextSize
Token window of a single slot: the longest prompt plus completion one request may occupy. Defaults to 8192. Values below 256 are raised to 256; the model's own trained window caps it at attach.
- SlotContextSizeOverride
SlotContextSize as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- SlotCount
Requests the pool decodes together. Defaults to 4, clamped to [1, 64].
- SlotCountOverride
SlotCount as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.
- TotalContextSize
Tokens the pool's context holds in total, derived from the slot shape. Excludes the runtime's own staging reserve.
- WhenAllSlotsBusy
What a request does when every slot is busy. Defaults to Queue, which makes a burst wider than the pool wait rather than fail.