Enum InferenceScheduling

Namespace

LMKit.Global

Assembly

LM-Kit.NET.dll

How the runtime allocates inference state and schedules requests onto it. Selected once for the process; the two strategies are never mixed.

public enum InferenceScheduling

Fields

PerRequestContext = 0

One inference context per completion, drawn from a recycling pool and returned to it when the completion ends.

Contexts are sized to the request, are reused across requests of compatible shape, may hibernate to disk while idle, and grow their window by shifting when a conversation outlives it. Speculative decoding runs the single-sequence draft machine. This is the long-standing behaviour and the default.

SharedSlotPool = 1

One inference context, allocated once at a fixed size divided into a fixed number of slots, resident for as long as the model is loaded. Every request is scheduled onto a slot.

Requests decode together in one batch, which is what makes concurrent throughput scale with the slot count rather than dividing a fixed budget between competing contexts. Nothing about the pool adapts at runtime: its size and slot count are exactly what SharedSlotPool asked for, so the memory footprint is known before the first request and does not move afterwards.

Remarks

The selection is honoured as given. Under SharedSlotPool, the only requests that decode on their own context are those hitting a documented capability limit of the pool, and each records a reason readable through ParallelDecodingDiagnostics.GetClassicFallbacks.

Distinct from InferenceMode, which says what KIND of inference a context performs. This says where that state lives and how requests reach it.