Class ContextInfo

Namespace

LMKit.Inference

Assembly

LM-Kit.NET.dll

Immutable, read-only snapshot of a single inference context (KV-cache) held in memory for a loaded model, as returned by GetLoadedContexts().

public sealed class ContextInfo

Inheritance

object

ContextInfo

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

A loaded model keeps one context per concurrent session or in-flight request, plus any contexts retained in the recycle pool for reuse. Each context owns a KV-cache, which is the dominant per-session memory cost on top of the model weights. This type exposes that cost and the context's lifecycle state so callers can account for a model's full memory footprint and understand what is keeping it resident.

Contexts come in two shapes, told apart by SlotCount. A classic context backs one session at a time. A parallel decoder's shared context backs SlotCount requests at once out of a single allocation, so its ContextLength is a pool sized for many requests rather than one session's window; reading it as a single window makes the largest KV allocation on the machine look like a wildly oversized session.

The values are captured at the moment of the call and never change afterwards; the underlying context is not exposed, so reading them cannot mutate inference state.

Constructors

ContextInfo(string, int, long, ContextResidency, bool, int, bool, bool, KVCacheType, long, long, int, int, int, int, int, long, long, bool)

Initializes a new ContextInfo snapshot.

Properties

BusySlots

Gets the number of slots decoding a request at the moment of the snapshot, or 0 when SlotCount is 0. Persistently far below SlotCount under load means requests are not reaching the decoder, not that the context is oversized.

ContextLength

Gets the context window size, in tokens.

DeviceNumber

Gets the number of the device the context resides on. -1 indicates the CPU; a value of 0 or greater is the GPU device number (see DeviceNumber), matching the convention used by MainGpu.

DraftMemorySize

Gets the size, in bytes, of the speculative-decoding draft (Multi-Token Prediction or attached draft-model) sibling context bound to this session: the draft's own compute buffers, plus its own KV-cache when it keeps one. Reported apart from MemorySize so the draft's footprint is visible on its own. When the draft shares the main context's KV-cache (an attached assistant draft linked through the target), that shared cache belongs to the main context and is counted in MemorySize, not here, so the two never overlap. Returns 0 when the session has no draft context or when the context is hibernated.

FlashAttention

Gets a value indicating whether flash-attention is enabled for the context.

HibernatedBytes

Gets the bytes of serialized context state currently on disk, or 0 when the context is not hibernated. This is what hibernation actually cost, read from the file rather than estimated: it is the disk price being paid to keep the session resumable while it holds no device memory. The file is deleted on rehydration, so this returns to 0 as soon as the context is live again.

Id

Gets the stable, unique identifier of the context.

IsCachePriority

Gets a value indicating whether the context is pinned, exempting it from cache eviction under memory pressure.

IsEmbeddingPool

Gets a value indicating whether this context is a pinned embedding pool rather than a completion context. Embedding pools reuse the slot fields to report their own shape (SlotCount is the packing sequence ceiling, SlotContextSize the packing window), so consumers aggregating completion decode slots must exclude contexts where this is true.

IsInUse

Gets a value indicating whether the context is actively held by a session or an in-flight request (true), or sits idle in the recycle pool kept warm for reuse (false).

KVCacheQuantization

Gets the data type the context's KV-cache is stored in, that is, its quantization level. F16 is the unquantized default; lower-precision types such as Q8_0 trade accuracy for a smaller per-token footprint.

MemorySize

Gets the context's main KV-cache plus scheduler-managed compute-buffer size, in bytes. Returns 0 when the context is hibernated (its memory has been released to disk; see Residency). This does not include the draft context (see DraftMemorySize) or the output/logits buffer (see OutputBufferBytes); the full resident footprint of the session is MemorySize + DraftMemorySize + OutputBufferBytes.

OutputBufferBytes

Gets the size, in bytes, of the context's output/logits buffer, allocated apart from the KV-cache and compute buffers and therefore not counted in MemorySize. This buffer scales with the model's vocabulary size and can be a substantial per-context allocation for large-vocabulary models. Returns 0 when the context is hibernated, or when the running native backend predates the query export (older redistributable binaries), in which case the bytes fall into the dashboard's unattributed bucket rather than being mis-reported.

ProjectedMemorySize

Gets the estimated bytes the context will occupy on its device once it is resident, cache and compute buffers included. Reported only while the context holds nothing (Residency is NotCreated or Hibernated); a resident context reports 0 here and its real cost in MemorySize.

QueuedRuns

Gets the number of requests admitted to the sharing parallel decoder and waiting for a free slot, or 0 when SlotCount is 0. A standing queue beside fully busy slots is the signal that the context is slot-bound.

Residency

Gets the residency of the context: whether it is live in memory, hibernated to disk, or not yet created.

SlotContextSize

Gets the token window (prompt plus completion) one slot of the sharing parallel decoder reserves for a single request, or 0 when SlotCount is 0. This, not ContextLength, is the ceiling an individual request on this context can occupy.

SlotCount

Gets the number of requests the parallel decoder sharing this context can generate for simultaneously, or 0 when the context serves a single session on the classic path. A value greater than zero identifies this as a decoder's shared context: one allocation serving many concurrent requests, which is why its ContextLength is a pool rather than one session's window.

TrackedSessions

Gets the number of conversations the sharing parallel decoder is tracking on this context, those holding a sequence and those whose state has been moved off-device together, or 0 when SlotCount is 0. Exceeding SlotCount is expected: the resident set is bounded by the pool, the number of live conversations it serves is not.