Class EmbeddingPoolOptions

Namespace

LMKit.Global

Assembly

LM-Kit.NET.dll

Shape and allocation policy of the resident embedding pool: the single pinned context each embedding-capable model decodes its embedding and reranking requests on.

public sealed class EmbeddingPoolOptions

Inheritance

object

EmbeddingPoolOptions

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Embedding inference is one-shot: no state survives a call, so nothing is retained, recycled, or hibernated. The pool is allocated once at the configured shape and stays resident for the model's lifetime; requests coalesce into shared decode passes on it instead of each creating and releasing a context of its own. Its memory footprint is therefore known before the first request, exactly as for SharedSlotPoolOptions, and it is governed by the same per-device budget (InferenceMemoryRatio). Unlike the completion pool, it exists under EVERY InferenceScheduling strategy.

Two scopes exist. EmbeddingPool is the process-wide shape every model uses by default. Each model also exposes its own instance through EmbeddingPool, which INHERITS every value it does not state: reading a property there answers with the process-wide value until that property is overridden, and follows later changes to it. The *Override properties are the tri-state view of the same values, where null means "inherit"; assigning null to one restores inheritance. Changing a value marks resident pools stale; each is rebuilt at the new shape once its in-flight requests complete.

Constructors

EmbeddingPoolOptions()

Creates a standalone shape whose unstated values fall back to the SDK defaults. Governs nothing on its own: only the instances EmbeddingPool and EmbeddingPool expose drive live pools.

Properties

Allocation

When the pool claims its memory. Defaults to OnFirstCompletion, so a process that loads an embedding model and never embeds reserves nothing.

ContextSize

Token window of the pool: both the longest single input it embeds in one pass and the packing budget of one shared decode. Defaults to 4096. Values below 256 are raised to 256; the model's own trained window caps it at attach.

ContextSizeOverride

ContextSize as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.

MaxSequences

Most inputs one shared decode pass may carry. Defaults to 32, clamped to [1, 64].

MaxSequencesOverride

MaxSequences as stated by THIS instance, or null when it is inherited. Assigning null restores inheritance.