Interface ITokenizingEmbedder

Namespace

LMKit.Abstractions

Assembly

LM-Kit.NET.dll

Optional capability for providers that expose their tokenizer and accept pre-tokenized input, enabling token-count aware batching.

public interface ITokenizingEmbedder

Remarks

This is typically a local-only optimization: a model running in-process can tokenize text off the inference path, measure the true token length of each passage to pack batches up to the model's limit, and feed the resulting token lists back for embedding. Remote providers that accept only text do not implement this interface, and consumers fall back to the text-based batch on IEmbedder.

The vectors produced from these tokens are identical to those produced from the original text; only the tokenization cost is paid once and away from the embedding context.

Methods

GetEmbeddingsAsync(IEnumerable<IList<int>>, CancellationToken)

Asynchronously generates embedding vectors for a batch of pre-tokenized passages.

GetEmbeddingsAsync(IList<int>, CancellationToken)

Asynchronously generates the embedding vector for a pre-tokenized passage.

TokenizeForEmbedding(string)

Tokenizes a passage exactly as the text embedding path does internally.