Method TokenizeForEmbedding

Namespace

LMKit.Embeddings

Assembly

LM-Kit.NET.dll

TokenizeForEmbedding(string)

Tokenizes a passage/document string exactly as GetEmbeddingsAsync(string, CancellationToken) does internally, returning the token sequence that the embedding model consumes.

public int[] TokenizeForEmbedding(string text)

Parameters

text string

The passage text to tokenize. Cannot be null or empty.

Returns

int[]

The token sequence the embedding model consumes for text.

Remarks

Use this to pre-tokenize text off the inference path (e.g. on a worker thread), measure its token length to pack multiple passages into a single batched call, and then feed the resulting token lists to GetEmbeddingsAsync(IEnumerable<IList<int>>, CancellationToken). The vectors produced from these tokens are identical to those produced from the original text, but the tokenization cost is paid once and away from the embedding context, and batches can be sized by their true token count.

Exceptions

ArgumentNullException

Thrown when text is null or empty.