Method TokenizeForEmbedding

Namespace

LMKit.Abstractions

Assembly

LM-Kit.NET.dll

TokenizeForEmbedding(string, EmbeddingRole)

Tokenizes a text exactly as the embedding path does internally, for the given retrieval role.

int[] TokenizeForEmbedding(string text, EmbeddingRole role = EmbeddingRole.None)

Parameters

text string

The text to tokenize. Cannot be null or empty.

role EmbeddingRole

Which side of a retrieval pair the text is. A model trained with asymmetric prefixes only places queries and documents in one comparable space when each side is tokenized in its own role, so an indexer that pre-tokenizes must pass Document rather than tokenize as an unattributed text. Defaults to None, which applies no prefix.

Returns

int[]

The token sequence the model consumes for text in that role.