Method TokenizeForEmbedding

Namespace

LMKit.Abstractions

Assembly

LM-Kit.NET.dll

TokenizeForEmbedding(string)

Tokenizes a passage exactly as the text embedding path does internally.

int[] TokenizeForEmbedding(string text)

Parameters

text string

The passage text to tokenize. Cannot be null or empty.

Returns

int[]

The token sequence the model consumes for text.