Method Tokenize

Namespace

LMKit.Tokenization

Assembly

LM-Kit.NET.dll

Tokenize(string)

Splits a text into smaller units (or tokens) that can be processed by the model.

public int[] Tokenize(string text)

Parameters

text string

The text to tokenize.

Returns

int[]

An array of integers where each entry represents a token identifier.