Method GetToken

Namespace

LMKit.Tokenization

Assembly

LM-Kit.NET.dll

GetToken(string)

Retrieves the token value for a specific chunk of text.

public int GetToken(string value)

Parameters

value string

The string token to search for within the vocabulary list.

Returns

int

The zero-based index of the token if found; otherwise, -1 if the token is not found in the list.