Table of Contents

Method GetToken

Namespace
LMKit.Tokenization
Assembly
LM-Kit.NET.dll

GetToken(string, bool)

Returns the token id whose decoded value matches value.

public int GetToken(string value, bool trim = false)

Parameters

value string

The decoded token text to look up. When trim is true, leading and trailing Unicode whitespace is ignored during comparison.

trim bool

If true, the lookup ignores leading/trailing whitespace on both the input and each vocabulary entry (using Trim() semantics). This is useful for tokens that may contain padding, but it is slower since it must trim each candidate. When false (default), an exact, ordinal, case-sensitive comparison is used.

Returns

int

The zero-based token id if found; otherwise -1.