Method GetToken
- Namespace
- LMKit.Tokenization
- Assembly
- LM-Kit.NET.dll
GetToken(string, bool)
Returns the token id whose decoded value matches value
.
public int GetToken(string value, bool trim = false)
Parameters
value
stringThe decoded token text to look up. When
trim
istrue
, leading and trailing Unicode whitespace is ignored during comparison.trim
boolIf
true
, the lookup ignores leading/trailing whitespace on both the input and each vocabulary entry (using Trim() semantics). This is useful for tokens that may contain padding, but it is slower since it must trim each candidate. Whenfalse
(default), an exact, ordinal, case-sensitive comparison is used.
Returns
- int
The zero-based token id if found; otherwise
-1
.