Namespace LMKit.Tokenization

Classes

Vocabulary

Handles model's vocabulary while providing advanced tokenization capabilities.
The vocabulary of a large language model differs from that of a human.
While a human's vocabulary is made up of complete words, the vocabulary of a language model is formed by "tokens."
These tokens can represent entire words, but at times, they may also be fragments of words.

Enums

VocabularyMode

Specifies the vocabulary modes used by different tokenizer models.