Namespace LMKit.Tokenization

Classes

Vocabulary

Provides advanced tokenization capabilities and manages the model's vocabulary. The vocabulary of a large language model is composed of tokens that may represent whole words or word fragments. This class lazily loads the vocabulary and offers methods for tokenization, decoding, and token healing.

Enums

VocabularyMode

Specifies the vocabulary modes used by different tokenizer models.