Method Decode

Namespace

LMKit.Tokenization

Assembly

LM-Kit.NET.dll

Decode(IEnumerable<int>)

Decodes a sequence of token identifiers into a human-readable string. Special tokens (beginning and end of sequence) are skipped.

public string Decode(IEnumerable<int> tokens)

Parameters

tokens IEnumerable<int>

A collection of token identifiers.

Returns

string

The decoded string.

Examples

using LMKit.Model;
using System;

LM model = LM.LoadFromModelID("llama-3.2-1b");

// Tokenize then decode
int[] tokens = model.Vocabulary.Tokenize("Hello, world!");
string decoded = model.Vocabulary.Decode(tokens);

Console.WriteLine($"Decoded: {decoded}");

Decode(IEnumerable<int>, bool, int)

Converts a sequence of token IDs back into text, choosing whether special tokens (chat-template markers, control tokens) appear in the output. Decode(IEnumerable<int>) strips them, which reads naturally; keeping them reproduces the exact stream a model decoded, which is what diagnostic transcripts need.

public string Decode(IEnumerable<int> tokens, bool skipSpecial, int capacity = 0)

Parameters

tokens IEnumerable<int>

The token IDs to decode.

skipSpecial bool

true removes special tokens from the output; false keeps them.

capacity int

Optional initial capacity hint for the decoded buffer.

Returns

string

The decoded text.