Method Decode

Namespace

LMKit.Tokenization

Assembly

LM-Kit.NET.dll

Decode(IEnumerable<int>)

Decodes a sequence of token identifiers into a human-readable string. Special tokens (beginning and end of sequence) are skipped.

public string Decode(IEnumerable<int> tokens)

Parameters

tokens IEnumerable<int>

A collection of token identifiers.

Returns

string

The decoded string.

Examples

using LMKit.Model;
using System;

LM model = LM.LoadFromModelID("llama-3.2-1b");

// Tokenize then decode
int[] tokens = model.Vocabulary.Tokenize("Hello, world!");
string decoded = model.Vocabulary.Decode(tokens);

Console.WriteLine($"Decoded: {decoded}");