Table of Contents

🔢 What are logits in LLMs?


📄 TL;DR:
Logits are the raw, unnormalized predictions produced by a Large Language Model (LLM), representing the likelihood of a token being the next in a sequence based on the input context. In LM-Kit.NET, logits act as the input data for various sampling methods that determine how tokens are selected during text generation. Through the LogitBias and RepetitionPenalty classes, you can adjust and influence these logits to customize the output, improving the relevance, creativity, and appropriateness of the model's generated text.


🔢 What Are Logits?

Logits are the raw output values generated by a model before any probability transformation is applied. They represent the model's unnormalized predictions for every possible token at each step of text generation. Since these logits are not yet transformed into probabilities, they serve as the input data for the sampling methods that decide which token to select as the next word in the sequence.

The process of converting logits into probabilities involves applying a function like softmax, which normalizes the logits and makes them interpretable as probabilities. Higher logits correspond to higher confidence in the model that the token should be selected next, while lower logits suggest that the token is less likely to be chosen.


💡 Why Do Logits Matter?

Logits play a crucial role in controlling token selection during text generation. Since they act as the input for the sampling methods, the way logits are adjusted can directly influence how likely certain tokens are to be selected.

By adjusting the logits, you can:

  • Guide token selection: Increase or decrease the likelihood of certain tokens appearing in the output.
  • Enhance customization: Adapt the model's behavior to specific tasks or contexts by manipulating which tokens are prioritized or excluded.
  • Control randomness: Using different sampling strategies on the logits can introduce creativity or deterministic behavior in the model's outputs, depending on the application.

⚙️ How Are Logits Used in LLMs?

  1. Logits as Input Data for Sampling:
    After processing the input prompt, the LLM outputs logits for each token in its vocabulary. These logits serve as input data for the sampling methods that decide how tokens are selected for text generation. The sampling method takes these logits and applies a strategy (e.g., temperature scaling, greedy decoding) to convert them into probabilities and select the next token.

  2. Softmax Transformation:
    The logits are transformed into probabilities using a normalization function like softmax, which scales the logits into a probability distribution. The model then selects the token based on the chosen sampling method, which may favor tokens with higher logits.

  3. Token Sampling:
    Depending on the sampling strategy, the logits are either adjusted (e.g., by temperature scaling) or used as-is (in greedy sampling). These methods determine which token to select as the next in the sequence and can be manipulated to influence randomness or determinism in the text completion process.


🛠️ Influencing Logits in LM-Kit.NET

In LM-Kit.NET, you can influence the logits, and therefore the text generation process, using the LogitBias and RepetitionPenalty classes. These classes enable you to modify how logits are handled before they are passed into the sampling methods, effectively controlling which tokens are more or less likely to appear in the output.

LogitBias Class:

The LogitBias class allows you to increase or decrease the likelihood of specific tokens by adjusting their logits before they are processed by the sampling method. You can use LogitBias to guide the model toward or away from certain words or phrases, depending on the desired output.

Key Features of LogitBias:

  • Increase or Decrease Token Likelihood: Apply biases to certain tokens, either increasing their probability by raising their logits or decreasing it by lowering them. This is useful in scenarios where specific terms need to be encouraged or avoided.

  • Exclude Tokens: By significantly lowering the logits of specific tokens, you can ensure that those tokens are not selected during text generation.

  • Reset Biases: Reset the logit adjustments to default, allowing for flexible control over different tasks without permanently altering the model's behavior.

RepetitionPenalty Class:

The RepetitionPenalty class helps prevent repetitive tokens from being generated by adjusting the logits of tokens that have already appeared in the output. This ensures that the model produces more diverse and engaging text, especially in longer sequences.

Key Features of RepetitionPenalty:

  • Token Frequency Penalty: Lower the logits of tokens that have already been generated to discourage their repetition, helping maintain variety in the output.

  • Customization: Control the penalty’s intensity, allowing more or less repetition based on the specific task or creative goal.


🧩 Logits as Input for Sampling Methods

Logits are always the starting point for any token sampling method. Once the logit adjustments from LogitBias and RepetitionPenalty are applied, the modified logits serve as the input for various sampling techniques, which ultimately decide which token to select.

Common sampling methods that use logits as input include:

  1. Greedy Sampling:
    In greedy sampling, the token with the highest logit (and thus the highest probability after softmax) is selected. This method is deterministic and often leads to repetitive text but ensures the most "confident" tokens are chosen.

  2. Temperature Sampling:
    Temperature sampling adjusts the logits before softmax, making the model more or less random. A higher temperature results in more diverse outputs by spreading the probabilities more evenly, while a lower temperature makes the model more deterministic by focusing on the highest logits.

  3. Top-K Sampling:
    Top-k sampling limits token selection to only the top (k) tokens with the highest logits, ignoring the rest. This ensures that only the most relevant tokens are considered, reducing the chance of the model selecting less likely or irrelevant tokens.

  4. Nucleus Sampling (Top-P Sampling):
    Nucleus sampling selects tokens from a dynamically chosen subset of the vocabulary, where the cumulative probability reaches a certain threshold (p). This method helps balance the relevance and randomness of the generated text.


⚙️ Practical Applications of Logit Control

  1. Chatbots:
    In chatbot applications, LogitBias can be used to steer the conversation toward polite or contextually appropriate responses by increasing the logits of desired tokens. RepetitionPenalty can be applied to prevent repetitive or stale dialogue, ensuring that the conversation remains engaging and dynamic.

  2. Creative Writing:
    For creative tasks like storytelling or poetry generation, adjusting logits via temperature sampling can introduce more variety and creativity, while RepetitionPenalty ensures that the text remains diverse without redundant phrases.

  3. Content Moderation:
    LogitBias can be used to prevent the generation of inappropriate or undesirable language by significantly reducing the logits of certain words or phrases. This ensures that the model adheres to content guidelines and generates safe outputs.

  4. Domain-Specific Text Generation:
    By applying LogitBias, developers can increase the likelihood of domain-specific terminology, ensuring that the output is relevant to specialized fields like medicine, law, or technical writing.


📝 Summary

Logits are the raw output values from a model that serve as the input data for sampling methods in text generation. They play a crucial role in determining the likelihood of token selection. By adjusting these logits using LogitBias and RepetitionPenalty in LM-Kit.NET, developers can control and customize the model's output, influencing which tokens are selected and how often tokens repeat.

In LM-Kit.NET:

  • LogitBias allows you to increase or decrease the likelihood of specific tokens or prevent them from being generated altogether by modifying their logits.
  • RepetitionPenalty helps avoid repetitive text and encourages more varied and dynamic outputs.
  • Sampling methods use the modified logits as input to decide which tokens are selected, balancing relevance and randomness depending on the task.

Mastering logit control in combination with appropriate sampling strategies enables fine-tuned control over the generated text, improving the model's performance across a variety of tasks.