Enum InputLengthOverflowPolicy

Namespace

LMKit.Inference

Assembly

LM-Kit.NET.dll

Defines strategies to adopt when the prompt length exceeds the context size.

public enum InputLengthOverflowPolicy

Fields

TrimAuto = 0

Automatically trims the input to fit within the context size. The method of trimming is determined by the system.

TrimEnd = 1

Trim tokens from the end of the input. This strategy removes the latest tokens in the input sequence until the length is within the acceptable limit.

TrimStart = 2

Trim tokens from the start of the input. This strategy removes the earliest tokens in the input sequence until the length is within the acceptable limit.

KVCacheShifting = 3

Shift key-value cache to accommodate the overflow. This strategy adjusts the key-value cache to manage the excess tokens without directly trimming the input, potentially allowing more context to be preserved.

Throw = 4

Throw aNotEnoughContextSizeException exception when the prompt length exceeds the context size. This strategy halts processing and signals an error condition, requiring the input to be adjusted manually to fit within the context size.