Property BatchInvariantDecoding
BatchInvariantDecoding
Whether the pool decodes batch-invariantly. Defaults to true. Every request's logits are then bit-identical whatever other requests share its decode passes, however many rows it contributes to a pass and wherever its cache cells sit, so identical greedy requests answer identically from one run to the next. Reading answers with the inherited value while BatchInvariantDecodingOverride is null; assigning sets the override.
public bool BatchInvariantDecoding { get; set; }Property Value
Remarks
Honoured on the CUDA backend, where every row's matrix products and attention then reduce in an order its own sequence alone decides: quantized products accumulate in one canonical order whatever the row count (a vector kernel reproduces the tiled kernel's per-element arithmetic at decode widths), and attention walks each sequence's own cells in position order with a fixed tile shape. Elsewhere the pool decodes as if it were off. Changing it rebuilds the model's pool once its in-flight requests finish.