Property LookupDraftEqualization
LookupDraftEqualization
Chain-length rule for lookup-drafted passes that carry several slots. The default is ExpectedValue on every backend: what a verified row costs depends on the device (its kernels, and on CUDA, where the rows are decided on the device and the pass shape is kept fixed, the host link and the host scans), so only a rule that prices the rows from the pass costs measured on the device itself drafts where it pays. Drafting changes how fast a completion arrives, never what it emits. Read on every pass, so a change applies to the next pass without a rebuild. Reading answers with the inherited value while LookupDraftEqualizationOverride is null; assigning sets the override.
public SharedSlotPoolOptions.DraftEqualization LookupDraftEqualization { get; set; }Property Value
Remarks
On a model carrying recurrent state, every drafting slot in a pass must verify the same number of rows: the recurrent memory decodes unequal per-sequence lengths as separate micro-batches, one per sequence, which costs more than the drafts return. The rows that bring a short proposal up to the pass length are fillers: decoded, verified, rejected and rolled back, each costing a row of decode time for nothing. Attention-only models have no such constraint; on them only ExpectedValue changes anything, by capping the depth when rows are expensive.