Property ConcurrentDeviceDecoding
ConcurrentDeviceDecoding
Whether per-request contexts decode under the SHARED device gate, so completions on DIFFERENT models overlap on one device instead of time-slicing at decode-call granularity. The exclusive gate serializes every device call a decode makes: the kernel launches of a text batch (its device work completes asynchronously and is awaited when the logits are read, outside the gate) and, in full, every synchronous operation such as an image encode, which holds the device for its whole duration; a vision pipeline is where the shared gate buys the most. Same-model overlap is allowed too, but two contexts of one model contend for the same weights and gain nothing over serial decoding, so a host enables this when it runs two models in a pipeline. Context creation and teardown, KV-cache edits and state transfers keep the exclusive lock, which drains in-flight shared decodes first. Pool and embedding engines always decode under the shared gate.
public static bool ConcurrentDeviceDecoding { get; set; }Property Value
- bool
Default is
false.
Remarks
Read when a context is handed to a request, so a change applies
to the next completion. The LMKIT_CONCURRENT_GPU=0
environment variable forces the exclusive gate everywhere,
whatever this says.