Interface ICompletionAdmission
Admits the language-model completions a process runs, one completion at a time, so a host can bound and observe exactly the work that occupies a model: token generation, never the steps a task performs around it.
public interface ICompletionAdmission
Remarks
Install an implementation through CompletionAdmission. Under SharedSlotPool, every completion asks for admission once its model's pool is ready and right before the completion is submitted to it, and returns the lease when the completion ends, whatever its outcome. Model loading, OCR, page rendering, document conversion and pool creation happen outside the lease. A task that runs several completions (an extraction read in several windows, a tool loop) asks once per completion and holds nothing between them.
The lease is held while the completion waits for a free slot of the pool and while it decodes. PerRequestContext does not consult it.
Methods
- EnterAsync(LM, CancellationToken)
Waits until a completion on
modelmay run and returns the lease that ends its admission when disposed.