Operations

Inference Capacity

"How many users can this box serve?" has a real answer here, because capacity is declared, not discovered: the server decodes a FIXED number of requests concurrently, each inside a FIXED context window, and everything else queues. This guide explains the slot model, what happens at saturation, and the levers in the Inference section that trade memory, speed, and capacity against each other.

1The slot model#

The engine allocates one inference pool per model, divided into slots that decode together:

Slot count is the concurrency: how many completions make progress at the same instant. Throughput scales with it, and so does the pool's memory.
Slot context size is ONE request's whole budget: prompt plus completion, in tokens. It is also the server's maximum context length, derived rather than configured separately, so a request the server admits is a request the pool can actually serve; nothing passes admission only to be refused by the engine.

The pool's shape is fixed up front, which is the operational point: memory footprint is known before the first request, and it does not drift under load. Sizing is a two-number decision: concurrent callers at peak, and the longest prompt-plus-answer you must honor. A box that serves short chat turns wants more slots with a smaller window; a box that digests long documents wants the opposite.

2Saturation is a policy, not a surprise#

When every slot is busy, the saturation policy decides:

Queue (the default): requests wait for a slot. Latency stretches under burst, nothing is refused. Right for interactive use and modest overcommit.
Reject: the server answers HTTP 503 with a Retry-After header immediately. Right when a load balancer or client-side retry owns the queueing decision and a slow answer is worse than an honest "busy".

Watch saturation where it shows: queue depth and active inferences on the dashboard, per-request timings in Requests, and the same measurements exported to your monitoring.

3Per-model overrides#

One shape rarely fits a mixed fleet: the long-context document model and the snappy chat model want different trade-offs. Model overrides in the Inference section give a specific model its own slot count and window, inheriting whatever they do not state. The panel prices each override against the device BEFORE it is saved, and a shape the hardware cannot hold is fitted down rather than refused, so an override is applied capacity, not a wish. Changes apply to the next request; no restart.

4Embeddings do not compete with chat#

Embedding and reranking requests run on their OWN pinned pool per model, sized by the embedding context size (the longest single input embedded in one pass; longer inputs are chunked and pooled) and a max-sequences ceiling (how many inputs coalesce into one decode). Concurrent embedding callers are packed together automatically, which is why bulk indexing in Search sustains throughput without stealing chat slots. The practical consequence: a heavy re-embed and an interactive chat session coexist by design, not by luck.

5The memory levers#

All in the Inference section, all trading something explicit:

Lever	Trades	Reach for it when
Inference memory ratio	How much device memory standing inference state (pools plus kept-warm contexts) may claim	The box shares the GPU with other workloads, or models get evicted too eagerly
KV cache quantization	Cache precision for cache size	You need a bigger window or more slots than the device holds at full precision
Speculative decoding drafts	A little memory for faster generation	A compatible draft model exists for your main model; free speed on long outputs
Session state offload	Idle conversations leave device memory and restore later instead of re-processing their history	Many long-lived conversations, few simultaneously active
Default image detail	Vision token cost per image	Vision traffic where throughput beats pixel-perfect reading, or the reverse

Model residency itself (what is loaded, load/unload/hibernate) lives in Models; this section governs how much the resident models may hold.

6A sizing method that works#

Fix the window first from your longest real prompt-plus-answer, not from the model's trained maximum: an oversized window spends memory on tokens nobody sends.
Set slots from peak concurrent callers, then confirm the shape fits: the panel prices it against the device.
Load-test with your real traffic shape and watch queue depth and per-request latency in Requests; the numbers, not intuition, say whether the next unit of memory goes to slots, window, or KV quantization.
Give outlier models an override instead of resizing the whole server around them.
When the tuned box still saturates, add boxes: identical nodes behind a load balancer scale the same traffic by replication, and the saturation gauge that told you so is the signal that drives the fleet (Scaling Out).

This method tunes the machine you have; choosing or provisioning the machine in the first place, budget by budget, is Hardware Sizing.

7Stated plainly#

Capacity is slots times window, declared up front: known memory, no drift, honest admission.
Saturation behavior is your choice: queue for interactive traffic, 503-with-Retry-After for fleets that manage their own retries.
Embeddings ride a separate pool, so indexing and chat scale independently; the memory levers are explicit trades, and the panel prices every shape before it applies.