Hardware Sizing

"What hardware do we need" decomposes into three budgets and one honest method. This page gives you the mechanics: what actually consumes memory, which trade-offs move which number, and how to size from evidence instead of folklore. Its sibling, Inference Capacity, tunes a machine you already have; this one helps you choose or provision it. Pricing the machine against metered APIs is The Cost Math.

1The three budgets#

The shape of the mistake to avoid: buying for weights alone. A machine that fits the model with nothing left serves exactly one short conversation at a time; the state budget is where "it works" becomes "it works for the team".

2Reading a model's footprint#

• File size is the floor. A model needs roughly its file size in memory for weights before any conversation exists; the admin catalog shows the size beside every model, which makes shortlisting a reading exercise.
• Quantization trades memory for fidelity. The same model ships at several precisions with visibly different file sizes; smaller quantizations serve on smaller devices at some quality cost. Judge the trade on your tasks (Measuring What Matters), not on principle.
• Context is a purchase. Every token of window a slot offers must be storable; doubling the per-request window roughly doubles that slot's state cost. Size windows to the work (documents want more, chat wants less), not to the model's trained maximum.
• Vision costs context too: images encode into tokens before the model reads them, and the detail setting is the price dial.

3Sizing for concurrency#

The server decodes concurrent requests in a slot pool: slot count is true concurrency, and each slot carries its own context window. Three consequences for the buyer:

1. Memory for state scales linearly with both numbers. Ten slots of a large window on a big model is a serious state budget; the same ten slots with a right-sized window is often half of it. The knobs and their behavior are Inference Capacity's subject.
2. The admission gate fits, the saturation policy protects. A pool the device cannot hold is fitted down (slots first, then window) rather than refused, and load beyond the pool queues or sheds by YOUR policy; undersizing degrades politely instead of crashing, which buys you time to resize, not an excuse never to.
3. Memory levers stretch a given card: KV-cache quantization, session-state offload for idle conversations, and the image-detail default all trade something measurable for memory, and all live in the Inference settings.

4CPU, one GPU, several GPUs#

• CPU-only is real, for the right sizes. Everything serves on CPU (thread count defaults to the physical cores); compact models and background workloads (extraction, embeddings, classification) can live there happily. Interactive chat on large models is where CPU latency stops being acceptable; judge with your own prompts.
• One GPU is the sweet spot when its memory fits weights plus your state budget with headroom; the machine's job is mostly to feed that card.
• Several GPUs pool memory. Distributed inference spreads each model's layers across devices (the global setting), which is how models larger than any single card's memory become servable. A single response still generates at one-card speed, since the devices hold different layers and take turns; expect the interconnect, not the silicon, to set the ceiling. Standing state (the inference pool's KV cache and compute buffers) spreads the same way and is budgeted per card, so with unequal cards the smaller one decides the pool shape; pair cards of similar size when the pool shape matters.
• RAM and disk still matter: models load from disk (fast storage shortens restarts and model switches), host RAM backs offloaded session state, and the embedding pool runs beside chat with its own reservation.
• Backend setup and verification (drivers, CUDA versus Vulkan versus Metal, proving the model actually sits on the GPU) is GPUs and Backends.

5The method: price it, then prove it#

1. Shortlist models by file size against candidate devices, weights plus a state budget you estimate from section 3's shape; the per-tier starting picks are The Right Model for Your Machine.
2. Let the server price it: configure the intended slot shape and per-model overrides in the Inference settings; the panel prices the shape against the actual device before you commit to it, which converts guesswork into arithmetic on YOUR card.
3. Load-test the real shape: your prompts, your document sizes, at the concurrency you claimed, watching the dashboard for queue depth, latency, and memory; the steady-state discipline is in Measuring What Matters.
4. Buy the headroom you observed you needed, not a round number. Model upgrades usually arrive as bigger-or-better files; a card sized exactly for today has no room for next quarter's default.

6Stated plainly#

• Size three budgets: weights (the file size, listed per model), inference state (slots times window, the concurrency bill), and governed headroom.
• Quantization, window size, KV quantization, and image detail are the levers that fit a workload onto a card; each trades something you should measure.
• CPU serves the compact and the background; GPUs serve the interactive; several GPUs pool memory for models no single card holds.
• Do not estimate what the server will tell you: configure the shape, let the panel price it against the device, then load-test before buying.