Hardware Sizing

"What hardware do we need" decomposes into three budgets and one honest method. This page gives you the mechanics: what actually consumes memory, which trade-offs move which number, and how to size from evidence instead of folklore. Its sibling, Inference Capacity, tunes a machine you already have; this one helps you choose or provision it. Pricing the machine against metered APIs is The Cost Math.

1The three budgets#

The shape of the mistake to avoid: buying for weights alone. A machine that fits the model with nothing left serves exactly one short conversation at a time; the state budget is where "it works" becomes "it works for the team".

2Reading a model's footprint#

• File size is the floor. A model needs roughly its file size in memory for weights before any conversation exists; the admin catalog shows the size beside every model, which makes shortlisting a reading exercise.
• Quantization trades memory for fidelity. The same model ships at several precisions with visibly different file sizes; smaller quantizations serve on smaller devices at some quality cost. Judge the trade on your tasks (Measuring What Matters), not on principle.
• Context is a purchase. Every token of window a slot offers must be storable; doubling the per-request window roughly doubles that slot's state cost. Size windows to the work (documents want more, chat wants less), not to the model's trained maximum.
• Vision costs context too: images encode into tokens before the model reads them, and the detail setting is the price dial.

3Sizing for concurrency#

The server decodes concurrent requests in a slot pool: slot count is true concurrency, and each slot carries its own context window. Three consequences for the buyer:

1. Memory for state scales linearly with both numbers. Ten slots of a large window on a big model is a serious state budget; the same ten slots with a right-sized window is often half of it. The knobs and their behavior are Inference Capacity's subject.
2. The admission gate fits, the saturation policy protects. A pool the device cannot hold is fitted down (slots first, then window) rather than refused, and load beyond the pool queues or sheds by YOUR policy; undersizing degrades politely instead of crashing, which buys you time to resize, not an excuse never to.
3. Memory levers stretch a given card: KV-cache quantization, session-state offload for idle conversations, and the image-detail default all trade something measurable for memory, and all live in the Inference settings.

4CPU, one GPU, several GPUs#

• CPU-only is real, for the right sizes. Everything serves on CPU (thread count defaults to the physical cores); compact models and background workloads (extraction, embeddings, classification) can live there happily. Interactive chat on large models is where CPU latency stops being acceptable; judge with your own prompts.
• One GPU is the sweet spot when its memory fits weights plus your state budget with headroom; the machine's job is mostly to feed that card.
• Several GPUs pool memory. Distributed inference splits tensor computation across devices (the global setting), which is how models larger than any single card's memory become servable; expect the interconnect, not the silicon, to set the ceiling.
• RAM and disk still matter: models load from disk (fast storage shortens restarts and model switches), host RAM backs offloaded session state, and the embedding pool runs beside chat with its own reservation.
• Backend setup and verification (drivers, CUDA versus Vulkan versus Metal, proving the model actually sits on the GPU) is GPUs and Backends.

5The method: price it, then prove it#

1. Shortlist models by file size against candidate devices, weights plus a state budget you estimate from section 3's shape; the per-tier starting picks are The Right Model for Your Machine.
2. Let the server price it: configure the intended slot shape and per-model overrides in the Inference settings; the panel prices the shape against the actual device before you commit to it, which converts guesswork into arithmetic on YOUR card.
3. Load-test the real shape: your prompts, your document sizes, at the concurrency you claimed, watching the dashboard for queue depth, latency, and memory; the steady-state discipline is in Measuring What Matters.
4. Buy the headroom you observed you needed, not a round number. Model upgrades usually arrive as bigger-or-better files; a card sized exactly for today has no room for next quarter's default.

6Stated plainly#

• Size three budgets: weights (the file size, listed per model), inference state (slots times window, the concurrency bill), and governed headroom.
• Quantization, window size, KV quantization, and image detail are the levers that fit a workload onto a card; each trades something you should measure.
• CPU serves the compact and the background; GPUs serve the interactive; several GPUs pool memory for models no single card holds.
• Do not estimate what the server will tell you: configure the shape, let the panel price it against the device, then load-test before buying.