The Cost Math

The build-versus-buy meeting needs one number: the monthly volume at which owning the hardware beats paying per token. This page gives you the arithmetic, worked for three workload shapes, and a method for reading your own break-even from the server's records after a pilot week. Every price is a variable you fill in from current quotes; prices move, the structure does not.

1The two sides of the ledger#

The hosted side scales with volume. Your monthly spend is token volume times price: with T_in and T_out your monthly input and output volume in millions of tokens, and P_in and P_out your provider's current per-million-token prices, the hosted bill is T_in x P_in + T_out x P_out. Look the prices up on the day you decide; they change, and any number printed here would rot.

The local side is (almost) fixed. The monthly cost of running this server is the same whether it decodes one million tokens or a hundred:

• Hardware amortization. Purchase price H spread over its service life of Y years: H / (12 x Y) per month. Three years is a defensible default for a GPU workstation; a card sized with headroom (see Hardware Sizing) survives model upgrades and stretches Y.
• Power. Average draw W watts, around the clock, at your electricity price E per kWh: (W x 730 x E) / 1000 per month. Use a measured average, not the PSU rating: an inference box idles far below its load draw unless it is saturated all day.
• Operator time. M hours per month at a loaded rate R: updates, monitoring, the occasional resize. Be honest about M; for a one-box deployment it is small but not zero, and for small teams it is often the largest line on the local side.
• The product license, on whatever terms apply to your deployment (see Licensing).

Call the sum L. The marginal cost of one more token on hardware you already own is a sliver of power, so the whole comparison collapses to one question: does your monthly hosted bill exceed L? The break-even volume is the T at which T x P equals L, and everything past it is margin in your favor.

2From seats to concurrency#

Hardware Sizing prices a machine from slot count and window, but a buyer starts from headcount. The conversion is the step that decides H, so do it explicitly:

1. Seats are not concurrency. An interactive user spends most of a session reading and typing; requests from different users rarely overlap. Expected concurrency is N x a x r x s / 3600: N seats, fraction a active in the peak hour, r requests per active user per hour, s seconds of decode per request.
2. Start with an assumption, replace it with a measurement. Absent your own data, one concurrent request per five to ten active seats is a common planning starting point for chat traffic. It is an assumption, not a fact about your team: the pilot in section 6 replaces it with the concurrency you actually observed.
3. The slot pool absorbs the error. Slot count is true concurrency and the default saturation policy queues the rest, so an underestimate stretches latency instead of dropping requests (Inference Capacity). Configure the shape in the Inference settings and let the panel price it against the actual device before you commit to buying it.

3Shape one: interactive team chat#

Chat volume per seat is usually modest, which makes this the shape where the fixed costs matter most. Gather four inputs:

Monthly volume is N x q x t x d tokens; divide by one million and price it at your provider's blended rate to get the hosted-equivalent bill, then compare with L. For a small team the token bill alone often does not clear L on arithmetic; what tips the decision is the part of the ledger with no per-token price: confidentiality, data residency, and offline operation. That trade has its own page, When to Run AI Locally.

4Shape two: batch document processing over a backlog#

A backlog job has a different structure: the hosted cost scales with the size of the backlog, the local cost scales with wall time. Document pipelines are input-heavy (the document dwarfs the extracted answer), so the input price dominates.

Hosted cost is B times the per-document token cost, paid once. Local cost is the power for the run plus the hardware's amortized share for the weeks it takes; if the box then serves other work, only that share belongs to the job. Batch work is also latency-tolerant: it can wait in the queue or run overnight when interactive traffic is quiet, so it rides hardware you bought for other reasons at close to zero incremental cost. Large backlogs are where local break-even arrives fastest, because a million documents at even a modest per-document token cost is a hosted invoice with a lot of digits.

5Shape three: always-on embeddings and search#

Embedding traffic is continuous and high-volume: the initial index, re-embedding on every content change, and a steady stream of query embeddings. Three inputs shape it:

On the hosted side, every one of those tokens is metered, and one decision you will eventually make (changing the embedding model) re-prices the entire corpus K at once. On this server, embeddings run on their own pinned pool beside chat (Inference Capacity), so indexing does not compete with interactive traffic and the marginal cost of the whole shape is power. Corpora are large and embedding models are small; this is usually the first shape to cross break-even.

6Read your own break-even from a pilot week#

Estimates get you to a shortlist; a pilot gets you a number worth putting in the memo. Run one week of real traffic through the server and read the ledger from its own records:

1. Count the tokens. The Requests page attributes every call to its key with route, model, and timing; the OpenAI-compatible surface returns standard usage token counts per request. Multiply the week's totals by 4.3 for a monthly figure, price it at your provider's current rates, and that is the hosted bill this workload would have generated.
2. Read the real concurrency. Queue depth and active inferences on the dashboard, and the continuous view in Telemetry (exportable to Prometheus or OTLP, see Observability), replace the seats heuristic from section 2 with your observed peak. Resize the slot shape to it.
3. Compare against L, honestly. Include the operator hours the pilot actually consumed; they are your best estimate of M. The measurement discipline (steady state, real concurrency, fixed workload shape) is in Measuring What Matters.

7Stay on hosted APIs if#

The arithmetic does not always land local, and pretending otherwise would make this page useless in a purchase memo. Stay on hosted APIs when volume is low and spiky (a fixed monthly L divided by few tokens is an expensive token), when the workload genuinely needs frontier-model quality that no local model on your shortlist reaches on your own labeled sample, when nobody will own the box (an unowned server is a liability, not a saving), or when the project is too short-lived to amortize anything. And the two are not exclusive: point the high-volume and confidential callers at this server and leave the frontier calls hosted; the mechanics of moving a caller are in Cloud-to-Local.

8Stated plainly#

• Hosted cost scales with tokens; local cost is a fixed monthly L (amortization, power, operator time, license). Break-even is the volume where the two lines cross.
• Convert seats to concurrency explicitly, size the slot shape from it, and let the admin panel price the shape against the actual device before buying.
• Batch backlogs and always-on embeddings cross break-even fastest; small-team chat usually decides on confidentiality, not on the token bill.
• Estimate to shortlist, then pilot for a week and read tokens, concurrency, and operator hours from the server's own records.