When to Run AI Locally

Whether to self-host AI is an architecture decision, and most pages about it are written by someone selling one side. This one is written for the CTO or lead who has to sign the memo: where local inference wins for structural reasons, where hosted APIs genuinely remain the better call, and how to replace opinion with two weeks of your own traffic.

1The shape of the decision#

"Local or hosted" is rarely one decision. It is one decision per workload, because the deciding factors (data sensitivity, volume shape, latency budget, quality ceiling, operational appetite) vary per workload, not per company. The most common correct answer is a split, and because this server clones the hosted API surfaces (API Compatibility), the split costs a base URL per client, not a rewrite (Cloud-to-Local Migration). Evaluate workloads one at a time rather than picking a side for the whole organization.

2Where local wins structurally#

These advantages are properties of the architecture, not of any vendor's current pricing page.

• Data that cannot leave. On a default install, nothing leaves the machine: prompts, documents, extractions, and indexes are processed and stored locally, and the complete list of opt-in egress channels is short enough to audit in one page (Where Data Lives). "The provider sees nothing" stops being a contractual promise and becomes a network fact your firewall can enforce. For regulated data, client-confidential material, and residency requirements, this criterion often decides on its own; the procurement companion that maps it onto reviewer vocabulary is Passing the Security Review.
• Predictable volume economics. Hardware is an amortized fixed cost; metered tokens scale with success. Steady, high-volume workloads (document pipelines, embeddings, classification at scale, a team's daily chat) cross the break-even line, and spiky low ones do not. The crossover is arithmetic, not ideology: The Cost Math works it through, and the pilot below feeds it your real numbers.
• Latency locality. Requests never cross a WAN, never sit in a shared provider queue, and never slow down because someone else's product launched. Tail latency becomes a function of your hardware and your capacity settings, which you can measure, size, and fix.
• Offline and air-gapped operation. A deployment with zero egress is a supported configuration, not a workaround (Air-Gapped Deployment). Hosted APIs have no equivalent at any price.
• Version control over behavior. A local model is a file you pin. It does not change under a deployed prompt, get deprecated on a provider's schedule, or drift between the day you validated it and the day it matters. Upgrades happen when you decide, proven against your own labeled sample before and after (Measuring What Matters).

3Where hosted wins#

Conceding these plainly is the point of this page; a framework that ignores them is a pitch.

• The frontier reasoning ceiling. The largest hosted models outperform anything that fits on hardware you would buy for this, and the gap is real on the hardest open-ended work: novel multi-step reasoning, frontier-level code generation, tasks with no structure to exploit. If a workload needs the best model that exists this quarter, hosted serves it. Section 5 covers how much of the gap closes for structured work; the residue stays.
• Spiky, low total volume. Hardware that sits idle is pure cost. A workload that runs a few thousand requests a month, or bursts hard and then sleeps, is exactly what metered pricing is good at. Run the arithmetic before buying a GPU for it.
• No one to own a server. Self-hosting is deliberately low-ceremony here, but not zero-ops: someone patches the machine, backs up the state directory, sizes capacity, and reads the dashboard when something looks wrong. If no one will own that, hosted is the honest choice.
• Elastic scale on demand. A tenfold traffic spike on a hosted API is a bigger invoice; on your own hardware it queues by default, or sheds with an honest Retry-After if your saturation policy says reject, until you add capacity. Workloads with violent, unpredictable peaks favor someone else's fleet.

4The hybrid middle#

Most organizations that adopt local inference do not leave hosted APIs entirely; they reroute the workloads where local wins and keep the rest. The pattern that recurs:

• Local: everything that touches sensitive documents or PII (extraction, redaction, the IDP pipeline), search and grounded answers over internal corpora, embeddings, classification, and other high-volume structured work.
• Hosted: the occasional frontier task, low-volume experiments, and workloads still being shaped.

Because clients switch per base URL, the routing lives in configuration (the per-client table), and a workload moves between the columns when its volume or sensitivity changes. The split also caps exposure: the hosted provider sees only the traffic you deliberately send it.

5Closing the quality gap#

The frontier gap in section 3 is measured on raw open-ended chat. Production workloads are rarely that, and three levers close most of the distance for the work local is best at:

1. Task-specific endpoints instead of raw chat. Extraction, classification, entities, PII, and their siblings run purpose-built pipelines with validation and confidence reporting, not a chat prompt hoping for JSON (Structured Extraction, Text Analysis). A compact model inside a constrained pipeline holds its own where the same model free-forming would not.
2. Grounding. Answers cited from your own indexed corpus depend on retrieval quality more than on model scale, and refusing when the corpus does not support an answer is a feature (Grounded Answers). Structured output contracts are enforced by grammar, not requested by prompt (Structured Outputs).
3. Fine-tuning as the last mile. Once a task has volume and your corrected outputs have accumulated, fine-tuning specializes a local model on exactly your distribution, which is the one comparison hosted generalists lose.

A fourth lever costs nothing but attention: prompts tuned on frontier hosted models underperform on compact locals until they are rewritten for them, so a disappointing first impression is often prompt mismatch rather than a model verdict (Prompting Local Models).

What "good enough" means for your tasks is an empirical question with a cheap answer: a labeled sample of thirty to a hundred real items per task, the method in Measuring What Matters.

6The two-week pilot#

The decision memo should cite numbers from your traffic, not this page's prose. The pilot that produces them:

1. Days 1-2: stand it up. Install, pull candidate models from the catalog (the per-tier picks are The Right Model for Your Machine), and validate in the Playground. Keep it loopback-only; a pilot needs no exposure ceremony.
2. Days 2-4: build the sample. Label thirty to a hundred real items per workload, stratified by difficulty, with ground truth written as data (the protocol). The sample outlives the pilot.
3. Week 1: quality. Run the sample against candidate models and against your current hosted setup, same items, blind where judgment is subjective. This is the go or no-go number per workload.
4. Week 2: shape and cost. Point one real client at the server (the switch) and run its production traffic shape at real concurrency. Read latency and outcomes from the server's own request records, let the Inference panel price the intended slot shape against the pilot machine's actual device, extrapolate to purchase candidates with the method in Hardware Sizing, and feed the measured volume into the break-even arithmetic.
5. Decide per workload. Quality at or above the bar, acceptable latency at concurrency, and volume past break-even: that workload goes local. Anything failing a criterion stays hosted, with a number attached to what would change the answer.

7Stated plainly#

• Local wins structurally on data that cannot leave, steady volume, latency locality, offline operation, and version control over model behavior.
• Hosted wins on frontier reasoning, spiky low volume, zero ops capacity, and violent scale; pretending otherwise would make the rest of this page worthless.
• The usual right answer is a split, and the compatibility surfaces make the split a per-client base URL, reversible in minutes.
• Task pipelines, grounding, and fine-tuning close most of the quality gap on structured work; measure the rest on your own labeled sample.
• Two weeks and one pilot replace this entire page with your own numbers.