The Right Model for Your Machine
You are at the model-pull step with a specific machine and a catalog of choices. This page turns that into a five-minute decision: honest starting picks per memory tier and per task, a working explanation of quantization, and the protocol for models beyond the catalog. The catalog is the living source; these picks are starting points to measure, not rankings, and the catalog will move faster than this page. The method outlives every pick.
1The one-paragraph method#
A model's file size, listed beside every catalog entry, is roughly what it needs in device memory before any conversation exists. Context adds working memory on top: plan for the file size plus a couple of gigabytes for a single chat, more for long documents or concurrent users (the full arithmetic is Hardware Sizing, and the Inference panel prices any exact shape against your actual device). So: find your memory tier below, pull the pick that matches your task, talk to it in the Playground, and only then optimize. A model slightly too small answers fast and can be upgraded; a model too big for the device is unusably slow, which is how first sessions die.
2Quantization, plainly#
Every catalog chat model ships quantized, almost always at Q4_K_M: weights stored at roughly 4.5 to 5 bits each instead of 16. That is why a 9B-parameter model is a 6 GB file rather than a 19 GB one. Reading a file size into a memory need is therefore direct: the file IS the weight footprint, whatever the precision inside it.
- What Q4_K_M keeps: for chat, summarization, and document work, the quality loss against full precision is small enough that Q4_K_M is the catalog's default trade.
- What Q8_0 buys: about half the compression, so roughly double the file. The gap shows, when it shows at all, on precision-sensitive work (code, math, strict extraction). If you import a Q8_0 build of a model, judge the difference on your own tasks (Measuring What Matters), not on principle.
- When to drop a size tier instead of a quant level: below 4-bit (Q3, Q2 imports), degradation grows faster than the memory shrinks. When a Q4_K_M file does not fit, step down to the next model size at Q4_K_M rather than crushing the same model harder. Verify on your sample; this rule of thumb is where to start, not where to stop.
3Starting picks by memory tier#
Sizes below are approximate download sizes, which double as the weight footprint. Leave headroom for context on top of them. All chat picks here are current-generation models with vision unless noted.
Around 8 GB of device memory (entry GPUs, small unified-memory machines):
| Model | Size | Why this one |
|---|---|---|
qwen3.5:4b |
2.9 GB | The default first pull: chat, vision, tools, and OCR in one compact model |
gemma4:e2b |
3.1 GB | Mixture-of-Experts with about 2B active parameters, so it decodes fast for its size |
gemma4:e4b |
4.8 GB | The stronger Gemma 4 compact, still with room for context on 8 GB |
qwen3.5:9b |
6.0 GB | Fits with a modest context window; the quality step up when 8 GB is all you have |
12 to 16 GB (mainstream GPUs, 16 GB Apple machines):
| Model | Size | Why this one |
|---|---|---|
qwen3.5:9b |
6.0 GB | Strong general model with generous room for context and concurrency |
glm-4.6v-flash |
6.8 GB | Vision-language with strong OCR and screenshot understanding, plus function calling |
gemma4:12b |
7.3 GB | Dense 12B with long context; a solid default chat slot |
gptoss:20b |
12.1 GB | Reasoning and tool use with about 3.6B active parameters; text only, needs the 16 GB end |
24 GB and up (workstation GPUs, 32 GB+ unified memory, multi-GPU):
| Model | Size | Why this one |
|---|---|---|
gemma4:26b-a4b |
15.4 GB | MoE flagship economics: 26B quality at roughly 4B active decode cost |
qwen3.8:27b |
17.4 GB | The dense flagship: vision, long-horizon agent work, adjustable reasoning effort |
glm4.7-flash |
18.1 GB | MoE built for agentic tasks, reasoning, coding, and math; text only |
qwen3-coder:30b-a3b |
18.6 GB | The coding specialist, text only; see the task table below |
gemma4:31b |
19.0 GB | Dense 31B with long context, the largest dense Gemma |
muse-glimmer:30b |
20.4 GB | Vision flagship for image and video-frame understanding; tight on 24 GB, size the window down |
Apple unified memory: the same files, served through Metal. Memory is shared with the OS and your applications, so budget below the machine's total: a 16 GB Mac lives in the 8 GB tier comfortably and reaches into the middle one, 32 GB runs the middle tier with headroom, 64 GB and up runs everything in the catalog. Backend details are in GPUs and Backends.
CPU-only: real, for the right sizes. Interactive chat is pleasant with qwen3.5:0.8b
(0.6 GB), qwen3.5:2b (1.5 GB), qwen3.5:4b, and gemma4:e2b; the MoE models are the
CPU-friendly surprise, since decode cost tracks active parameters rather than file size.
Embeddings, reranking, OCR, and speech (all below) run well on CPU, which makes a GPU-less box
a capable document and search machine even when large-model chat is not its job.
4Starting picks by task#
Chat is not the only slot to fill. The server assigns a default model per capability (Choosing and Operating Models); these are the starting points per task:
| Task | Start with | Also consider |
|---|---|---|
| Chat and general work | The tier pick above | The next tier up when answers feel shallow; before switching, tune how you prompt it (Prompting Local Models) |
| Coding | qwen3-coder:30b-a3b (18.6 GB, agentic coding, long context) |
qwen3.5:9b, gemma4:12b, or glm4.7-flash where memory or breadth matters |
| Vision | Your chat pick already sees images (qwen3.5 and gemma4 families, qwen3.8:27b) |
glm-4.6v-flash for documents and screenshots, muse-glimmer:30b as the large-tier flagship |
| Document OCR | glm-ocr (0.8 GB) or paddleocr-vl-1.6:0.9b (0.8 GB), both specialists |
infinity-parser2-flash (1.4 GB) for low-latency parsing of tables, charts, and formulas |
| Text embeddings | embeddinggemma-300m (0.2 GB) or harrier-oss:0.6b (0.4 GB) |
bge-m3 (0.4 GB) for long multilingual inputs; qwen3-embedding:0.6b, :4b, :8b as the quality ladder |
| Reranking | bge-m3-reranker (0.4 GB) |
Sizing guidance in Embeddings and Reranking |
| Speech to text | whisper-large-turbo3 (0.9 GB) |
whisper-small (0.3 GB) or whisper-tiny (44 MB) on constrained CPU boxes; details in Transcription |
The specialists are deliberately small: an OCR model, an embedder, and a speech model together cost less memory than one mid-size chat model, so they coexist with your chat pick rather than competing with it. Embedding and reranking load in their own roles and never evict the chat model.
5Beyond the catalog: importing GGUF#
The catalog is curated, not exhaustive. The wider world lives on Hugging Face: search the model name plus "GGUF", open the repository's file list, and pick the Q4_K_M file (the quant naming in those lists is exactly what section 2 explained). Any GGUF file imports from the Models section by local path or URL, so that file's URL pastes straight into the import; the import probes the file first and refuses what cannot serve, rather than failing later on a request. The importer owns what the catalog otherwise guarantees, so run this protocol before an import touches real traffic:
- Check the license. Catalog entries carry known licenses; an imported file's terms are your responsibility to read (Licensing).
- Prefer the source's own GGUF builds or a converter you trust; quantization quality varies between publishers of the same model.
- Size it like a catalog model: the file size is the weight floor, same arithmetic as section 1.
- Sanity-check before rollout. Run your labeled sample against it and against the catalog model it would replace, one variable changed, fixed seed (Measuring What Matters). An import that cannot beat the incumbent on your traffic is a curiosity, not an upgrade.
Already running a GGUF library under another local tool? The import path accepts it as-is: Coming from Ollama is the migration map.
6Stated plainly#
- File size is the memory floor; add context headroom, and let the Inference panel price the exact shape.
- Q4_K_M is the catalog's default trade; below 4-bit, drop a model size instead of a quant level, and verify either move on your own sample.
- Pull the tier pick, talk to it in the Playground, then optimize; too small is recoverable, too big kills the first session.
- Specialists (OCR, embeddings, speech) are small enough to run beside your chat model, not instead of it.
- These picks age; the catalog is the living source, and the method here is what to reapply when it moves.