Prompting Local Models

A prompt polished against a frontier hosted model often produces visibly worse output on a local model, and the first-week conclusion "local models are not good enough" is usually this mismatch, not the model. This page covers what to change: the prompt patterns that work at 1B-30B scale, the generation parameters the served APIs accept, and the point where prompting stops being the right tool.

1Why the same prompt reads worse here#

A frontier hosted model has enough capacity to infer what you meant: it unpacks a vague mega-prompt, resolves implied constraints, and self-corrects mid-answer. A compact local model spends its capacity following what you actually wrote. The practical consequence is one rule: everything a frontier model would infer, a local model must be told, and everything it must be told competes for its attention. Prompts get shorter and more literal at the same time, not longer. The model is not broken; the prompt was written for a different reader.

2The patterns that work at this scale#

• Shorten the system prompt to what changes behavior. Persona essays, redundant safety clauses, and page-long style guides dilute the instructions that matter. A compact model follows five literal rules better than thirty aspirational ones.
• One task per request. "Classify this ticket, extract the customer name, and draft a reply" degrades all three. Split it: three small requests are cheaper to run locally than one bad answer is to debug, and each can use the settings its task wants.
• Show, do not describe. Where a frontier model needed zero examples, give a local model one to three: a real input and the exact output you want. Few-shot examples are the highest leverage edit on small models, and they double as the seed of a future training dataset.
• State the output before the content. Put the instruction and the expected shape first, then the material to work on. Instructions buried after three pages of pasted text are the ones small models miss.
• Say what to do, not what to avoid. "Answer in one sentence" lands more reliably than "do not be verbose". Negated instructions are the first casualties at small scale.

Iterate in the Playground: it is the same engine and the same models as the API, so a prompt proven there behaves the same in code. Prompting outside English has its own layer (which language the system prompt should speak, terminology in the target language): Working Across Languages.

3Output shape is a contract, not a request#

The largest single upgrade over cloud-era habit: stop pleading for JSON in prose. This server enforces output shape by grammar-constrained decoding, so a declared schema cannot be violated, which retires the "respond ONLY with valid JSON" paragraph and every retry-and-reparse loop built on it. Declare response_format with json_schema and spend the freed prompt budget on what the fields mean. The mechanism and its limits are Structured Outputs.

A worked before and after, on ticket triage. The cloud-style version:

You are an expert support analyst with deep empathy. Read the ticket, think about the
customer's intent, categorize it, and respond ONLY with valid JSON like
{"category": ..., "urgency": ...}. Do not add commentary. Do not use markdown.

The local version: a literal instruction, one example, and the shape moved out of the prose into an enforced schema:

System: Classify the support ticket. Categories: billing, bug, how-to, account.
Urgency: low, normal, high.

User: Ticket: "I was charged twice this month, please fix this today."
Assistant: {"category": "billing", "urgency": "high"}

User: Ticket: "<the ticket>"

sent with response_format: {"type": "json_schema", ...} and temperature 0. The JSON is now guaranteed by the decoder, the example shows the judgment you want, and nothing in the prompt is decoration.

4The generation knobs#

The OpenAI-compatible surface (POST /v1/chat/completions) accepts the standard sampling fields; the full request shape is in the API reference:

The Anthropic dialect carries its native temperature, top_p, max_tokens, and stop_sequences; the Ollama dialect additionally maps top_k, min_p, and repeat_penalty from its options block (API Compatibility).

Starting points by task type, then measure on your own traffic:

• Extraction, classification, function calling (Function Calling): temperature 0. These tasks have a right answer; randomness only adds variance.
• Summarization, translation, grounded answers: low temperature, around 0.2 to 0.4. Enough variation for natural prose, not enough to invent.
• Rewriting, drafting, brainstorming: 0.7 and up, and consider top_p below 1 to trim the incoherent tail.

5Seeds and the honest limits of reproducibility#

An unseeded request draws fresh randomness on every run, so regenerating an answer produces a different answer. An explicit seed pins the request's own sampling stream, which is what comparable evaluation runs need: same prompt, same settings, same seed. Be honest about the ceiling: the server decodes concurrent requests together, and what shares a batch can shift the arithmetic, so a seeded request is not guaranteed to replay bit-identically under load (see Inference Capacity for how serving is scheduled). For comparisons, fixed seeds plus a labeled sample and one changed variable is the protocol, and it is written down in Measuring What Matters.

6When to stop prompting#

Prompting has a ceiling, and this server ships the levers that sit above it:

• The task-specific endpoints. Classification, entity extraction, PII detection, summarization, and translation run purpose-built pipelines that outperform a hand-rolled chat prompt for the same job: your labels become request parameters instead of prompt prose (Text Analysis, Document Classification, Structured Extraction).
• Retrieval, when the gap is knowledge. No prompt makes a model know your documents; indexing them and asking with citations does (Grounded Answers).
• Fine-tuning, when the gap is behavior. A format, tone, or domain-labeling requirement you keep re-prompting is a training signal: your corrected outputs become the dataset (Fine-tuning).
• A different model, when the task is beyond the current one. Escalate size, or switch capability, from the catalog in Models (The Right Model for Your Machine), and let your labeled sample decide rather than a hunch.

The tell that you have hit the ceiling: accuracy stops responding to prompt edits on your sample. At that point more prompt text is cost, not signal.

7Stated plainly#

• Local models follow what you wrote, not what you meant: shorter, more literal prompts, one task per request, examples where a frontier model needed none.
• Never plead for JSON: declare a schema and the decoder enforces it.
• Temperature 0 for tasks with a right answer; seeds pin randomness for comparisons, with bit-exact replay best-effort under concurrent decoding.
• When prompt edits stop moving your measured accuracy, the next lever is an endpoint, retrieval, or fine-tuning, not a longer prompt.