Chat with Images
Vision-capable models read images inside ordinary conversations: screenshots, photos, scans, charts, whiteboards. This server serves image input on every chat dialect, locally, which matters precisely for the images people hesitate to upload anywhere: ID documents, contracts, medical forms, internal dashboards. This guide covers sending images, the one lever that governs their cost, and the boundary where chat stops being the right tool.
1Sending images, per dialect#
| Dialect | The shape |
|---|---|
OpenAI (/v1/chat/completions) |
image_url content parts: a data URL or an http(s) URL, with an optional detail |
Anthropic (/v1/messages) |
image content blocks (base64 source), alongside text blocks |
Responses (/v1/responses) |
input_image items: a data URL, an http(s) URL, or an uploaded file_id (Responses and Vector Stores) |
| Claude Desktop gateway | Attach in the app; on a text-only model the image degrades to a visible notice instead of failing the turn (Claude Desktop) |
Multiple images per message ride fine; interleave them with text so the model knows what each one is for.
2The detail lever is the cost lever#
An image is encoded into vision tokens before the model reads it, and the pixel budget it gets decides both fidelity and cost: more detail means more tokens, more prefill time, and more context consumed. Two controls, deliberately layered:
- The server default: the Inference section's Default Image Detail setting governs every image whose request says nothing, so an operator prices vision once for the whole server.
- Per request: an explicit
detailon an OpenAIimage_urlpart overrides the default for that image; absent one, the server default holds.
The practical rule: reading dense text or small UI elements wants high detail; "what is in this photo" does not. When a whole workflow needs maximum-fidelity page reading, that is a sign you are in document territory, not chat.
3Choosing the model#
Vision is a capability, not a given: the catalog marks vision-capable models, and the default vision model slot decides which one serves when a request does not name one. Two honesty contracts to know:
- A model without vision weights refuses image input with the reason; nothing silently drops your image and answers from the text alone. (The one deliberate exception is the Claude Desktop gateway's visible-notice degradation, because failing a desktop conversation outright serves nobody.)
- Vision quality varies more than text quality across model sizes. Small vision models caption; larger ones read. Evaluate on YOUR images in the Playground before standardizing.
4When chat is the wrong tool#
Chat answers questions ABOUT an image. When you want the image's content AS DATA, the document surfaces are built for it and return evidence, not prose:
| You want | Use |
|---|---|
| Faithful text from a scan or photo of a document | OCR or document-to-Markdown |
| Fields out of a photographed form or invoice | Structured Extraction: typed values with confidence and coordinates |
| To find sensitive content in images and remove it | Redaction via the detection pipeline |
| Image vectors for similarity search in your own store | Embeddings and Reranking: images embed natively |
The tell: if the answer should be checkable against the pixels (a value, a quote, a box on a page), you want a document endpoint; if the answer is judgment or conversation, you want vision chat.
5Stated plainly#
- Images ride every dialect in that dialect's native shape, processed locally end to end.
- One detail lever prices vision: a server-wide default, overridable per image where the dialect carries it.
- Vision-less models refuse honestly rather than ignoring your image; pick from the catalog's vision-capable set and evaluate on your own material.
- Questions about images are chat; data out of images is the document pipeline.