Home Assistant

How to serve Home Assistant's Assist conversation agent from this server: which stage of the voice pipeline this replaces, the two supported network topologies and why authentication decides between them, and how to pick models that hold up in a voice round trip. Written for a smart-home installation where keeping everything local is the point, not a compromise.

1Where this server fits#

Home Assistant's Assist pipeline has distinct stages: wake word, speech to text, the conversation agent, and text to speech. This server replaces exactly one of them, the conversation agent, the stage that reads the transcribed request and decides what to say or do.

• The conversation agent runs here. Home Assistant's Ollama integration points at this server's Ollama dialect; the model, the prompt, and the reply never leave your network.
• Speech to text and text to speech stay on Home Assistant's side. Those stages speak the Wyoming protocol, which this server does not serve. Keep your existing local Wyoming add-ons for them; this guide covers the language model stage only.
• Controlling the home requires tool calling. When Assist is allowed to control entities, Home Assistant sends the model tool definitions. The Ollama dialect supports tool calling and reports the tools capability through /api/show, so capability-gated options in Home Assistant resolve correctly, provided the model you pick actually handles function calls well (see section 4).

2Authentication decides the topology#

The deciding constraint is not the integration, it is this server's network posture: keyless calls are accepted only from loopback while the anonymous toggle is on, and a network-reachable server refuses every call without an API key, on the Ollama dialect like everywhere else. That yields two supported topologies:

Same machine is the zero-friction default. Point the integration at http://127.0.0.1:PORT and it works with no credential ceremony. One caveat for containerized Home Assistant: the call must genuinely arrive over loopback. A container on a bridge network reaches the host through the bridge address, which is not loopback, so the keyless path does not apply; run the container with host networking or use the keyed path below.

Across the LAN needs a key-capable client. Mint a key in Access (a Read-only key is not enough here, since chat is a POST; use a full-scope key with no Search access). Whether Home Assistant's own Ollama integration can attach that key depends on your release: if its configuration offers no API key or authorization field, that integration can only reach this server where keyless calls are permitted, which is loopback. In that case, use an OpenAI-compatible integration instead (section 3), which takes a key by design.

The server listens on both an HTTP and an HTTPS port (shown in the Network section). For LAN traffic, use plain HTTP inside a network you trust, or HTTPS once the machine running Home Assistant trusts the server's certificate; a self-signed certificate it does not trust will fail the integration's connection check.

3The two connection paths#

The Ollama integration is the reliable, in-core path. It is configured with a base URL and a model, and everything it expects answers here: /api/tags lists the models on this server, so the integration's model picker fills itself from your library; /api/chat serves the conversation, streaming included; /api/pull can even fetch catalog models, though pulling from the Models section gives you progress and sizing in one place. Set the base URL to the server root (no /v1 suffix), pick a model, done.

OpenAI-compatible community integrations are the path when you need to send a key or want request options the Ollama integration does not expose. Any integration that accepts an OpenAI-compatible base URL, an API key, and a model id (Extended OpenAI Conversation is the established community example) points at http://your-server:PORT/v1 with an lmk_ key and works unchanged; the surface it hits is the same one documented in API Compatibility. Community integrations move faster and break more often than core ones: what stays stable is the trio of base URL, key, and model id.

Both paths land on the same engine and the same models. If you are consolidating an existing Ollama box into this server, the wider migration is Coming from Ollama.

4Choosing models for a voice round trip#

A voice command is a latency budget, not a throughput problem: the pipeline already spends time on wake word and speech to text before the model sees a token, and the reply is spoken, so nobody reads ahead of the synthesis. Two roles, two model choices:

• Intent-style traffic (turn on the lights, is the garage closed) wants the smallest model that reliably calls tools. Start with qwen3.5:0.8b, qwen3.5:2b, or gemma4:e2b; step up to qwen3.5:4b or gemma4:e4b if tool calls come back malformed or entity names get mangled.
• Conversational chat (the same agent asked open questions) justifies a larger model: qwen3.5:9b, gemma4:12b, gptoss:20b, or glm4.7-flash, hardware permitting. The catalog in Models is the living list, with sizes and context lengths beside each entry; The Right Model for Your Machine is the longer treatment.

Two Assist pipelines can carry the two choices: a fast one wired to your voice satellites and a richer one for the dashboard's chat, each naming its own model, since a model named in the request always wins over the server default. Keep the voice model resident (the Models section loads it explicitly): first-use loading is fine for a chat tab and unacceptable in a voice exchange. Then measure on your own commands rather than trusting any list, including this one: Measurement shows how, and Hardware Sizing covers what the box needs to hold both models.

5Stated plainly#

• This server replaces Assist's conversation agent; speech to text and text to speech stay on Home Assistant's Wyoming side, which this server does not speak.
• Keyless calls work only over loopback: same-machine setups need no credential, LAN setups need a key and therefore a key-capable integration.
• The Ollama integration needs the server root as base URL; OpenAI-compatible integrations need /v1, a key, and a model id.
• Small model for voice intents, larger model for chat, both resident, both measured on your own traffic.
• Nothing in the exchange leaves your network: that is the point, and Where Data Lives is the accounting.