Voice assistants and LLM integration

Why this matters

You already built a dashboard in Chapter 6 and wrote an automation in Chapter 8. Neither helps when you carry a pot into the kitchen or cannot reach your phone from bed. In those moments, voice control is more practical than a dashboard.

Home Assistant’s built-in voice assistant is called Assist. Its basic features are included with Home Assistant. Unlike Google Assistant or Alexa, you can decide whether each processing stage runs locally or in the cloud.

After this chapter, you can:

• Explain how Assist differs from a large language model (LLM).
• Choose which entities voice assistants can control and which must remain private.
• Build an Assist pipeline and choose local or cloud processing for each of its four stages.
• Configure an OpenAI, Google, or Anthropic model as a conversation agent and allow it to control selected devices.
• Explain the role of Model Context Protocol (MCP) in Home Assistant.
• Evaluate cost, latency, and privacy trade-offs.

Assist: three terms to understand

Three related terms often cause confusion, so start by distinguishing them clearly.

The key point is that Home Assistant’s built-in conversation agent is a rule-based sentence matcher, not an LLM. It compares your request with predefined sentence patterns and triggers the corresponding intent when it finds a match. For example, “turn on the lights in the living room” can trigger HassTurnOn.

This design has two clear consequences:

• Advantages: It is extremely fast, fully local, free to run, and does not invent responses. If you tell it to turn off a light, it turns off the light.
• Limitations: A request that does not match a supported pattern may fail. For example, it may not understand “Could you make the living room darker?” It is also not designed for open-ended conversation.

The built-in agent has become more flexible. Version 2025.9 added a fuzzy matcher that can recognise an intent despite extra words or different phrasing. Home Assistant documentation notes that this feature was implemented in English first, with other languages to follow. Clear names and aliases therefore remain especially important in other languages.

To try Assist without additional setup, sign in to Home Assistant, select the conversation icon in the upper-right corner, and type “turn on the living room light” or speak a sentence in your configured language. This is the simplest usable form of Assist.

A voice pipeline has four independently configurable stages

Think of a voice assistant as a production line. You speak a request; the pipeline turns it into an action and a reply. The four stages can each use either local or cloud processing.

For typed requests—for example, when using Assist from the mobile app—stages 1, 2, and 4 do not run; only the conversation agent is used. Connecting ChatGPT for typed conversation and speaking to a voice satellite therefore involve very different configurations.

Local processing options

• Whisper: General-purpose speech-to-text with broad language support, including Chinese. The Home Assistant add-on defaults to auto; it selects tiny-int8 on ARM hardware such as a Raspberry Pi and base-int8 elsewhere. Larger models such as small and medium can improve recognition but require more CPU resources.
• Speech-to-Phrase: Fast speech-to-text for a limited set of phrases derived from your Home Assistant configuration. It works on modest hardware but does not support unrestricted speech. Chinese is not currently listed as a supported language, so choose another engine if you need Chinese recognition.
• Piper: Fast, resource-efficient local text-to-speech.
• openWakeWord: Wake-word detection on the Home Assistant host. Home Assistant documentation states that a Raspberry Pi 4 can handle approximately 5 voice satellites streaming audio simultaneously. For larger installations, move wake-word detection to another machine.

These are all official add-ons. Once installed, Home Assistant discovers them through the Wyoming Protocol integration. Appendix A explains how to use the Add-on Store.

Entity exposure: decide what Assist can access

This is the most important—and most frequently missed—setting in this chapter. Home Assistant does not make every entity available to voice assistants. You must explicitly expose an entity before an assistant can access it. This is a deliberate safety measure.

1. Open the exposure settings

   In the sidebar, select Settings → Voice assistants, then open the Expose tab.

2. Review current exposure

   The list shows exposed entities and the assistants to which each one is exposed. This can include not only Assist, but also Google Assistant or Alexa if you have connected them.

3. Expose only required entities

   Select Expose entities to add several entities at once. Select an individual entity to choose which assistants can access it.

4. Add useful aliases

   Select an entity to add an alias. If the entity is named “living room main light” but your household calls it “main lamp,” add that phrase as an alias. You can also open the entity dialog, select the gear, and open Voice assistants.

5. Remove sensitive controls

   Keep door locks, alarm systems, water heaters, and any device that creates heat or presents a safety risk off the exposure list unless you have assessed the risk carefully.

What happens if you expose too much?

Home Assistant’s guidance is straightforward: expose only the minimum necessary entities. Excessive exposure creates two problems:

Aliases for areas and floors

Entity exposure works with areas and floors. For “turn off all the lights in the living room” to work, those lights must belong to the living room area. You can also add aliases to areas and floors: go to Settings → Areas, labels & zones, select the pencil on an area card, then Add alias. For a floor, use its three-dot menu → Edit floor → Add alias.

Hands-on: build a local voice pipeline

Build an assistant that you can use from your phone. This example combines local Whisper, local Piper, and the built-in conversation agent. It is an inexpensive, privacy-conscious starting point.

1. Install the speech-to-text add-on

   Go to Settings → Add-ons → Add-on Store, find Whisper, and install it. Before starting the add-on, open its Configuration tab and change language from auto to the language you intend to use.

2. Install the text-to-speech add-on

   Find Piper in the same store, install it, and start it.

3. Accept Wyoming Protocol discovery

   Return to Settings → Devices & services. The Wyoming Protocol integration should automatically discover the two add-ons. Select Add for each one. If either is not discovered, select Add integration and search for Wyoming Protocol.

4. Create an assistant

   Go to Settings → Voice assistants and select Add assistant. Enter a descriptive name, such as “Local Home,” and choose a language.

5. Configure the pipeline stages

   On the same screen, set Conversation agent to Home Assistant. Under Speech-to-text, select the Whisper service you added and choose its language. Under Text-to-speech, select Piper and choose a voice. If you do not yet have a physical voice device, you can leave the wake-word setting unchanged.

6. Expose entities

   Switch to the Expose tab and add only the lights, outlets, and fans that you want to control. Voice commands cannot operate entities that are not exposed.

7. Test from your phone

   Open the Companion app installed in Chapter 7, then select the three-dot menu in the upper-right corner → Assist. Test with a typed request first. Once typed commands can control a light, select the microphone and test a spoken request.

8. Set as default assistant

   After every stage works, return to Settings → Voice assistants and set this assistant as the default. All Assist entry points will then use it.

Cloud options: what Home Assistant Cloud provides

Home Assistant Cloud, operated by Nabu Casa, is the official subscription service; its revenue supports Home Assistant development. Its voice features include cloud speech-to-text and cloud text-to-speech.

After you subscribe and sign in, Home Assistant uses your language settings to create an assistant automatically. Go to Settings → Voice assistants, select the Home Assistant Cloud assistant, confirm the STT and TTS languages, and leave the conversation agent set to Home Assistant.

Connect an LLM as the conversation agent

The built-in agent recognises supported sentence patterns. If you want the assistant to interpret an indirect request such as “It’s a bit cold today; please do something about it,” or to respond in a particular style, consider using an LLM as the conversation agent.

Official integrations

Setup

1. Add integration

   Go to Settings → Devices & services → Add integration. Search for OpenAI Conversation, Google Generative AI, Anthropic, or Ollama, then enter the API key or local URL requested on screen.

2. Create a new assistant

   Go to Settings → Voice assistants → Add assistant. Enter a name, then select the LLM integration under Conversation agent.

3. Define its response style

   Select the gear beside the agent name to open its prompt template. For example, enter “You are a concise butler. Answer in no more than two sentences,” or define another appropriate persona. The field supports Home Assistant template syntax, so you can include current information.

4. Enable control permissions

   On the same settings screen, enable Control Home Assistant. Without this permission, the model can converse but cannot operate devices. The next section explains the requirements in detail.

5. Test by typing first

   Select “Start conversation” from the agent menu and test with typed requests first. Add speech-to-text and text-to-speech only after the conversation agent works, so you can isolate any later audio failure.

Allow an LLM to control exposed entities

An LLM may reply that it turned off a light even though the light remains on. That usually means one of the following three requirements is not met.

First: enable Home Assistant control

The OpenAI, Google Generative AI, and Anthropic integrations include a Control Home Assistant option. Enable it to give the model the Assist API as a tool. When it is disabled, the model is only a chatbot.

In older versions, this setting may be a menu rather than a switch. Select Assist instead of “No control” to grant the same permission.

Second: expose the entity

This is the most common cause. The integration documentation states that a model can query or control only exposed entities. Enabling control without exposing any entities gives the model no available controls. Complete the earlier entity-exposure steps.

Third: verify that the model supports tool use

If you use a small local model through Ollama, remember that smaller models make more errors than larger ones. Home Assistant warns that small models may struggle to maintain a conversation when control is enabled. Its suggested workaround is to create two Ollama configurations: one without control for conversation, and another with control for device operations.

Reduce cost and risk by preferring local handling

The assistant’s three-dot menu includes a Prefer handling commands locally switch. When enabled, Home Assistant first tries to match the request with the built-in conversation agent. A matching command runs locally; only an unmatched request is sent to the LLM.

This approach provides three benefits:

• Speed: A simple command such as “turn on the living room lights” is handled locally with minimal delay.
• Lower cost: Approximately 80% of routine commands can be handled without calling the LLM API or incurring token charges.
• Reliability: Deterministic device commands are handled by the rule-based agent rather than delegated to a generative model.

Expose scripts as controlled capabilities

Scripts created in Chapter 10 can also be exposed to voice assistants. Wrap a sequence of complex actions in a clearly named script, expose that script, and let the model call it as one controlled capability. This is more reliable than asking the model to assemble a long action sequence.

Model Context Protocol (MCP): server and client roles

Model Context Protocol (MCP) is an open protocol that allows large language models to communicate with external tools. Home Assistant has described it as “a very thin layer that allows LLMs to connect to anything.” Home Assistant supports two directions through two separate integrations, which are easy to confuse.

The client integration was added in version 2025.2. Once configured, it makes external tools available to the conversation agent in the same way as Home Assistant’s native Assist API. You must still enable those tools in the conversation agent’s settings.

Add an external page to the sidebar

You may only need quick access to an AI page, not an API integration. Add the page to the Home Assistant sidebar so it opens inside the interface, which can be useful on a wall-mounted tablet.

1. Go to dashboard settings

   Settings → Dashboards.

2. Add a dashboard

   Select Add dashboard, then choose Webpage from the list of types.

3. Enter a name, icon, and URL

   Choose a name that is easy to recognise in the sidebar, select an icon, and enter the URL that you want to embed.

4. Choose who can see it

   You can set whether only administrators can see it and whether it should be displayed in the sidebar. Use the administrator-only option when access should be restricted.

5. Create and test the dashboard

   Return to the main screen and select the new sidebar item to confirm that the page loads.

To embed a page within one dashboard view rather than create a full sidebar item, use a Webpage card. Add it using the card workflow described in Chapter 6.

Voice hardware: where to speak

Once the pipeline is configured, you need a device that can capture your request. The options below range from low-cost testing tools to dedicated hardware.

How to call Assist on Android

• In the app, open the three-dot menu in the upper-right corner and select Assist.
• Set Home Assistant as the system’s default digital assistant, then launch it with the gesture configured by your device, such as swiping up from a lower corner or holding the power button. Once it is the default, you can also launch it from the lock screen.
• Add a Home Assistant widget: touch and hold the Home screen → Widgets → Home Assistant, then drag the Assist widget into place.
• Configure wake-word detection under Settings → Companion app → Assist for Android.

How to call Assist on iPhone

• In the app, open the three-dot menu in the upper-right corner and select Assist.
• Use the iOS Shortcuts app to create an Assist shortcut, then assign your preferred trigger: Back Tap, the Action button, Control Centre, or a Lock Screen widget.
• Optional: iOS Voice Control can associate a custom phrase such as “Okay Nabu” with that shortcut. This is the closest iPhone equivalent to a wake word, but Home Assistant notes that it is slower than pressing a button or asking Siri. Voice Control also listens continuously, so assess that trade-off before enabling it.
• Shortcuts triggered through Siri use Siri’s language rather than the pipeline language. For manually triggered shortcuts, you can select a language. To support two languages, import the shortcut twice and assign one language to each copy.

Official hardware: Voice Preview Edition

The Voice Preview Edition is the most straightforward dedicated option. It combines an ESP32-S3 with 16MB of flash and 8MB of PSRAM, plus an XMOS XU316 audio processor for echo cancellation, noise reduction, and automatic gain control. Its dual-microphone array is designed to hear commands while other audio is playing. It includes a built-in speaker, a 3.5mm audio output, a physical mute switch, a volume dial, an LED ring, and a Grove expansion port for sensors. It measures 84×84×21mm and weighs 96 grams. It does not require Home Assistant Cloud and can run with an entirely local pipeline, although Home Assistant recommends a relatively capable host—approximately an Intel N100 or better—for the complete local experience.

Voice satellites can initiate interactions

Voice devices do not have to wait passively for a command. The assist_satellite domain provides three actions for use in automations:

actions:
  - action: assist_satellite.announce
    target:
      entity_id: assist_satellite.living_room
    data:
      message: The washing machine is ready

The other actions are assist_satellite.start_conversation (start a conversation) and assist_satellite.ask_question (ask a question and capture the answer). For example, after a door opens, an entryway satellite can ask whether to turn on the living-room lights and act on the reply.

Avoid starting an announcement while the satellite is already interacting with someone, because the two messages can overlap. Starting with version 2026.2, automation conditions can check whether a voice satellite is idle, listening, processing, or responding. Before an announcement, add a condition that requires the satellite to be idle so that it does not interrupt a household member.

Privacy, cost, and latency: balancing the trade-offs

There is no single best configuration. Use this comparison to choose the balance that suits your household.

Control token costs

Cloud LLMs are billed by usage, and provider prices change frequently. Check each provider’s current pricing rather than relying on fixed figures here. The following practices help prevent unexpected charges:

• Every request includes the exposed-entity list. Exposing 200 entities can cost an order of magnitude more than exposing 20.
• Long prompt templates are charged on every request. Keep persona instructions to three to five concise sentences rather than writing an extensive backstory.
• Enable “Prefer handling commands locally.” Routine commands that match locally will not call the API. This is the most effective cost-control setting.
• Set a usage limit in the provider’s console before adding the API key. OpenAI, Google, and Anthropic all provide spending or usage controls.

Troubleshooting

1. No device responds, and Assist says “Sorry, I don’t know about that”

   First confirm that the entity is exposed under Settings → Voice assistants → Expose. Next, compare the spoken name with the entity name and aliases; add the phrase your household actually uses. Finally, verify the entity’s area assignment as described in Chapter 3.

2. You cannot identify which pipeline stage failed

   Use the built-in debugging tools. Go to Settings → Voice assistants, select your assistant, and open the Debug dialog. Choose the latest run from the menu to inspect the recognised text and the time spent in each stage. From the icon in the upper-right corner of the debug screen, select Run text pipeline to test the complete text-processing path with a typed sentence.

3. You want to test sentence parsing without running an action

   Use Assist’s sentence-pattern testing tool under Settings → Tools → Assist. It reports the parsed intent and targeted entities but does not execute the action. This is the quickest way to validate a renamed entity or a new alias.

   If you cannot find Tools, check your version. Before this change, this section is called Developer tools and appears at the bottom of the sidebar. Starting with version 2026.2, it appears as Tools at the bottom of Settings. On a computer, press Ctrl + K—or Cmd + K on a Mac—to open Quick search, then enter “tools” or “developer.”

4. It responds but produces no sound

   First check Settings → System → Network and confirm that the Local network Home Assistant URL is correct. The voice device uses this URL to retrieve generated audio; an incorrect URL can produce a visible response with no sound. If you maintain YAML manually without default_config:, also confirm that media_source: is present.

5. Assist recognises an entity only when I use its full formal name

   Add an alias under Settings → Voice assistants → Expose → select the entity. An entity can have multiple aliases, so add the everyday terms used by different household members or languages.

6. A request for one entity controls another

   This usually results from exposing too many entities with similar names. First remove entities that do not need voice access. Then rename easily confused entities using the “area + description” pattern. Also verify each entity’s domain and device class; Assist cannot infer that a valve represented incorrectly as a switch should respond to “open.”

7. Local Whisper is unacceptably slow

   Try three adjustments. First, change language from auto to a specific language; automatic detection is much slower. Second, choose a smaller model such as the int8-compressed tiny-int8 or base-int8. Third, move speech-to-text to Home Assistant Cloud while keeping the conversation agent local. Whisper’s default beam_size is 0, which automatically uses 1 on ARM systems and 5 on other systems. A larger value may improve accuracy but increases processing time.

8. The wake word is frequently missed or triggers unexpectedly

   First determine whether wake-word detection runs on the device with microWakeWord or on the host with openWakeWord. For host-based detection, Home Assistant states that a Raspberry Pi 4 can support approximately 5 simultaneous satellite streams; additional streams may cause delays. For on-device detection, review the microphone gain and noise-suppression settings in the device’s ESPHome configuration.

9. The LLM claims to have acted, but nothing happened

   This is a common model hallucination. Check, in order, that Control Home Assistant is enabled in the integration settings, that the entity is exposed, and that the model is capable enough for reliable tool use. Small local models may struggle even to maintain a conversation after control is enabled. Enable “Prefer handling commands locally” so that deterministic commands bypass the model.

10. An embedded sidebar page is blank

    The remote site may use browser security headers that prohibit iframe embedding. You cannot override this restriction; use a different site or open the page in a new tab. Also check that you are not embedding an HTTP page in an HTTPS Home Assistant instance.

FAQ

conversation:
  intents:
    HassTurnOn:
      - "activate [the] {name}"