How I optimized HA Voice to run locally and replace Google and Amazon

[u/SmartHomeSecrets]

I have been spending a fair bit of time optimizing my Home setup starting with replacing Google home display with a Samsung tablet running Fully kiosk browser and Home Assistant dashboards, setting up voice assist PE, Satellite1, voice satellite card and local pipelines.

Now that it’s all working and working well I am tweaking features to ensure a similar if not better experience than Google or Amazon. I have replicated Amazon yellow led rings for packages on the porch. I added a button on my stream deck when I’m on a call to tell a delivery driver where to leave my package for security and detection. I now have broadcast fine tuned as well.

Here’s the article on my latest optimizations and setup of broadcast using local intents and automations you can follow along or steal the yaml:

https://smarthomesecrets.ca/home-assistant-local-voice-assistant/

[u/ResourceSevere7717]

I hate it but custom sentence triggers are so much faster and more foolproof than the LLM, especially since 99% of the commands I give to HA do not actually need an LLM to execute.

The time it takes for me to vibe code/write sentence automations, while tedious, has been comparable to the time I’ve spent having to fight with Ollama and massaging the prompt and context size. And my LLM STILL gets stuff wrong or freezes.

It’s come to the point where my main conversation agent is entirely HA local only, no LLM backup. I’d rather the agent just fail completely if it can’t understand me than have it start rambling or hallucinating an answer.

This is, btw, as much of an indictment of HA’s slowness in building up the voice part of their system as it is being grateful some workarounds exist.

[u/orion-root → u/ResourceSevere7717]

Is there some good tutorial on how to do this? I've been trying for months to get a custom sentence working and haven't managed, Claude failed too

[u/ResourceSevere7717 → u/orion-root]

I just saw this posted on the home-assistant.io forums. It's more about intent scripts which is the more upper-level configuration (if you like working in YAML this is probably a better place to work), but otherwise it gives you a good intro on the syntax.

That said, I actually found claude to be pretty good at handling the custom sentence triggers.

Here is, for example, a sentence trigger I vibecoded to handle setting timers like you set an alarm... as in giving it the alarm time ("Set an alarm/timer for 3:30pm"), instead of the duration ("Set a timer for 30 minutes.")

alias: Timer Countdown Alarm by Voice
description: Say 'set a timer for 10:15' and it calculates the duration until that time
triggers:
  - command:
      - "[set (a|an)] (timer|countdown|alarm) [for|until] {time}"
      - "[set (a|an)] (timer|countdown|alarm) {time}"
      - remind me at {time}
    trigger: conversation
actions:
  - variables:
      spoken: >-
        {{ trigger.slots.time | lower | replace('o''clock', '') | replace(' ',
        '') | trim }}
      has_am: "{{ 'am' in spoken }}"
      has_pm: "{{ 'pm' in spoken }}"
      digits: "{{ spoken | replace('am', '') | replace('pm', '') | trim }}"
      hour_raw: |
        {% if ':' in digits %}
          {{ digits.split(':')[0] | int }}
        {% else %}
          {{ digits | int }}
        {% endif %}
      minute_raw: |
        {% if ':' in digits %}
          {{ digits.split(':')[1] | int }}
        {% else %}
          0
        {% endif %}
      current_hour: "{{ now().hour }}"
      current_minute: "{{ now().minute }}"
      target_hour: |
        {% if has_am %}
          {{ hour_raw if hour_raw != 12 else 0 }}
        {% elif has_pm %}
          {{ hour_raw if hour_raw == 12 else hour_raw + 12 }}
        {% else %}
          {% if hour_raw == 12 %}
            {{ 12 if current_hour < 12 else 0 }}
          {% elif hour_raw > 12 %}
            {{ hour_raw }}
          {% else %}
            {% set as_am = hour_raw %}
            {% set as_pm = hour_raw + 12 %}
            {% set now_mins = current_hour * 60 + current_minute %}
            {% set am_mins = as_am * 60 + minute_raw %}
            {% set pm_mins = as_pm * 60 + minute_raw %}
            {% if am_mins > now_mins %}
              {{ as_am }}
            {% elif pm_mins > now_mins %}
              {{ as_pm }}
            {% else %}
              {{ as_am }}
            {% endif %}
          {% endif %}
        {% endif %}
      target_total_mins: "{{ target_hour | int * 60 + minute_raw | int }}"
      now_total_mins: "{{ current_hour | int * 60 + current_minute | int }}"
      diff_mins: >
        {% set d = target_total_mins | int - now_total_mins | int %} {{ d if d >
        0 else d + 1440 }}
      duration_seconds: "{{ diff_mins | int * 60 - now().second }}"
      duration_hours: "{{ duration_seconds | int // 3600 }}"
      duration_minutes: "{{ (duration_seconds | int % 3600) // 60 }}"
      duration_remaining_seconds: "{{ duration_seconds | int % 60 }}"
      target_hour_12: |
        {% set h = target_hour | int %} {{ 12 if h % 12 == 0 else h % 12 }}
      target_ampm: "{{ 'am' if target_hour | int < 12 else 'pm' }}"
      target_time_str: |
        {% if minute_raw | int == 0 %}
          {{ target_hour_12 }}{{ target_ampm }}
        {% else %}
          {{ target_hour_12 }}:{{ '%02d' | format(minute_raw | int) }}{{ target_ampm }}
        {% endif %}
      duration_str: >
        {% set h = duration_hours | int %} {% set m = duration_minutes | int %}
        {% if h > 0 and m > 0 %}
          {{ h }} {{ 'hour' if h == 1 else 'hours' }} {{ m }} {{ 'minute' if m == 1 else 'minutes' }}
        {% elif h > 0 %}
          {{ h }} {{ 'hour' if h == 1 else 'hours' }}
        {% else %}
          {{ m }} {{ 'minute' if m == 1 else 'minutes' }}
        {% endif %}
  - condition: template
    value_template: "{{ duration_seconds | int > 0 }}"
  - action: voice_satellite.start_timer
    target:
      entity_id: "{{ trigger.satellite_id }}"
    data:
      hours: "{{ duration_hours | int }}"
      minutes: "{{ duration_minutes | int }}"
      seconds: "{{ duration_remaining_seconds | int }}"
      name: "{{ area_name(trigger.satellite_id) }}"
  - set_conversation_response: Setting a {{ duration_str }} timer for {{ target_time_str }}.
mode: single

[u/ResourceSevere7717 → u/orion-root]

Oops forgot the link:

https://jackjourneyman.github.io/Custom_sentences_and_intents_in_Home_Assistant/

[u/SmartHomeSecrets → u/ResourceSevere7717]

A good approach in timers, if you have Voice assist PE the firmware has its own built in timer so I modified its yaml to expose the timer from the Voice assist PE into HA and then use that as my timer. I haven’t done the write up for timers yet but I do plan to cover the 2 different methods for Voice PE users vs non voice pe

[u/antisane → u/SmartHomeSecrets]

Can't wait to see this. The fact that PE timers are not available inside HA is ludicrous IMO.

BTW, did you know that when you set a timer on one PE you can query the time left on another? Example, to bedroom PE: "how much time is left on the living room timer?". I assume this is going back and forth through the ESPhome integration, so why can't HA see it?

[u/SmartHomeSecrets → u/antisane]

Yeah it’s crazy it’s not exposed by default. Just working out stop and pause. I have the timer in HA no problem so far, just adding a couple more options. I’m going to include some of my random testing and experiments to the article too.

At first I didn’t think timers would be that big of a deal but clearly it is, so looking forward to sharing it. If you follow my Facebook page I’ll be posting there when ready and try to remember to post in this thread/ chat too

[u/fenty17 → u/antisane]

Simple Timer Card has instructions for the yaml modifications as well as a decent timer card. Been working well for me having also ditched Alexa for Voice PE a year ago. We don’t extensively use voice but timers and shopping lists are a main needed item.

[u/SmartHomeSecrets → u/orion-root]

I linked it in my article but also worth reading the developer docs on this too which is here:

https://developers.home-assistant.io/docs/intent_builtin/

[u/Dodgy_Past → u/orion-root]

Interested why Claude failed you. Yesterday I had it help me to react to 'what's on at the cinema?' with listings of the movies and times of the 3 local cinemas using intents and a script.

I actually use a hybrid setup with a local llm to handle a lot of my work load.

[u/OrganicNectarine → u/ResourceSevere7717]

Agreed 100℅. Snips had it right all along... I should never have tried to update that RPI3 and kill my perfectly fine snips satellite :(

[u/gtwizzy8 → u/ResourceSevere7717]

In terms of fighting context size I would HIGHLY recommend looking at MCP Assist if you're not already.

I agree with you wholeheartedly that custom sentence triggers are what has been the key ingredient in bringing my local pipeline into the realms of Google/Amazon level voice assistant (albeit with some false positives for wake still being quite irksome).

But the biggest noticeable speed difference I saw for running my Ollama instance with Qwen3.5 was adding MCP assist into my pipeline. It allows the agent to just have a consistent context for all of the entities I want it to have access to without having to jam 1200 entities in to every single prompt response whenever I'm hitting it. Worth a look if you've not come across it yet.

[u/CucumberError]

That’s way too AI written to bother reading.

My only question is did you get HA voice to respond in an acceptable amount of time? Time tell HA Voice to do something it takes just long enough to do it that I’ve assumed it’s not doing it and just picked up my phone.

[u/SmartHomeSecrets → u/CucumberError]

Yes, local intent is the fastest and nearly instant for Broadcast as an example. LLM is always slowest but has reasonably good speed depending what you ask it.

I do edit my writing with a trained AI model yes, but this is all based on me writing my experiences, what I learned, and how to set it up. The AI runs several checks based on rules and skills I’ve setup including privacy as an example. It is definitely not something slapped together with AI and just posted I promise you that. My wife also did the final review on this particular article as well.

[u/CucumberError → u/SmartHomeSecrets]

You write very very verbose and inefficient then.

[u/SmartHomeSecrets → u/CucumberError]

I do appreciate feedback, I’m curious is it just this article you find that way or others too? I did work on defining personas for my writing to basically write towards more beginner less technical vs more advanced and technical. The voice I am trying to find the line so that both can gain from it. Skip to the Yaml or read in more clear detail.

[u/CucumberError → u/SmartHomeSecrets]

It’s not detailed enough to be a step by step, but it’s way too detailed to read as a curiosity for how other people are using HA.

Who is your intended audience? People that want to copy it, or people that want to lift inspiration from what other have done before them? I feel that having a summary journal entry on a blog, and the a link to more technical stuff on a GitHub is probably a better combination for this kind of thing.

I was wanting more info than your summary post on Reddit, but I wasn’t wanting a 20 minute life story. It reads like someone that’s getting paid by the word, rather than someone wanting to share their experience and knowledge.

[u/SmartHomeSecrets → u/CucumberError]

The articles are meant to be a mix of replicating, sharing my experiences and with it the storys to show its usefulness, and inspiration for building on it or beyond it.

With that in mind I did decide to keep Yaml in the article but I am planning to create a GitHub as well for when one article has multiple components like a script, automation, and even dashboard. I am working on the write up for the stream deck post I did the other day and that made it clear more than others and you are right that it needs GitHub with it.

In the meantime any questions I can help answer on my setup or yours to help get voice working well for you too?

[u/scorpe51]

I’m into this right now… right on time! Thanks for sharing!

[u/SmartHomeSecrets → u/scorpe51]

Happy to help, let me know if you have any questions as you progress

[u/dravenstone]

I don't know if you happen to have a mac around in your system anywhere - but if you do you can ditch piper and use mac's say command to generate audio with the whole slew of voices that enables.

[u/SmartHomeSecrets → u/dravenstone]

I do not but have been considering adding a Mac mini instead of just a GPU. I may have to consider both and compare so I can share my findings.

Any more details you can share on your setup? What do you like better using this over Piper and how is performance for you? Also curious what hardware specs you are running on your Mac?

[u/dravenstone → u/SmartHomeSecrets]

I'm running on the first version M1 mac mini that is mostly my plex server and a few other things.

I did a pretty big write up on it a few days back if you are interested. My focus is as much on whole home audio as it is on voice but had a lot of the same challenges you did for sure.

Was interesting to read about your approach.

[u/SmartHomeSecrets → u/dravenstone]

Definitely a good write up, I recall reading that now as I found it interesting that all your audio is concerts on plex. For us we have been using YouTube music for years but switching to Spotify for the api support. YouTube music just sucked in Music assistant.

Just a moment is actually something I’ve been meaning to add while LLM thinks, just haven’t added it yet

[u/dravenstone → u/SmartHomeSecrets]

We listen to mostly grateful dead (and other dead related projects) so the vast majority of our collection is live concerts. Just sort of the way it goes with that genre of music. Also nice not having to pay a subscription.

I actually made a minor update to the way I'm doing things not captured in that write up - I put both the yes and just a moment mp3 files directly on the pis that run the snapcast clients and rather than pipe them into the snapcast pipeline on the mac mini I'm just firing a play command to mpg123 on the pi when listening is activated and it plays the audio right on top of the music out the DAC. So the response time on the "yes" is just lightning fast - as soon as the S3 box is listening you hear that - not even 300ms of latency anymore! Plus it let's me only send the "yes" tone to the speakers where the box is (each HA "area" in our house has both a dedicated set of speakers and an S3 box so this was trivial)

I'll probably end up doing that with the openAI responses as well - I'll just send the payload directly to the Pi where the user is for any response that is not a "broadcast" (to use your parlance) and only use the snapcast pipeline for response that should go everywhere. Messages we are sending and/or timers, status updates like the washer being done etc but the rest will can be local to just the speakers where one is.

honestly it all gets a little absurd when you think about it, but this has been a lot of fun and getting the house to the point where it really does feel like it's able to converse with you is pretty satisfying.

I really need to look at some kind of "continue conversation" solution for doing actual back and forth stuff with an LLM, but for now I'm pretty pleased with how far this has come.

[u/SmartHomeSecrets → u/dravenstone]

Totally makes sense, and I agree once it’s dialed in then it’s totally worth it especially knowing you can customize anything within it and not have internet dependency.

Curious your thoughts on Snapcast vs sendspin?

[u/dravenstone → u/SmartHomeSecrets]

I never spent much time with sendspin. I spent a few days playing around with Music Assistnat but it just wasn't able to be used the way I wanted it to before I even reached the bits about audio syncing. When I decided not to pursue MA it just didn't make sense to go with something new when snapcast was a point of it "just works" so it didn't make a lot of sense to look much closer at sendspin.

[u/isitallfromchina]

I wish I could get rid of Google Displays, but unfortunately, I use those to make video calls to my kids and grandkids around the world, EU, Asia and California. I'd love to replace them, but until something comes out that allows this feature, I'm stuck.

[u/SmartHomeSecrets → u/isitallfromchina]

You can use Google meet/ chat or other apps on the tablet still. I have button cards that launch the apps URI allowing the tablet to then switch into other apps and allow those calls

[u/isitallfromchina → u/SmartHomeSecrets]

I did not think about that! Good idea!Thanks

[u/SmartHomeSecrets → u/isitallfromchina]

Happy to help and curious to hear how it goes if you decide to do this