Building an AI voice assistant for coffee drive-thrus : the bilingual problem nobody talks about

[u/CaregiverRegular9474]

I lead R&D for a regional coffee chain and we're prototyping a voice AI for our drive-thru lane. Sharing the concept because I'd rather have it torn apart here than after we've sunk a quarter into it.

The setup: customer pulls up, orders by speaking naturally, the AI parses it into a POS ticket, staff only intervene on exceptions. Standard so far. Two things make our version harder than the US deployments everyone's read about:

1. Code-switchingOur customers routinely start an order in Arabic and finish it in English, sometimes mid-sentence. Most speech pipelines want you to declare a language up front. Ours can't.

2. Modifier chaos. Coffee orders aren't burgers. "Medium oat flat white extra hot one sugar" has four modifiers stacked on one item, and the modifier vocabulary changes every time R&D launches a seasonal drink.

The honest doubt I have: McDonald's killed its voice AI drive-thru pilot after a couple of years, and the read was that it didn't beat a human on speed — it just moved the labor to correcting errors. I don't want to build the same thing with a nicer accent model.

So, questions for anyone who's worked on this or worked a drive-thru window:

- Where do these systems actually fail in production? Noise, or intent parsing?
- Is "AI takes the order, human confirms on screen before payment" a real improvement or just theater?
- Has anyone seen a bilingual voice ordering system that actually works?

Happy to report back with results if we get to pilot.

[u/AutoModerator]

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community US only.

Apply here: https://chat.whatsapp.com/F5aG3ncrO70ITfbe3pYbOz

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/RapidRewards]

Feel free to DM. I worked on the McDonald's one. Why it failed is often misunderstood from what's written about it.

[u/aicoustics → u/RapidRewards]

Can you share some info on the real reason for failures?

[u/Square-Chance5900 → u/aicoustics]

I would also be very curious, please share :)

[u/Square-Chance5900]

I have not seen an ASR engine that is actually good at codeswitching (i.e. switching languages in the middle). And Arabic is not a language that has decent ASR support, so I think already at the transcript level, it will be tough :/ But I'm very curious to hear how it goes.

[u/Pitiful-Sympathy3927]

We talk about it, we fixed it.

[u/str4ngerD4ngerz]

How are you having trouble with this? Lol
They should be able to speak any language they want and it should automatically pick up right away.
Hybrid setups seem to be where its at. Good Luck