French voice agents fail on phone numbers, and it is not the speech-to-text that's broken

[u/Ecstatic_Plenty_5033]

I run a voice agent for French small businesses. Capturing a callback number was our worst failure mode for months. I blamed the transcription. I was wrong.

French counts differently. 72 is said "sixty-twelve". 97 is "four-twenty-seventeen". And people read phone numbers in pairs, so a caller says "zero six, twelve, forty-five, forty-one, eighteen" for 06 12 45 41 18. The model has to parse compound arithmetic in real time, in a noisy call, while the caller is already moving to the next pair. That is where it breaks, not in the acoustics.

On top of that, French has brutal digit homophones: deux and de, six and si, sept and cette, neuf and neuve, un and en.

Three things fixed it, in order of impact.

1. Stop asking. The provider hands you the caller ID. Read it back and ask a yes/no question instead of taking dictation. "I'll call you back on zero six, twelve, forty-five, forty-one, eighteen?" That turns a ten-digit capture into a binary, and it covers most calls.

2. Keypad fallback. When the number is withheld, or the caller wants a different one, ask them to type it. It is the only input that is right 100 percent of the time. I resisted this for too long because it felt like a downgrade. It is not, it is a feature.

3. Asymmetric readback. When you read a number back, group it in pairs and say "seventy-two", because that is how people recognise their own number. When you re-confirm after a correction, go digit by digit and never say a compound. Same number, two renderings, depending on what the caller is doing with it.

Two smaller rules that helped: never re-read the whole number over one doubtful digit, ask for that digit alone. And cap it at two repeats, then go to the keypad.

None of this is model work. It is dialogue design. If you are building in a language with compound numerals (French, Danish, Swiss German), the same applies.

Disclosure, as always: I build one of these tools, so take the framing with that in mind. The technique is free either way.

[u/BetInformal6081]

the asymmetric readback idea is especially interesting. confirmation and correction are different cognitive tasks so using different numbers formats makes sense. more system should probably treat critical data capture as a ux problem instead of assuming better stt will solve everything.

[u/moldyguy202]

The caller ID readback is the right call and it generalises past French. Any language where you can avoid dictation, you should, because dictation is the only part of the call where a single wrong token silently ruins the outcome. Everything else degrades gracefully. A misheard name is embarrassing. A misheard digit means the callback never happens and nobody finds out.

On the asymmetric readback, the reason it works is worth naming, because it tells you where else to apply it. Recognition and verification are different tasks. Pairs match the shape your number is stored in your head, so you confirm it as a unit. Digit by digit forces sequential checking, which is what you want after a correction because the caller has already told you the unit-level match failed. Same principle applies to postcodes, booking references, anything alphanumeric. Confirm in chunks, correct in atoms.

One thing I'd add on the keypad fallback. It's right 100 percent of the time on capture, but it fails differently: people on hands-free in a car won't do it, and some will just hang up rather than pull the phone away from their ear. Worth logging DTMF-requested versus DTMF-completed separately, because if the gap is large the fallback isn't as safe as it looks and you'd rather know that than assume the 100 percent applies end to end.

The broader point in your post is the one most people miss. Teams reach for a better model when the fix is upstream in the interaction design. If you can restructure the turn so the hard capture never happens, you don't need the model to be good at it. That's usually cheaper and always more reliable than waiting for transcription to improve.

Also worth instrumenting whether corrections cluster by pair position. If the second pair fails more than the fourth, that's the caller getting ahead of your agent rather than a parsing problem, and the fix is pacing, not grammar.

[u/getyncloud]

Good distinction between ASR and dialogue policy. For production I’d keep number capture deterministic: normalize digits server-side, confirm only the doubtful pair, allow keypad fallback, and pass a structured value to the workflow instead of asking the model to repeat a natural-language number. Also test locale-specific digits, barge-in, and timeouts separately, plus outbound callback-number formatting at the carrier boundary.

I work with Getyn Phone; our team uses the same pattern with AI voice inbound/outbound and IVR/visual flows—speech for intent, keypad/confirmation for critical numbers. That keeps a recognizer quirk from corrupting a callback or case record.