Will an AI receptionist understand your customers - accents, bad lines and all?
Daan van den Bergh
An AI receptionist will mishear some of your callers. In the Dutch study below, a recogniser's word error rate was nearly four times as high on phone conversations as on broadcast news, and three studies found speech recognition did worse with some accents and dialects than with others. People mishear phone calls too: in Microsoft's benchmark, professional transcribers had a word error rate of 5.9% on recorded phone conversations between people who had just met. In my view, whether the job gets booked depends on what the agent does when it isn't sure.
How often does speech recognition mishear a phone call?
A word error rate is a standard measure of how far a transcript strays from a careful human one, and on the widely used test set of recorded phone calls below it stays above zero for machines and professional transcribers alike.
Microsoft's researchers measured the human side in a technical report revised in February 2017. They had professional transcribers type up the NIST 2000 test set, recorded telephone conversations from the task speech researchers have tested against since the 1990s. The transcribers had a word error rate of 5.9% where newly acquainted pairs of people discussed an assigned topic, and 11.3% where friends and family talked freely. Microsoft's own system scored 5.8% and 11.0%. The report's comment on the gap: the same informality, multiple speakers per channel and recording conditions that make the friends-and-family calls hard for computers "make it difficult for people as well".
Phone conversation is a hard case. A Dutch recogniser that a TU Delft research team trained for a 2021 study had a word error rate of 5.5% on broadcast news and 20.8% on conversational telephone speech, nearly four times as high. Microsoft's report calls conversational telephone speech especially difficult, because the speech is spontaneous and full of self-corrections and hesitations.
Do accents make it worse?
Yes, measurably, in Dutch and in English. Three studies, four kinds of speaker:
- Non-native Dutch. The same Dutch study found that "speech of native Dutch speakers is much better recognised than that of non-native speakers, irrespective of age". Among the non-native adults it tested, who ranged from the lowest level, A1, up to B1, better Dutch did not help: the researchers found no correlation between proficiency level and how well the system did. They offer two possible reasons. At those levels the focus is on vocabulary and grammar more than pronunciation, so pronunciation differs little between them, and a proficiency level may simply be a poor proxy for the strength of an accent.
- Flemish. The same system, trained on speech from the Netherlands only, did worse on Flemish speakers than on any region of the Netherlands in every age group but the over-65s, which the authors put down to "the large accent difference" between the two.
- English from around the world. The Edinburgh International Accents of English Corpus is almost 40 hours of video calls between friends, in first- and second-language English. In the corpus's 2023 paper, the best model had a word error rate of 19.7%, against 2.7% on clean read US English, and the authors saw every model drop on Indian, Jamaican and Nigerian English speakers.
- A US dialect. A 2020 study in PNAS ran interviews through five commercial recognisers, from Amazon, Apple, Google, IBM and Microsoft. Their average word error rate was 35% for Black speakers and 19% for white speakers. Counting a transcript with a word error rate above 50% as unusable, as the authors did, that was 23% of the Black speakers' audio snippets and 1.6% of the white speakers'. The authors put the gap down mainly to the systems' acoustic models, and suggest the systems are confused by how African American Vernacular English sounds.
In Edinburgh's tests, the stronger models held up better: Whisper and the commercial model it tested had "a more consistent performance across accents" than the academic-scale models. None of these studies measured an AI receptionist on real business calls, and the newest is from 2023, so use these figures to see where the trouble lies, and test today's systems yourself.
Why a word error rate is not a booking error rate
This part is my own reasoning. A booking hangs on a handful of facts: who is calling, where the job is, what is wrong and when you can come. A dropped "the" or a misheard "really" leaves all four intact. "There's water coming from under the boiler" with one word lost is still a leak at the boiler.
The facts that do matter are the awkward ones to hear. Jansen and Janssen are both common Dutch surnames and are pronounced the same, so a name heard perfectly can still go on the record wrong. Street names and phone numbers fail the same quiet way: a misheard street still points at a real house that belongs to somebody else, and a callback number with one wrong digit rings a stranger while the caller who owns the right one never hears from you.
So the useful question about any AI receptionist is whether it checks those facts with the caller before it writes them down. A mishearing that is caught on the call costs a few seconds.
A mishearing the caller corrects costs a few seconds. One that reaches the job sheet can send the van to the wrong door.
What does SwiftGuard's agent do when it isn't sure?
It reads the facts that matter back to the caller and waits for a yes. Its instructions, as the code has them today:
| What could go wrong | What the agent does |
|---|---|
| The caller's name | Spells it back letter by letter ("that's J-A-N-S-S-E-N, is that right?") and waits for a yes before it goes on a record |
| The address | Asks for the postcode and house number instead of the street, reads back the street and town the map finds, and books nothing until the caller says yes |
| A callback number | Reads the digits back in small groups and waits for a yes; the number they are ringing from comes from caller ID |
| A detail it didn't catch | Asks the caller to say it again |
| The caller's language | Switches only when asked, or after a whole sentence in another language; if it can't make sense of them two turns running, it asks in English which language they want |
| Anything it still can't sort out | Takes a message, so you can ring them back |
The retries have a limit. When it looks up a returning customer and the postcode and house number don't match, it asks for them once more, because its instructions name a bad line as the usual reason. After a second miss it stops asking and offers to text a code to the number they are calling from instead. Its instructions forbid it to fill an answer in for the caller.
On language, it speaks English, Dutch, German, French, Italian and Spanish. It opens in the language set for your business. That setting, which you choose, decides the greeting. A caller speaking something else is told, in English, which languages it can help in. It listens with Deepgram's Nova-3 model on Deepgram's EU servers, and each call primes that model with your business name and, for English and Dutch, a list of trade and booking words such as "stopcock" or "cv-ketel".
One thing it cannot do is put the caller through to you live. When it can't help, the way to a person is the message it takes, and your call back.
Where will it still struggle?
On the calls where the line itself is the problem. A caller on a motorway hard shoulder or on one bar of signal gets asked to repeat. If two turns running come through garbled, the agent's instructions treat that as a sign the caller may be speaking a language it isn't listening for, so it asks in English which language they want, and that question can reach someone who was speaking the right language all along. When nothing else works, its instructions are to take a message for you. You still get the lead, and a call back to make before the job is booked.
The other weak spot is the caller who speaks none of the six languages. The agent tells them in English which languages it can help in, and asks which of those they would like to use.
How the rest of the call works, from the greeting to the calendar entry, is its own post. And if your worry is less "will it understand them" than "will they hang up on it", what callers actually do when AI answers has the survey numbers.
How do you find out whether it understands your callers?
Try it on your own calls before you trust it with all of them.
- Start with only the calls you miss. Forward the calls your phone doesn't answer and keep taking the rest yourself; the forwarding codes are here.
- Read the transcripts. SwiftGuard keeps each call's recording for up to 90 days, and its written transcript until the call is 90 days old, so you can check what it heard against what the caller said.
- Ring it from a bad spot. Ask the broadest-accented person you know to call from the van with the engine running, then check the name and address that landed on the booking.
If it passes those, widen the forwarding. If you are still choosing between vendors, run the same three tests on each; the rest of the checklist for a demo is here. When you are ready to try SwiftGuard on your own callers, the plans are here.