Automatic speech recognition (ASR) is technology that converts spoken language into written text. It is the first step in voice automation. An AI voice agent, or voicebot, turns the caller's words into text before it can understand or act on them.
ASR is also called speech-to-text (STT). In a contact center, it feeds voice agents during a call and speech analytics after it. Its accuracy is usually reported as word error rate.
How automatic speech recognition works
The classic pipeline. Earlier systems combined separately built parts. An acoustic model maps sounds to speech units, a pronunciation model links those units to words, and a language model scores likely word sequences.
End-to-end models. A single neural network learns the whole path from audio to text. One early model, Listen, Attend and Spell (2015), learns all the components of a recognizer jointly. A 2017 Google paper describes such models as folding the acoustic, pronunciation, and language models into one network.
Large multilingual models. OpenAI's Whisper (2022) was trained on 680,000 hours of multilingual audio with transcripts collected from the internet. Its authors report that it transfers to new test sets without fine-tuning.
Formatting. Many systems then add punctuation and write numbers, dates, and amounts in a standard form. A voice agent leans on this step when a caller reads out an account number or a payment date.
Streaming and batch recognition
Streaming. Real-time recognition returns text while the person is still speaking, with intermediate results that update as more audio arrives. Microsoft's speech documentation (vendor docs) lists contact center voice agents and agent assist as streaming scenarios.
Batch. Batch transcription processes stored recordings, often in large volumes. The same documentation lists post-call analytics for QA, compliance, customer satisfaction, and agent training as batch work.
A streaming recognizer has to commit to words before the sentence ends. End-to-end models were adapted for this. A 2018 Google paper says a model on a phone must decode speech in a streaming fashion, in real time. For a voice agent, recognition time adds to the pause the caller hears (see turn-taking).
ASR on telephone audio
Narrowband sound. Google Cloud's best practices (vendor docs) note that telephony audio is commonly sampled at 8,000 Hz. They advise sending it at that native rate without resampling, and add that lower sample rates may reduce accuracy.
Noise and compression. The same guide says background noise and echoes may reduce accuracy, especially with a lossy codec. Calls from cars, shops, and speakerphones are common in a contact center.
Two speakers. Many contact center recordings keep the agent and the customer on separate channels. Where both share one channel, speaker diarization works out who said what.
Custom vocabulary. Tariff names, product codes, and local street names are often missing from a general model. Microsoft's documentation describes phrase lists, which tell the recognizer which words to expect. It also describes custom models trained on domain text, or on audio with reference transcripts.
Multilingual ASR for AI voice agents and voicebots
Accuracy varies by language. In the Whisper paper, word error rate halved for every 16-fold increase in a language's training data. Most languages in that training set had less than 1,000 hours of audio, because the data was heavily English.
Accents and dialects matter too. A 2020 study in PNAS, led by Allison Koenecke at Stanford, tested five commercial US systems on interviews. Speakers in the Corpus of Regional African American Language had an average word error rate of 0.35, against 0.19 in a California corpus. That is almost twice the error rate. The authors traced the gap to the systems' acoustic models.
For a European contact center, it helps to test each language and region separately. Callers may mix English product names into Polish, German, or Romanian, and many regions have strong dialects. Microsoft's documentation also describes language detection for audio where the language is not known in advance.
A fair test uses recorded calls from each market, scored the same way for every system. Multilingual support covers the wider service side.
Frequently asked questions
Is ASR the same as speech-to-text?
Yes. Automatic speech recognition and speech-to-text name the same task: turning speech into text. "Voice recognition" sometimes means recognizing who is speaking. That task, speaker recognition, is separate from transcription.
What is the difference between ASR and NLU?
ASR produces the words. Natural language understanding works out what the words mean, such as the caller's intent and the dates or amounts in the request. An ASR error passes into NLU, so a misheard date can lead to a wrong action.
How accurate is automatic speech recognition?
It depends on the audio, the language, and the speakers. Accuracy is usually reported as word error rate. Published figures come from someone else's test set, so a test on a sample of your own calls gives a more useful number.
Can ASR work in real time on phone calls?
Yes. Streaming recognition returns text while the caller speaks, which voice agents and agent assist tools need. Waiting for more audio helps the recognizer, and it also makes the caller wait longer, so speed and accuracy are tuned together.
How we approach this at Zowie
Zowie's AI agents work on voice, chat, and email, with over 100 million conversations a year across our clients. On a call, recognition comes first. Then the language model speaks. A separate, deterministic Decision Engine returns every amount, rate, offer, disclosure, and escalation from your rules, with an audit log. To try it on your own calls, you are welcome to book a demo.
Explore: Word error rate (WER), Speaker diarization, Natural language understanding (NLU), Turn-taking, Voice AI, Multilingual support