Speaker diarization is the process of labeling an audio recording by speaker, so that each stretch of speech shows who spoke when. The labels are anonymous, such as Speaker 1 and Speaker 2.
A 2021 review by Tae Jin Park and colleagues sums up the task as "who spoke when". In a contact center, diarization turns one block of transcript into a dialogue between agent and customer. Calls with an AI voice agent, or voicebot, need the same split when they are reviewed.
How speaker diarization works
Finding speech. Speech activity detection marks where anyone is speaking and sets aside silence, music, and noise.
Grouping by voice. Many systems cut the speech into short segments, describe each one by its voice characteristics, and cluster similar segments into speakers. They often need the number of speakers, or a range. Google Cloud's documentation (vendor docs) asks for a minimum and a maximum.
End-to-end models. A 2019 paper by Yusuke Fujita and colleagues notes that clustering-based methods cannot handle overlapping speakers correctly. Their neural model outputs the diarization directly from a recording with several speakers, overlaps included.
Joining the transcript. Speaker labels are then attached to the words from automatic speech recognition. The Park review also covers systems that model diarization and recognition together.
Mono, stereo, and AI voice agent (voicebot) calls
Stereo, or dual-channel, recordings. The agent is on one channel and the customer on the other. Amazon Transcribe's documentation (vendor docs) calls this channel identification and transcribes each channel separately. When the two sides talk over each other, their timestamps overlap.
Microsoft's real-time service also tags results by channel, for example "the two sides of a customer support call".
Mono recordings. Both voices share one channel, so diarization has to separate them. A second step then decides which speaker is the agent, often from cues such as the greeting.
Mixed cases. A customer on speakerphone with a relative, a three-way transfer, or hold music can put several voices on one channel. In these cases a stereo recording still benefits from diarization within a channel.
Calls with an AI voice agent. The agent's side is synthetic speech that the system produced itself. Its turns can usually be taken from the system's own logs, so diarization matters mainly on the caller's side.
Measuring diarization: diarization error rate (DER)
Diarization error rate is the usual measure. NIST's Rich Transcription 2005 evaluation plan defines it as the fraction of speaker time not attributed correctly to a speaker. System speakers are first mapped one to one onto the real speakers, for the best overall fit.
Missed speaker time. Speech the system did not assign to anyone.
False alarm speaker time. Speech the system reported where there was none.
Speaker error time. Speech assigned to the wrong speaker.
Scoring settings change the number. NIST's 2005 plan did not score 250 milliseconds around each segment boundary, to forgive timing errors in the reference. Its primary score left out overlapping speech, which was reported separately. The open-source pyannote.metrics library scores overlap by default and uses no collar unless asked. Two DER figures compare fairly only when these settings match.
DER measures time. A misattributed "yes" lasts a fraction of a second and barely moves it. In a consent or a payment promise, that one word can matter most.
Where diarization is used in a contact center
Quality assurance. Scoring an agent's behavior depends on knowing which lines the agent said (see quality assurance).
Compliance review. A required disclosure has to come from the agent, and a consent from the customer.
Speech analytics. Talk-time share, interruptions, and silence all depend on speaker turns. See speech analytics and call center analytics.
Privacy. Diarization groups voices within one recording. Under the GDPR, biometric data processed to uniquely identify a person is a special category (Articles 4(14) and 9(1)). Before linking voices across calls, it is worth checking the setup with your data protection officer.
Frequently asked questions
What is the difference between speaker diarization and speaker recognition?
Diarization answers who spoke when within one recording, using anonymous labels. Speaker recognition matches a voice to a known person, for example to verify a caller. Recognition that uniquely identifies a person processes biometric data under the GDPR.
Do you need speaker diarization if calls are recorded in stereo?
Often less. Separate channels already split the agent from the customer. Diarization still helps when more than one person speaks on the same channel, such as on a speakerphone or a conference transfer.
What is a good diarization error rate?
It depends on the recordings. The number of speakers, overlap, audio quality, and scoring settings such as the collar all move the figure. A fair comparison scores every system on the same recordings with the same settings.
How does diarization handle people talking over each other?
Clustering-based systems often struggle with overlap, as Fujita and colleagues note. End-to-end neural models can mark two speakers at once. On stereo recordings, overlap shows up as speech on both channels at the same time.
How we approach this at Zowie
Zowie's AI agents work on voice, chat, and email, with over 100 million conversations a year across our clients. On a call, the language model speaks. A separate, deterministic Decision Engine returns every amount, rate, offer, disclosure, and escalation from your rules, with an audit log. To talk through how your calls are recorded and reviewed, you are welcome to book a demo.
Explore: Automatic speech recognition (ASR), Speech analytics, Quality assurance, Call center analytics, PII redaction, Word error rate (WER)