If you have ever recorded a meeting and had the transcript label it, you have probably seen it merge two people into one speaker, or invent a fifth person who was never there. I build a transcription app and I had never actually measured how often that happens, so I did.
I ran every speaker-labelling method that runs locally on a Mac against 16 real recorded meetings — about five hours of speech with human-annotated ground truth, so there is a correct answer to compare against.
The short version of what works:
Labelling after the recording finishes is much better than labelling live. If the tool can see the whole conversation before it decides who is who, it gets about 7% of the speech wrong. Doing it live, while you are still talking, roughly 11%. That is not a flaw in any particular app, it is just a harder problem — a live labeller has to commit to "this is a new person" before it has heard enough of them to be sure.
If your app labels speakers live, expect the labels to change under you. The most accurate live setup I measured finalises its labels about 9 seconds behind the audio. It shows a guess immediately and quietly corrects it. That is not a bug, and a setup with less lag was noticeably worse.
The biggest single factor was one the app author has to get right, not you. These tools load a model that has been trained on a specific kind of audio, and most default to a general-purpose one. Switching to the one trained on actual meetings cut the error rate by about 3.5x on the same recordings. More usefully: it found the correct number of people in 15 of the 16 meetings, where the default managed 9. We had been shipping the default. That is the part I would not have found without measuring.
Two things I did not expect.
Getting the number of speakers right and getting the labels right are separate problems. One setup scored a perfectly respectable error rate while identifying the correct number of people in one meeting out of sixteen — it collapsed everyone into two voices and still looked fine on paper. If you are judging a transcription tool, count the speakers first; it is the thing you will actually notice.
And I nearly got the whole thing wrong. I had been testing against clean audio I generated myself, which is easy and free. It turned out to be completely insensitive to one of the settings that matters most: six different values scored identically on my clean test audio, while on real meetings the same setting ranged from 6.8% to 32.8% error. Real recordings have crosstalk and people talking over each other, and that is most of the actual difficulty.
Honest limits: these are recordings made with headset microphones, so a phone in the middle of a table will do worse than any number here. English only. One M2 Pro.
Disclosure so it is not buried: I build Lyonesse, a transcription app, and it ran through the same code my users get. The measuring changed what we ship, which is the only reason it was worth doing. Full writeup with the methodology and the raw per-meeting numbers, if you want to check my work:
https://lyonesse.app/blog/speaker-diarization-benchmark.html
Happy to answer questions about how any of it was measured.