Speaker diarization benchmarks: pyannote, Sortformer, MOSS, Speech Cloud

Five diarization systems scored with one ruler on the same 132 recordings: 12.9 hours across eight languages. Every system receives the audio and nothing else. No backend is told how many speakers to expect, which is the condition that separates a diarizer from a clustering script with the answer in hand.

How to read these numbers

Each cell is DER / speaker-count agreement. DER is reference-speaker-time weighted with a 0.25 s collar; count agreement is the share of files whose inferred speaker count exactly matches the reference inside the UEM. Lower DER is better, higher agreement is better. Real conversation and synthetic rails are different evidence tiers and must not be read across.

Results by language

Real conversation

Broadcast, telephone and meeting audio. 96 recordings, 12.63 hours.

System	Arabic	German	English	Spanish	Mandarin
Speech Cloud API	21.02% / 33.3%	7.62% / 91.7%	13.21% / 66.7%	15.92% / 58.3%	9.63% / 58.3%
Sortformer streaming	22.32% / 33.3%	8.38% / 83.3%	13.81% / 66.7%	16.33% / 58.3%	9.95% / 58.3%
Sortformer offline	17.92% / 33.3%	8.13% / 83.3%	14.34% / 66.7%	16.43% / 58.3%	10.52% / 41.7%
MOSS	14.57% / 41.7%	16.95% / 83.3%	16.35% / 72.9%	22.51% / 75.0%	10.12% / 83.3%
pyannote Community-1	15.57% / 58.3%	20.22% / 83.3%	15.22% / 77.1%	22.78% / 91.7%	16.19% / 83.3%

Synthetic rails

Concatenated read speech with turn-taking sampled from distributions measured on the CallHome slices. These detect gross domain failures and nothing more. Do not compare these numbers with the table above.

System	Azerbaijani	Swiss German	Russian
Speech Cloud API	4.74% / 83.3%	19.33% / 41.7%	8.15% / 75.0%
Sortformer streaming	5.25% / 75.0%	21.59% / 58.3%	5.35% / 91.7%
Sortformer offline	5.16% / 75.0%	28.03% / 50.0%	8.20% / 100.0%
MOSS	11.30% / 58.3%	26.59% / 41.7%	15.14% / 50.0%
pyannote Community-1	35.23% / 16.7%	45.21% / 0.0%	35.26% / 8.3%

What the numbers say

Rankings are language-dependent. This is the finding that matters most, because a single-language comparison would order these systems wrongly. MOSS is best on Arabic and competitive on Mandarin, yet second-worst on German. pyannote Community-1 inverts the usual trade-off: it posts the best speaker-count agreement on Spanish (91.7%) and English (77.1%) while recording the worst DER on both, because it loses 17.5% of Spanish speech to missed detection.

Speech Cloud leads seven of eight languages, including all three real-conversation slices added in this round. Its German cell, 7.62% DER at 91.7% count agreement with 0.91% speaker confusion, is the strongest result any system posts on real conversation here. Arabic is its weak cell, where MOSS is 6.5 points better. Across all 132 files under one deployment label it records 13.19% DER, 68.9% count agreement, 6.24% missed speech, and zero failures.

Real conversation costs every system missed speech. On the CallHome slices each system misses 4-19% of speech, mostly short backchannels. Spanish is the hardest real slice at 15.92-22.78%, with count agreement collapsing to 58.3% for the leaders: one call carries five speakers, above Sortformer's four-speaker capacity.

Streaming does not inherently cost accuracy. On identical v2.1 weights, streaming and offline Sortformer sit within 0.5 points on German, Spanish, Mandarin and English. Offline wins Arabic by 4.4 and loses Azerbaijani by 5.4. Checkpoint generation matters more than regime: offline v1 trails offline v2.1 by 4.1 points.

Methodology and limits

One scorer, one item set. Every cell comes from the same harness and the same pinned utterances, so two systems are never compared with two different rulers.
No speaker count is supplied to any backend.
Anonymous speaker timelines only. These are diarization results, not named speaker attribution.
Synthetic rails are not real conversation. The az, gsw and ru rails were rebuilt on turn-taking distributions measured from CallHome. Earlier rails used a fixed gap pattern that produced a 100% speaker-change rate, where real conversation repeats a speaker about 16% of the time. That rewarded any diarizer that simply alternates, and it ordered the field wrongly on two languages.
Sortformer caps at four speakers, making exact count agreement structurally impossible on 22 of the 96 original files.

Data provenance

Data is published as soniqo/speech-bench-data at revision 9c91223. The German, Spanish and Mandarin slices come from talkbank/callhome at revision 17c8a153 under CC BY-NC-SA 4.0. Reference turns are the corpus's own timestamps: no VAD, clustering or embedding-derived identity is involved, and overlap is preserved as published. Because that licence forbids commercial use, these slices are evaluation evidence and are not training-approved.

Run it yourself

All five systems run through the same CLI. See the speaker diarization guide for engine selection, DER scoring and RTTM output on Apple Silicon, or the full benchmark index for the ASR and TTS rows measured the same way.