Speaker Diarization

Identify who spoke when in a multi-speaker recording. Three engines are available: lightweight Pyannote (the CLI default), full-pipeline Community-1 (CoreML + native PLDA/VBx), and streaming Sortformer (CoreML, Neural Engine).

Engines

Select the engine with --engine pyannote (default), --engine community1, or --engine sortformer.

Pyannote (default)

Two-stage pipeline: Pyannote segmentation processes overlapping windows with activity-based speaker chaining (Pearson correlation in overlap zones) to assign global speaker labels. Post-hoc WeSpeaker embedding extraction enables target speaker identification via enrollment audio.

Community-1 (CoreML + native VBx)

The full Pyannote Community-1 CoreML bundle: 10-second PyanNet segmentation windows advance every second, masked WeSpeaker embeddings feed centroid AHC and a 256→128 PLDA transform, then native Swift VBx and constrained assignment produce global clusters. It returns one 256-dimensional centroid per detected speaker and supports exact, minimum, or maximum speaker-count bounds.

Sortformer (CoreML)

NVIDIA's end-to-end neural diarization model. Directly predicts per-frame speaker activity for up to 4 speakers without separate embedding or clustering stages. Runs on Neural Engine via CoreML with streaming state buffers (FIFO + speaker cache).

For live audio, SortformerStreamingSession runs the .streaming Sortformer model incrementally with 16 kHz mono PCM buffers of any size. Once its 560 ms lookahead is available, it advances in 480 ms audio steps; a five-file M-series Neural Engine pilot measured 12.4 ms median latency per 500 ms push. Speaker labels follow the Arrival-Order Speaker Cache instead of re-clustering each window, and every push returns a whole-stream snapshot. finish() flushes the buffered tail. In the model-gated equivalence test, final segments matched whole-buffer .streaming diarization within about one 80 ms frame.

Published CALLHOME Baseline

The following is NVIDIA's published raw anonymous-speaker baseline for diar_streaming_sortformer_4spk-v2. It is an external reference, not a local measurement of speech-swift's CoreML conversion and not a named-speaker attribution result.

All rows use 1.04 s input-buffer latency, overlap included, a 0.25 s collar, and post-processing tuned on disjoint CALLHOME Part 1. CALLHOME is licensed and is not bundled. Local diarization-bench reports comparable results separately by reference speaker count in the console and in JSON under byReferenceSpeakerCount.

Community-1 Pipeline

Community-1 runs both neural stages with CoreML and all host processing in Swift, so it does not load MLX or require a Metal shader library. The published defaults are preserved: hard powerset decoding, clean single-speaker masks with the upstream short-track fallback, 0.6 centroid-linkage threshold, Fa=0.07, Fb=0.8, and at most 20 VBx iterations.

On the fixed five-file VoxConverse release subset (1,057.49 seconds, 0.25-second collar, overlap included), Swift output rescored with pyannote.metrics measured 4.66% DER / 21.43% JER; the published CoreML reference measured 4.65% / 21.42%. This is a small parity check, not a dataset-wide quality claim.

With speech-swift's frame-grid scorer on the same subset, Community-1 measured 9.31% DER / 16.20% JER, 25.5× real-time, and 1.54 GB peak RSS. The default Pyannote engine measured 5.03% / 9.28%, 105.7× real-time, and 393 MB. Community-1 is therefore an optional upstream-parity pipeline with speaker-count bounds and returned centroids, not a universal accuracy or speed upgrade. Scores from the two scoring methods are not directly comparable.

Pyannote Pipeline

The default pipeline runs in two stages:

Stage 1: Segmentation + Speaker Chaining

Pyannote segmentation-3.0 processes 10-second sliding windows with 50% overlap. A powerset decoder converts the 7-class output into per-speaker probabilities (up to 3 local speakers per window). Adjacent windows share a 5-second overlap — speaker identity is propagated across windows by computing Pearson correlation between probability tracks in the overlap zone, with greedy exclusive matching for consistent global speaker IDs.

Stage 2: Post-hoc Embedding

After diarization, WeSpeaker ResNet34-LM extracts a 256-dimensional centroid embedding per speaker. These embeddings enable target speaker extraction (--target-speaker) but do not drive the speaker assignment itself.

Migrating from pyannote.audio

If you are coming from the Python pyannote.audio library — replacing a Pipeline subclass that sets pipeline.segmentation = ..., or moving away from a server hosting pyannote/speaker-diarization-3.1 — Soniqo wraps the same Pyannote-Segmentation-3.0 model and runs it entirely on-device on Apple Silicon. No Python runtime, no CUDA, no Hugging Face token at inference time.

The Pyannote-Segmentation-3.0 weights are converted from the upstream HuggingFace checkpoint, so segmentation logits are numerically equivalent within float-precision tolerance. The post-segmentation chaining (Pearson correlation across overlapping windows + greedy exclusive matching) and post-hoc WeSpeaker embedding stages are reimplemented in Swift but produce comparable RTTM output to the reference Python pipeline.

CLI Usage

# Basic diarization (pyannote, default)
.build/release/speech diarize meeting.wav

# Community-1 (CoreML + native PLDA/VBx)
.build/release/speech diarize meeting.wav --engine community1
.build/release/speech diarize meeting.wav --engine community1 --num-speakers 2

# End-to-end Sortformer (CoreML)
.build/release/speech diarize meeting.wav --engine sortformer

# RTTM output format (for evaluation)
.build/release/speech diarize meeting.wav --rttm

# JSON output
.build/release/speech diarize meeting.wav --json

Target Speaker Extraction

Provide enrollment audio of a known speaker to extract only their segments from a recording. The pipeline computes the speaker embedding of the enrollment audio and finds the cluster with the highest cosine similarity.

# Extract segments for a specific speaker
.build/release/speech diarize meeting.wav --target-speaker enrollment.wav

DER Scoring

Evaluate diarization quality by scoring against a reference RTTM file. The pipeline computes the Diarization Error Rate (DER), which measures the proportion of time that is incorrectly attributed.

# Score against reference RTTM
.build/release/speech diarize meeting.wav --score-against reference.rttm

RTTM Output

The --rttm flag produces Rich Transcription Time Marked output, a standard format used for diarization evaluation. Each line follows the format:

SPEAKER filename 1 start_time duration <NA> <NA> speaker_id <NA> <NA>

Options

Model Downloads

Models are downloaded automatically on first use:

Swift API

import SpeechVAD

let pipeline = try await DiarizationPipeline.fromPretrained()
let result = pipeline.diarize(audio: samples, sampleRate: 16000)
for seg in result.segments {
    print("Speaker \(seg.speakerId): [\(seg.startTime)s - \(seg.endTime)s]")
}

// Target speaker extraction
let targetEmb = pipeline.embeddingModel.embed(audio: enrollmentAudio, sampleRate: 16000)
let segments = pipeline.extractSpeaker(
    audio: meetingAudio, sampleRate: 16000,
    targetEmbedding: targetEmb
)

// Full Community-1 pipeline with bounded speaker counts
let community1 = try await Community1DiarizationPipeline.fromPretrained()
try community1.prewarm()
let attributed = try community1.diarize(
    audio: meetingAudio,
    sampleRate: 16000,
    speakerBounds: Community1SpeakerBounds(minimum: 2, maximum: 6)
)