Measuring voice agents: turn-taking, tool calls and expressive speech

Three measurements for three decisions

We measure these three abilities separately. An assistant can wait for you to finish speaking and still choose the wrong tool. It can also speak clearly while missing the emotion or style you asked for.

We ran the local models on an Apple M5 Pro with 48 GiB of memory. For cloud services, we tested the model versions and settings listed in the results.

Each section explains how many recordings we used, which languages we tested and any results still being checked. These details help you compare the scores.

TurnBench: when a pause is a turn end

TurnBench distinguishes a speaker finishing from a pause inside a turn. It also distinguishes an interruption from a backchannel such as “mm-hmm.” Our table below focuses on end-of-turn detection.

We used TurnBench’s public development recordings and its own scoring code. The table shows the run size for each model. The official leaderboard uses a separate test set.

Recall is the share of real turn endings the model detects. FPR (false-positive rate) is the share of labelled pauses within a turn that it incorrectly treats as the speaker finishing.

Our leaderboard orders models by recall minus FPR. The official test leaderboard ranks models by recall, placing those with an FPR above 15% below those that meet the limit.

We added four TurnBench submissions by rescoring their published predictions on the same five conversations, without changing their decisions or rerunning their models. Nunchi v2 has the highest recall-minus-FPR score on this slice: 96.6% recall and 6.4% FPR. VAP reaches 88.2% recall with 7.4% FPR; Mimi makes fewer false detections, at 2.0%, but catches fewer turn endings and takes longer to decide.

VAP (Voice Activity Projection) listens to both speakers on separate audio channels and predicts their upcoming speech activity. It uses the recent conversation to estimate whether the current speaker will continue or hand over to the other person. Research on VAP shows that it learns to use cues in the voice, such as pitch and rhythm.

The version we ran was trained further on 104 hours of conversational audio from otoSpeech. Our interpretation is that this training and its view of both speakers help explain its strong result in our comparison.

SLURP: choosing from all available actions

SLURP contains spoken home-assistant requests with an annotated action and argument values. We are moving our command comparison to the original closed-set task: every recording offers all 60 intent labels, including the correct one. There is no missing-tool condition or no-action target in this run.

The first completed rerun uses local Parakeet transcripts and MiniLM to select the nearest labelled training example. It chooses the correct intent on 691 of the same 1,000 recordings: 69.1% overall, 349/480 (72.7%) on headset recordings and 342/520 (65.8%) on far-field recordings. Eleven empty transcripts count as errors. The cloud configurations below have not yet been rerun with all 60 choices, so these scores do not establish a local-versus-cloud ranking.

InstructTTSEval: delivery has its own failure modes

For InstructTTSEval, we keep the spoken script fixed and generate separate clips using three types of instruction:

• APS (Acoustic-Parameter Specification) gives detailed voice settings, such as pitch, pace, volume and vocal texture. For example: “Low pitch, slow pace, soft volume and a slightly breathy voice.”
• DSD (Descriptive-Style Directive) describes the delivery in natural language, leaving some voice traits open. For example: “Speak warmly and reassuringly.”
• RP (Role-Play) gives a character or situation and asks the model to infer how to sound. For example: “You’re a doctor comforting an anxious patient.”

The percentage for each type shows how often the generated speech followed that instruction.

The larger cohort contains 25 English and 25 Chinese source lines, three checks each: 150 clips per system. Nova covers the 25 English lines, with 75 checks. Overall is total passes divided by total checks; with equal task sizes, it is the mean of APS, DSD and RP. A saved Gemini Pro judge supplies the style verdicts, while transcription checks the spoken words separately.

Original recordings are the benchmark’s human source audio. We score them with the same instructions and judge used for the 150-check runs to give the synthesized voices a reference point. Their 73.3% means the judge accepted 110 of 150 instruction checks.

On the matched 150-check cohort, FireRedTTS3-Instruct scores 65.3% and MOSS-VoiceGenerator scores 56.0%. Google’s TTS run scores 67.3%. On the 75 English checks, FireRed scores 72.0%, MOSS 52.0% and Google 72.0%. VoxCPM2’s BF16 run remains at 59.3%; its role-play result, 46.0%, is weaker than its voice-description and direction results. The larger Qwen CustomVoice checkpoint scores 52.7%, versus 48.0% for the smaller experimental-input run. Size alone did not determine the score.

Our ElevenLabs instruction-to-tag conversion dropped requested singing and some emotion changes. CosyVoice3 failed the word-content check on 48 of 50 APS clips; its generation setup also omitted reference conditioning present in the official implementation. These issues still need corrected-input validation. Fish’s corrected run and several other rows contain judge responses that claimed audio was unavailable; we preserve those verdicts and mark the affected rows provisional. These are results for our ports, input adapters and judge, rather than a finding that an upstream model has the same limits in every implementation.

Most failures on the human recordings are in role-play: 29 of 50 checks fail, compared with eight for APS and three for DSD. A real recording can fit a detailed voice description yet not match the role assigned to it, and the judge can also make mistakes. The reference row helps us interpret instruction scores; it does not rate the quality of human speech.

Voice design can be a synthesis backend

Qwen’s VoiceDesign checkpoint generates a voice from a written description. CustomVoice instead applies directions to a selected preset voice. The checkpoint and control channel matter: a field named “instruction” does not give a model a capability it was never trained to use.

We tested Qwen3-TTS 1.7B VoiceDesign and Breeze TTS 2 on the same five English and five Chinese sources, giving 30 checks per model. Four calibration checks are outside their scores. We retained all outputs, including one spoken-word flag per model.

VoiceDesign passed 23/30 checks and Breeze passed 20/30. With ten source recordings shared across the three tasks, this is a development screen. The paired 10-point difference has a wide source-cluster interval that includes zero. The judge setting also differs from the larger cohort, so the pilot does not establish that VoiceDesign outperforms Google’s 67.3% result.

FireRedTTS3-Instruct and MOSS-VoiceGenerator now each have 150 style checks in the table above. The separate content check flagged 0/150 FireRed clips and 12/150 MOSS clips for word error above 30%. Their short-script runtime samples remain in the Space. First available PCM was about 4.50 s for FireRed and 0.47 s for MOSS on the warm English script; FireRed returns a complete waveform while MOSS streams. Those timings do not establish instruction quality or a matched quality-speed ranking. Existing-model latency and memory certification remains inconclusive.

Turn-taking research now asks for more than an endpoint

Research snapshot: 30 October 2026. The official TurnBench held-out board has newer external submissions beyond the paper’s original baselines. p99 turn-1-mini is listed at 97.5% recall, 7.5% false positives and 595 ms median latency. Its Apache 2.0 release includes a 5.3-million-parameter model and streaming ONNX graphs. That is an organiser-published test result we have not reproduced. Our small dev slice cannot rank against it.

LiveKit’s eot-bench complements this with real human-to-agent turns in 14 languages. It sweeps endpointing policies and compares latency at a fixed false-cutoff budget, including the full tradeoff curve. That is a useful next step for a multilingual product because a threshold that works in English need not suit another language.

MultiPartyTurnBench, submitted this month, expands floor control to two to five speakers. Its authors report strong turn-entry performance alongside near-chance selective interruption handling for the evaluated agents. Its generated conversations provide controlled tests, with real-audio checks needed alongside them. Meetings and shared rooms require more than extending a two-speaker endpoint rule.

Full-Duplex-Bench v3 pairs real disfluent speech with chained tool calls, exposing self-corrections and multi-step failures. DuplexAct, also submitted this month, evaluates interaction actions and whether the agent follows through. We see these as complementary tests for the gap between detecting an event and completing a useful interaction. We have not run these newer benchmarks in this post.

Private synthesis, directed dubbing and English voice choices

Soniqo’s private runtime and on-device SDKs provide building blocks for speech applications in your own environment. We work with teams building expressive synthesis and emotionally directed dubbing pipelines: retain the intended words, choose a voice, supply a delivery brief, then review the result in context.

You can plug a voice-design backend into the synthesis stage while keeping turn policy and tool execution separate. For a narrated scene, the application can specify calm reassurance or rising urgency rather than infer an emotion and change the performance without a director’s control. A dubbing workflow also needs timing, speaker continuity and a listening review; the three benchmarks above do not certify that complete workflow.

English voice choices include American and British preset voices in the Kokoro catalogue. Style-aware backends add natural-language voice briefs. Preset selection, voice design and cloning are distinct controls, and support varies by model. A requested regional accent needs evaluation with listeners familiar with that variety; transcription accuracy and speaker similarity do not establish authenticity.

With a local backend and model assets provisioned, synthesis can stay in the deployment environment. The exact deployment and model choice depend on the target hardware and the model’s terms. Research-only benchmark data and restricted model weights are not a blanket commercial licence.

The choices these measurements support

• Choose a turn policy by its operating point. In our dev slice, VAP and Mimi offer different recall, false-alarm and delay tradeoffs. Current held-out research has moved beyond those original baselines.
• Compare command models with the same choices. Our all-60-intent rerun starts at 69.1% for Parakeet → MiniLM. The earlier ten-candidate experiment also exposed incorrect argument values and missing fields; its scores describe a different task.
• Test delivery by instruction type. Role-play is a weaker area in several runs. The VoiceDesign pilot warrants a larger matched comparison, rather than a claim of a new overall winner.

Our next useful measurements are a larger VoiceDesign cohort with matched judging, multilingual endpointing at fixed cutoff rates, and interaction tests that carry a correction through to the final tool call. Human review of accent, naturalness and performance continuity belongs alongside the automatic scores for production voice work.

Read the live results and protocols, explore speech-swift, or discuss a private deployment. The tables above retain the run sizes and settings from the linked snapshot.