Smart Turn — End-of-Turn Detection

Smart Turn v3.2 answers one question after the VAD reports a pause: has the speaker finished, or are they mid-sentence? It listens to the audio itself — prosody, pace, intonation — rather than a transcript, covers 23 languages, and runs once per pause on the last 8 seconds of the turn. The weights are the official Pipecat Smart Turn v3.2 release (BSD-2-Clause), compiled to CoreML for Apple platforms and exported to ONNX for Speech Core.

How it works

Smart Turn is not a VAD and does not detect speech on its own. It is paired with the streaming Silero VAD, which decides when to ask. With a classifier attached, a confirmed pause becomes a question rather than a verdict:

1. Ask once per pause. When Silero confirms a pause (minSilenceDuration after the offset threshold), the processor hands the classifier the audio of the turn so far — from 0.5 s before the VAD onset up to the current chunk, capped at the last 8 seconds.
2. Complete. Probability at or above threshold (0.5): .speechEnded fires as usual.
3. Hold. Below the threshold the segment stays open. If the speaker resumes, the same segment continues — no second .speechStarted — and the next confirmed pause asks again with the longer turn.
4. Silence cap. If silence reaches maxSilenceDuration (2.0 s, measured from the start of the pause), the segment ends anyway, so a speaker who trails off still gets a reply. The segment's endTime is where the speech stopped, not when the cap expired.

flush() settles a held segment at end of stream and reset() clears it. A classifier that throws counts as "complete", so a failing model never stalls the conversation. isHoldingTurn reports the hold state and lastTurnCompletionProbability the most recent answer.

Model

The exports embed the Whisper log-mel front-end, including the waveform normalisation the upstream model was trained with, so callers pass raw audio — there is no feature extraction in Swift or C++.

Performance

Accuracy is measured on 1,000 clips from the upstream test set; the CoreML and fp32 ONNX exports match the upstream fp32 model exactly on those clips. Latency is one 8-second window on an Apple M5 Pro. One call per pause adds nothing noticeable to the pause the VAD already waited for.

Swift API

One-shot check on a recorded utterance:

import SpeechVAD

let turn = try await SmartTurnModel.fromPretrained()
try turn.prewarm()   // compile the graph before the first real pause

let probability = try turn.turnCompleteProbability(
    audio: utterance, sampleRate: 16_000)
if probability >= SmartTurnModel.defaultThreshold {
    // finished turn — reply now
}

Attached to the streaming VAD, so every confirmed pause is checked before a segment ends:

let vad = try await SileroVADModel.fromPretrained(engine: .coreml)
let turn = try await SmartTurnModel.fromPretrained()

let processor = StreamingVADProcessor(
    model: vad,
    config: .sileroDefault,
    turnCompletion: turn,
    turnCompletionConfig: TurnCompletionConfig(threshold: 0.5, maxSilenceDuration: 2.0))

for chunk in microphoneChunks {            // Float32 @ 16 kHz
    for event in processor.process(samples: chunk) {
        switch event {
        case .speechStarted(let time): print("speech at \(time)s")
        case .speechEnded(let segment): print("turn \(segment.startTime)–\(segment.endTime)s")
        }
    }
}
let remaining = processor.flush()

SmartTurnModel conforms to the TurnCompletionProvider protocol in AudioCommon, so any other classifier with the same shape can be attached instead. fromPretrained(cacheDir:offlineMode:) follows the same cache and offline rules as the other models.

VoicePipeline

VoicePipeline takes the same classifier, so a voice agent stops cutting people off without any change to its STT, TTS or VAD setup. Two PipelineConfig fields and one method wire it in; they require speech-swift v0.0.27 or later (SpeechCore.xcframework from speech-core v0.0.14).

import SpeechCore
import SpeechVAD

let smartTurn = try await SmartTurnModel.fromPretrained()
try smartTurn.prewarm()

var config = PipelineConfig()
config.turnCompletionThreshold = 0.5   // pause ends the turn when p >= this
config.turnCompletionMaxSilence = 2.0  // seconds of silence that end a held turn anyway; 0 = never

let pipeline = VoicePipeline(stt: stt, tts: tts, vad: vad, config: config, onEvent: { print($0) })
pipeline.setTurnCompletion(smartTurn)   // before start(); nil detaches
pipeline.start()

With a classifier attached, a VAD pause only ends the user's turn when the probability reaches the threshold; below it the pipeline keeps listening, speech that resumes continues the same turn (no second speechStarted), and turnCompletionMaxSilence ends the held turn anyway. Eager STT respects the veto, and the classifier runs once per pause on the audio thread, so call prewarm() after loading.

Tuning

Silero's minSilenceDuration still decides when a pause is confirmed, and therefore when the classifier is asked. Call prewarm() after loading so the first real pause does not pay for graph compilation.

CLI Usage

# Probability, complete/incomplete verdict and inference time for one utterance
speech turn utterance.wav

# Custom threshold, JSON output
speech turn utterance.wav --threshold 0.7 --json

# Local bundle, no download
speech turn utterance.wav --model-dir ~/smart-turn-coreml

# Streaming VAD with Smart Turn confirming each pause; mid-sentence pauses merge
speech vad-stream call.wav --smart-turn
speech vad-stream call.wav --smart-turn --turn-threshold 0.6 --turn-max-silence 1.5

speech turn loads the file (any sample rate), scores the last 8 seconds and prints the probability, whether it clears the threshold, and the inference time. Use it to tune the threshold on your own recordings. The full flag list is in the CLI reference.

C++ / Android

Speech Core ships the same model as OnnxSmartTurn for Linux, Windows, and Android (ONNX Runtime only; there is no LiteRT variant). VoicePipeline::set_turn_completion() attaches it before start(); a VAD pause then only ends the user's turn when the probability reaches turn_completion_threshold (0.5), and turn_completion_max_silence (2.0 s) ends a held turn anyway. The C API exposes the same hook as sc_turn_completion_vtable_t + sc_pipeline_set_turn_completion(), so Kotlin or Swift hosts can bridge their own classifier the same way as the VAD.

#include <speech_core/models/onnx_smart_turn.h>

speech_core::OnnxSmartTurn model("/models/smart-turn-v3.2-int8.onnx");

// Audio of the turn so far at any rate (resampled to 16 kHz internally).
float p = model.turn_complete_probability(turn.data(), turn.size(), rate);

// Or let the pipeline ask it on every VAD pause (before start()).
pipeline.set_turn_completion(&model);

The speech CLI of Speech Core scores a recording the same way as on macOS:

speech download-models        # fetches the int8 graph with the other ONNX models
speech turn recording.wav --threshold 0.5 --json

Model files, the C API contract and the CLI are documented on the Speech Core page and in the repository's docs/models.md.

Model Downloads

Source

• Sources/SpeechVAD — SmartTurnModel and StreamingVADProcessor
• docs/models/smart-turn-v3.md — model notes
• docs/inference/smart-turn.md — inference and tuning
• speech-core docs/models.md — OnnxSmartTurn
• Upstream: pipecat-ai/smart-turn / pipecat-ai/smart-turn-v3 (Daily / Pipecat, BSD-2-Clause)