Use case · Conversational

Voice in.
Voice out.

Three shapes of voice-first interfaces — native full-duplex speech-to-speech models, a compositional wake → VAD → ASR → LLM → TTS pipeline you fully control, and wake-word activation for hands-free entry. All on-device, no cloud APIs, no audio leaving the device.

Get started Speech-to-speech guide

Three sub-use-cases

Pick the shape that fits your product.

Drop-in dialogue model, compositional pipeline with per-stage control, or a thin wake-word trigger. Each runs entirely on-device.

Full-duplex speech-to-speech

Native models stream microphone audio to spoken output without a classical ASR → LLM → TTS turn boundary. Choose VoiceChat’s asymmetric stack or PersonaPlex’s Moshi-family streams.

VoiceChat RTF 0.92–0.94 on M5 Pro · PersonaPlex RTF ~1.4 on M2 Max

Compositional voice pipeline

AEC → enhance → VAD → STT → LLM → TTS → audio, with a five-state turn detector, deferred interruption handling, and tool-call loop. The canonical orchestrator lives in speech-core.

State machine + interruption handling documented in `speech-core/docs/pipeline.md`

VoicePipeline

TurnDetector

STT/LLM/TTS interfaces

Open in speech-core

Wake-word activation

Hands-free trigger for any voice flow. Custom keywords with per-phrase thresholds, plug into the same VAD pipeline as compositional agents.

~4 MB INT8 on Neural Engine · 26× real-time · 88% recall on 12 keywords

KWS-Zipformer-3M

WakeWordProvider

Learn more

Native speech-to-speech

Two model families, two duplex designs.

Both accept continuous speech and generate speech without a classical ASR → LLM → TTS turn boundary. Text can still exist inside the model as an aligned reasoning and control stream.

SALM-Duplex lineage · asymmetric

VoiceChat 11B

Causal FastConformer perception feeds Nemotron-H text and function channels. Speech is rendered by a separate text-conditioned EAR-TTS decoder — not a head on the language model — and a 22.05 kHz neural codec.

Architecture

16 kHz → causal FastConformer

Nemotron-H · text + function

separate EAR-TTS → 22.05 kHz codec

Measured locally

RTF 0.92–0.94

M5 Pro · 48 GB · Release · protected-head INT5

44.3–46.7 ms first spoken text token · 74.0–76.4 ms first playable audio

76.2–78.6 ms frame p95 · ~8.70 GB peak RSS

Moshi family · multi-stream

PersonaPlex 7B

PersonaPlex starts from Moshi weights and jointly models user audio, agent text, and agent audio through Mimi and Depformer streams. It supports simultaneous listening and speaking, text role prompts, and audio voice prompts.

Architecture

Moshi weights

Mimi RVQ + Depformer

user audio ↔ agent text ↔ agent audio

Measured locally

RTF ~1.4

M2 Max · 8-bit

~112 ms / step · 18 selectable voices

~9.1 GB bundle · ~11 GB peak RAM

RTF is wall-clock generation time divided by output-audio duration; below 1 can sustain streaming on that tested device. These runs use different Macs, so they are not a model ranking. First output and frame throughput are hot compute measurements, not learned turn-taking latency.

Relevant research

Architecture lineage and duplex evaluation.

BESTOW captures NVIDIA’s preceding streamable SpeechLLM lineage, but it maps speech to text; it is not a duplex S2S model. Moshi and PersonaPlex describe the multi-stream family, while SALM-Duplex describes efficient asymmetric duplex modeling. Full-Duplex-Bench evaluates pauses, backchannels, turn-taking, and interruptions — interaction behavior that RTF does not capture.

Compare setup, APIs, bundles, and benchmarks

speech voice-chat · recorded session

Full duplex on a MacBook, calling real tools.

A real session running NVIDIA's Nemotron VoiceChat 11B locally: a spoken turn, an interruption mid-sentence, and a complete MCP tool call into Apple Reminders. RTF 0.92 on an M5 Pro, 7.5 GB resident, nothing leaving the machine.

Watch on YouTube Read the VoiceChat guide

New on device

Wait, speak and act, locally.

Recent work on the three decisions a voice agent makes: when to wait, when to speak, and when to act. All of it runs on the device.

Full duplex that calls tools

VoiceChat keeps listening while it answers, and its function head turns a spoken request into an MCP tool call. On a Mac it lists, creates and updates Apple Reminders, with nothing leaving the machine.

Read the write-up

Waiting for the end of the thought

Smart Turn v3.2 asks whether the speaker has finished before a request is committed, so “uh, Friday” can still replace Thursday. The 23-language model runs on CoreML and Swift, in the speech-core C++ runtime and on Android, in a few milliseconds on an M5 Pro.

Smart Turn guide

A command loop in 1.2 GB

On Android, speech becomes a real device action and a spoken reply on one phone: Silero VAD, Parakeet STT, FunctionGemma tool calls and Pocket TTS, fully offline in 1.2 GB of RAM.

Watch the demo

Quickstart · PersonaPlex

Use OpenAI Realtime clients locally.

For PersonaPlex, speech-server exposes /v1/realtime — the same session.update / input_audio_buffer.append / response.audio.delta event shape as the OpenAI Realtime API. Existing clients keep working when pointed at the local URL. VoiceChat uses the native Swift session API documented in the guide.

brew install speech
# Start the local Realtime-compatible server (PersonaPlex backend)
speech serve --model personaplex --port 8080

# In your existing OpenAI Realtime client, swap the base URL:
#   wss://api.openai.com/v1/realtime  →  ws://localhost:8080/v1/realtime
# session.update / input_audio_buffer.append / response.audio.delta — identical event schema

Quickstart · Compositional

Compose your own pipeline in Swift.

The Apple-side VoicePipeline (in speech-swift) wraps the same orchestrator. Feed it any conforming STT / LLM / TTS / VAD implementation; it runs the five-state turn detector, fires events for each transition, and handles deferred interruption when the user speaks over the agent.

import SpeechCore
import SilereoVAD
import ParakeetSTT
import Qwen3Chat
import KokoroTTS

let pipeline = VoicePipeline(
    vad: SileroVAD.streaming(),
    stt: try await ParakeetSTT.fromPretrained(),
    llm: try await Qwen35Chat.fromPretrained(systemPrompt: "..."),
    tts: try await KokoroTTSModel.fromPretrained(voice: "af_alloy"),
    config: .init(
        maxUtteranceDuration: 15,
        minInterruptionDuration: 0.4
    )
)

for await event in pipeline.events {
    switch event {
    case .userSpeechStarted: print("listening")
    case .transcriptionCompleted(let text): print("user:", text)
    case .responseAudioDelta(let chunk): player.enqueue(chunk)
    case .interruption: player.cancel()
    default: break
    }
}

Read the full state machine, AEC integration, and tool-call loop in speech-core/docs/pipeline.md ↗.

On-device performance

Hardware-qualified local measurements.

VoiceChat and PersonaPlex were measured on different Macs, so these figures are capacity checks rather than a cross-model ranking. RTF below 1 is the threshold for sustained real-time output.

VoiceChat 11B

RTF 0.92–0.94

M5 Pro · protected-head INT5 · ~8.70 GB peak RSS

PersonaPlex 7B

RTF ~1.4

M2 Max · 8-bit · ~112 ms/step · ~11 GB peak RAM

Streaming Dictation

340 ms partial

Parakeet-EOU · 25 langs · 30 ms compute / chunk

Silero VAD v6.2.1

sub-ms / chunk

CoreML · streaming