Speech-to-Speech Models

Soniqo supports two native MLX model families for conversational speech-to-speech: PersonaPlex 7B and VoiceChat 11B. Both consume speech and generate model audio directly, but they use different duplex architectures and runtime contracts.

Choose a model

These are conversational speech-to-speech models. Hibiki also maps speech directly to speech, but its task is streaming speech translation, so it is documented separately.

PersonaPlex 7B

Full-duplex speech-to-speech dialogue model based on the Moshi architecture (Kyutai). PersonaPlex 7B generates spoken responses directly from spoken input — no intermediate text pipeline required. The model ships with 18 voice presets and is available in 8-bit (recommended) and 4-bit quantization. 8-bit is the default — it is 30% faster and produces coherent responses, while 4-bit degrades output quality.

VoiceChat 11B

Duplex speech-to-speech reference: SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model (Interspeech 2025), cited by NVIDIA's upstream VoiceChat model card.

M5 Pro, 48 GB, Release, 80 ms live cadence, protected-head INT5: three controlled runs measured whole-pipeline RTF 0.94, 0.92, and 0.92, with about 8.70 GB peak RSS. First spoken text arrived in 44.3–46.7 ms and first playable audio in 74.0–76.4 ms. Those are hot compute figures, not learned turn-taking latency; optimized INT8 performance has not yet been remeasured.

VoiceChat is the other native MLX speech-to-speech runtime in speech-swift. It combines causal FastConformer perception, Nemotron-H duplex text/function channels, standalone text-conditioned EAR-TTS, and a 22.05 kHz neural codec. A session accepts 16 kHz mono audio continuously and returns text events plus one 1,764-sample output frame every 80 ms.

Use VoiceChatModel.load(from:), start a VoiceChatSession, and feed audio with pushAudio. A complete bundle must contain encoder/, llm/, and tts/; older understanding-only bundles are rejected.

Live VoiceChat CLI

Talk with Soniqo using the complete protected-head INT5 bundle, powered by NVIDIA Nemotron VoiceChat 11B. The command keeps capture and playback on one AVAudioEngine for Apple AEC, streams the bundle's RNN-T user captions, and plays 22.05 kHz model audio while continuing to listen.

The default Soniqo prompt is capability-grounded: this CLI can converse and answer questions, but it cannot access calendars, reminders, apps, accounts, devices, or external services. For those requests Soniqo clearly says it cannot perform the action, does not ask for a confirmation it cannot fulfill, and briefly suggests what the user can do instead.

speech voice-chat

speech voice-chat --input question.wav --output response.wav

Configurable MCP tools

MCP is disabled unless --mcp-config is supplied. The provider-neutral JSON file can launch any stdio MCP server and explicitly chooses which discovered tools reach VoiceChat. The deliberately narrow Apple Reminders example exposes list, create, and update workflows; custom configurations can enable other server tools:

speech voice-chat \
  --mcp-config Examples/VoiceChatMCP/apple-reminders.json

The bundled example pins an EventKit-backed provider behind an apple-reminders-eventkit adapter. VoiceChat exposes only the canonical list_reminders, create_reminder, and update_reminder operations—not the provider's delete, list-mutation, calendar, or subtask actions. List discovery stays private to the adapter. A reminder and its due date are committed in one EventKit save, avoiding the former partial-write failure that could leave an undated reminder behind. The provider may compile its small Swift helper on first use, so startup has a separate 60-second allowance; normal tool calls keep the 15-second default.

Only enabledTools are shown to the model, with a maximum of five per session. Tools are writes unless the configuration explicitly lists them in readOnlyTools. Writes default to --mcp-write-policy allow, so one complete native call executes immediately and its real provider result drives the reply; the reminders demo does not add a second confirmation turn. An identical completed write is still suppressed until fresh acoustic activity. Use confirm for explicit model-mediated confirmation or deny to disable writes.

There is no transcript router, regular-expression matcher, slot extractor, reminder draft state, argument repair, spoken-name resolver, or scripted assistant speech. The checkpoint alone selects the tool and arguments. The coordinator validates enabled-tool membership and JSON types, applies policy, calls MCP, and returns structured success or failure through the trained function-response channel. Failed provider results, invalid arguments, denied writes, and unavailable tools suspend new function starts until the checkpoint opens a spoken assistant turn; this prevents a silent immediate-retry loop while leaving later retries model-controlled. The Reminders facade requires only name for creation and flattens all EventKit lists in one list_reminders read. It presents long provider UUIDs as stable, session-scoped references such as r1, then resolves an exact model-returned reference before update_reminder execution. This is lossless ID compression, not transcript or intent matching. Missing or invalid values go back to the model for a natural clarification; they are never silently inferred from the displayed transcript.

Model-native call JSON runs asynchronously on a cached silence embedding. Once a call is open, the forced-PAD text channel skips its unused 131,072-row projection. After EOTC, the two-phase path freezes the shared language and EAR-TTS timeline while the external result is pending, matching NVIDIA's reference wrapper; microphone frames continue through perception and RNN-T but are not inserted as synthetic PAD positions. Known result tokens replay in bounded 16-token causal-prefill chunks. The session yields between call tokens and result batches, external MCP I/O runs in a separate task, and live microphone perception plus RNN-T captions remain eligible throughout. Retained user speech re-enters the shared timeline at no more than one old semantic frame per live callback; the current frame remains queued, elapsed replay audio is muted, and idle EAR-TTS positions synchronize later in bounded eight-frame chunks. The live loop reads coherent function and MCP status snapshots every 160 ms instead of making several actor calls after every microphone frame. The interactive dashboard reports the active tool phase; the structured benchmark records native decode, MCP round trip, result synchronization, model/cache compute, and live interleave residual separately. Ordinary conversation and barge-in keep NVIDIA's 40-frame safety fallback; an already-proposed native tool candidate may commit after eight blank frames.

On an idle M5 Pro with 48 GB, three native reminder-list phrases reached model-authored post-tool speech 3.56–4.00 seconds after end of input with a fixed 200 ms provider. Native JSON took 1.15–1.28 seconds for 27–31 steps; the 67-token result took 0.58–0.72 seconds in five prefills. A focused update regression required 67 native steps / 5.25 seconds when the model repeated a realistic EventKit UUID, versus 37 steps / about 2.29 seconds with the losslessly resolved r1 reference—45% fewer token steps and about 56% less decode time. A separate two-second-provider stress kept provider-wait microphone work at 19.6 ms p95 and 19.8 ms maximum, so external I/O adds reply delay without holding capture. A final clean run measured 118 ms microphone-service p95 and 139 ms p95 lateness beyond the 80 ms input deadlines; the latter exposes short result-sync queueing bursts hidden by foreground RTF. The local EventKit read itself measured 358 ms after one-time MCP startup. One tested create phrase was transcribed correctly but selected ordinary speech rather than a native call; this remains model tool-selection quality, not a hidden runtime router.

The concurrency boundary is the shared Nemotron-H KV cache: ordinary result-conditioned speech cannot form a second independent causal future while function tokens are being generated or replayed. Soniqo resumes coherent, model-authored speech after <SPECIAL_22> closes the tool response. Fully overlapping arbitrary speech would require a speculative second cache and merge policy, or a separately trained function decoder.

Child MCP servers inherit only PATH, HOME, temporary-directory, and locale variables. Credentials must be passed explicitly in the server's env map; unrelated parent-process secrets are not forwarded. The configured command still runs locally with the current user's filesystem and application permissions, so use only trusted MCP configuration files and server packages.

The 8-bit function projection is about 518 MB. During ordinary speech, VoiceChat evaluates only a cached 36 KB PAD/tool-start probe. If tool-start wins before the turn boundary, only its 4,096-value hidden-state candidate is retained; full-vocabulary verification runs when learned BOS arrives or the final RNN-T safety endpoint expires. For a generic native call, verified call JSON then runs at full language-model speed on a perception embedding cached from one second of silence instead of once per 80 ms microphone frame. Once an MCP result is known, VoiceChat replays it in bounded 16-token causal-prefill chunks, preserving the trained function feedback and language/speech cache state without one 11B decode per result token. The session yields between generated call tokens and response-prefill chunks so live microphone perception, RNN-T captions, and sustained-speech interruption detection continue; separate call and response bounds convert malformed PAD-only work into an explicit retry rather than a permanent stall.

Playback prebuffers three 80 ms frames (240 ms) by default. After a speaker gap, it waits for an eight-frame recovery cushion before restarting. Voice refinement drops from eight to two steps after the first 88 ms callback or when three frames queue; a 120 ms callback, six queued frames, or input resynchronization goes directly to one emergency step. Only acoustic refinement changes—perception, RNN-T, and language state remain intact. If the queue still reaches its default eight-frame/640 ms limit, only the minimum oldest microphone audio needed to admit fresh input is discarded. Sustained overload is treated as one recovery episode, and a user turn already confirmed by RNN-T remains armed instead of being erased by repeated resets. Skipped words are still marked and may require repetition.

The dashboard uses human-facing metrics: behind is queued microphone audio, RTF 0.93× is the rolling mean over the latest 120 microphone callbacks of wall time divided by 80 ms of audio (below 1 keeps up; above 1 falls behind), and last 74 ms for 80 ms audio means the latest callback took 74 ms. Replay events are not counted as extra audio denominators. Because RTF is a window, it can remain high briefly after one expensive operation even when last has recovered. Below the conversation, speaker gaps, old-audio skips, and microphone audio skipped make recovery explicit. PLEASE REPEAT appears only when microphone audio was actually skipped.

Microphone mode uses NVIDIA-style RNN-T turn-taking by default. The transcript head treats its first prediction on every 80 ms frame as blank or non-blank. Two initial speech frames and three on later turns remain sustained-activity fallbacks, while one recognized lexical token immediately arms an idle turn so a complete “yes” or “who are you?” cannot appear in the caption and then wait for more speech. Punctuation and unknown labels do not use that shortcut. VoiceChat's learned BOS/EOS predictions remain the normal low-latency decisions. Forty blank frames (3.2 seconds) force BOS only if the learned head never opens a response, and forty fresh non-blank frames provide the matching RNN-T EOS safety fallback. The speech counter resets when Soniqo starts, so activity from the completed user turn cannot immediately cut the response. Every audio frame still reaches the duplex model; use --no-rnnt-turn-taking to leave BOS/EOS decisions entirely to the learned language head. The dashboard reports RNN-T barge-ins separately from playback underruns.

The RNN-T frontend matches NeMo's reflection padding at each rolling microphone-window edge. This removes a zero-padding mismatch that could alter marginal greedy token decisions; it does not replace the checkpoint's compact 1,024-token transcript head with a general large-vocabulary ASR.

In normal mode, model-native BOS is suppressed until user speech is confirmed, so Soniqo does not speak first. --greet explicitly permits the initial greeting. After audible reply content, blank PAD stays inside the open turn for at least 16 frames and for three frames per content token when that is longer. The first excess blank PAD closes the logical turn. This content-scaled tail preserves delayed words that a fixed 1.28-second cutoff can mute.

Interactive mode uses a fixed dashboard in the terminal's alternate screen, so status updates redraw in place instead of filling scrollback. Use --plain for append-only output.

For local diagnosis, --debug-timeline timestamps each user and Soniqo phrase and inserts the parsed native call arguments, MCP start/completion, and result-cache synchronization into the same chronology. It also marks pronunciation ended at the final audible generated-PCM window; this is model-output time, not speaker-device completion. The demo intentionally omits historical phase-timing blocks; detailed native-decode, provider, cache, and microphone-pressure metrics remain in voicechat-bench JSON. The timeline can expose tool arguments, so do not use it in shared logs. Phase timers reset between native decode and provider wait.

Live sessions keep the immutable 37-frame speaker prompt plus 20 seconds of recent EAR-TTS history, while Nemotron-H keeps semantic conversation state separately. PAD inside the content-scaled acoustic tail is rendered normally; only later silent PAD advances causally in batches of eight. This prevents TTS attention and idle work from growing without bound without cutting the spoken reply; --live-speech-context-seconds 0 restores full TTS history, and file mode remains exact full-history by default.

In a 63.6-second M5 Pro profile with the content-safe tail, the live path measured aggregate RTF 0.87, final-window RTF 0.71, 4.1 ms/frame final-window synthesis, 8.72 GB peak RSS, and 23.67 GB peak physical footprint. Speech-active eight-step windows reached RTF 1.05 and 1.12, so the interactive CLI uses a reversible two-step fallback and one-step emergency mode. One-step synthesis assigns all 31 RVQ codebooks in a single pass; it protects live capture but is not acoustic parity with the eight-step schedule. Sequential and batched idle-cache states had minimum cosine 0.9999999.

For audio diagnosis, the benchmark compares the bounded live codec-window render with a continuous full-context render. A long pause present in both was generated by EAR-TTS or the codec, not by the speaker queue; a discontinuity found only in the live render points toward codec-window stitching. Speaker gaps remain a separate playback-underflow metric. A dedicated onset-pop detector scans the generated lead-in plus the first 50 ms of sustained speech and flags sample jumps above both 0.05 amplitude and six times the steady-speech p99 baseline. Scenarios can set maximum_suspect_onset_transients to zero, while perceptual listening remains the final quality check.

PersonaPlex architecture and usage

PersonaPlex is a multi-stream autoregressive model with three core components:

The model processes 17 streams simultaneously: 1 text stream + 8 user audio streams + 8 agent audio streams. This architecture enables full-duplex conversation where the model can listen and speak at the same time.

Voice Presets

PersonaPlex includes 18 built-in voice presets across natural and varied styles:

Inner Monologue

PersonaPlex generates two parallel streams at every step: 8 audio codebook tokens for the Mimi codec and one text token for the model’s internal monologue. The text stream is what the model is “thinking” as it speaks — it can diverge slightly from the final audio, but in practice it mirrors the spoken response closely enough to use as a live transcript.

The text tokens come back as raw SentencePiece piece IDs. Decode them with the SentencePieceDecoder that PersonaPlex ships:

import PersonaPlex
import AudioCommon

let model = try await PersonaPlexModel.fromPretrained()
let decoder = model.tokenizer  // SentencePieceDecoder?

let result = model.respond(userAudio: userSamples, voice: .NATM0)
let transcript = decoder?.decode(result.textTokens) ?? ""
print(transcript)        // "Sure, I can help with that..."
playAudio(result.audio)  // 24 kHz mono Float32

In streaming mode, respondStream emits textTokens chunks as they are produced — decode them incrementally to drive a live caption view while the audio is still generating. The --transcript CLI flag does exactly this behind the scenes.

Why it matters: SentencePieceDecoder is built on the shared AudioCommon.SentencePieceModel protobuf reader, so PersonaPlex, OmnilingualASR and any future SentencePiece-based model decode through the same tokenizer implementation. See the SentencePieceModel reference.

System Prompts

Pass any custom system prompt as a plain string — no external tokenization needed:

let response = model.respond(
    userAudio: audio,
    voice: .NATM0,
    systemPrompt: "You enjoy having a good conversation."
)

Or use a built-in preset:

• assistant — General-purpose helpful assistant (default)
• focused — Concise, direct responses
• customer-service — Polite, solution-oriented support agent
• teacher — Patient, explanatory teaching style

CLI Usage

Generate a spoken response from an audio input:

# Basic speech-to-speech
.build/release/speech respond --input question.wav

# Choose a voice preset
.build/release/speech respond --input question.wav --voice NATM0

# Stream audio output during generation
.build/release/speech respond --input question.wav --stream

# Custom system prompt text
.build/release/speech respond --input question.wav --system-prompt-text "You enjoy having a good conversation."

# Use a preset system prompt
.build/release/speech respond --input question.wav --system-prompt customer-service

# Get transcript alongside audio
.build/release/speech respond --input question.wav --transcript

# JSON output with metadata
.build/release/speech respond --input question.wav --json

Options

Sampling parameters can also be overridden:

Streaming

The --stream flag enables real-time audio output. Audio chunks are emitted as they are generated, so playback can begin before the full response is complete. This is particularly useful for interactive applications where low latency matters.

Performance

Model Variants

Swift API

import PersonaPlex
import AudioCommon

let model = try await PersonaPlexModel.fromPretrained()
let response = model.respond(
    userAudio: userSamples,
    voice: .NATM0,
    systemPrompt: "You are a helpful assistant."
)
try WAVWriter.write(samples: response.audio, sampleRate: 24000, to: URL(filePath: "answer.wav"))