Architecture

Breeze TTS 2 · BreezeTTS

English/Chinese TTS with voice design, reference cloning and direction, and streamed audio (MLX BF16/8-bit, 24 kHz; research/non-commercial license).

BreezeTTS · CLI · Swift API

speech speak "The quick brown fox jumps over the lazy dog." --engine breeze \
  --instruct "A clear, calm English narrator." --stream -o design.wav

--breeze-model-id: aufklarer/Breeze-TTS-2-MLX-8bit
--breeze-bundle-dir
--breeze-reference-text
--breeze-temperature: 0.9
--breeze-top-k: 50
--breeze-top-p: 1
--breeze-cfg-scale: 4
--breeze-max-frames: 750
--breeze-seed: 42
--first-chunk-frames: 3
--chunk-frames: 25

Native Apple runtime · PocketTTS

PocketTTS adds native CoreML speech synthesis to speech-swift. All stages default to CPU-only execution. The English Alba preset streams 1,920-sample frames at 24 kHz (80 ms per frame).

Swift and CLI usage

speech-swift is organized as a modular Swift package with shared protocols, independent model modules, and a unified CLI. All inference runs on-device using MLX (Metal GPU) or CoreML (Neural Engine).

Module Dependency Graph

                    ┌──────────┐
                    │ AudioCLI │  (entry point)
                    └────┬─────┘
                         │
                  ┌──────┴──────┐
                  │ AudioCLILib │  (commands)
                  └──────┬──────┘
                         │
       ┌─────────┬───────┼───────┬──────────┬──────────────┐
       │         │       │       │          │              │
  ┌────┴───┐ ┌──┴──┐ ┌──┴──┐ ┌─┴────┐ ┌───┴────┐ ┌──────┴───────┐
  │Qwen3ASR│ │Qwen3│ │Cosy │ │Perso-│ │Speech- │ │  Speech-     │
  │Parakeet│ │ TTS │ │Voice│ │naPlex│ │  VAD   │ │Enhancement   │
  └────┬───┘ └──┬──┘ └──┬──┘ └──┬───┘ └───┬───┘ └──────┬───────┘
       │        │       │       │         │             │
       └────────┴───────┼───────┴─────────┘             │
                        │                               │
                 ┌──────┴──────┐                        │
                 │ Qwen3Common │  (shared layers)       │
                 └──────┬──────┘                        │
                        │                               │
                 ┌──────┴──────┐                        │
                 │ AudioCommon │ ◄──────────────────────┘
                 └─────────────┘  (protocols, audio I/O)

MLX buffer cache budget

MLX model runtimes share a default reusable-buffer cap of min(2 GiB, physical RAM / 4) from weight loading through the model lifetime. Lower caller budgets, including MLX.Memory.cacheLimit = 0, and existing model-specific budgets are preserved. The cap limits idle buffers; active weights, KV caches and total process memory can exceed it. MLX trims on allocation, so the last freed buffer may temporarily put the cache above its budget until the next allocation.

unload() releases the model’s registration; runtimes without that method release it on destruction. The last owner restores the caller’s budget, while earlier unloads preserve the remaining models’ ceilings. CoreML-only paths do not acquire MLX registrations.

MOSS Transcribe Diarize

MossTranscribe returns speaker-attributed timestamp segments through two native backends. CoreML uses a fixed 1,024-token MLState for short recordings. MLX uses a dynamically growing 131,072-token context, affine INT5/INT8 decoder weights, and optional FP16/INT8 KV caches for globally contextualized long-form transcription. Both are offline, autoregressive runtimes rather than streaming models.

VoiceChat 11B

Task and reference: duplex speech-to-speech; SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model (Interspeech 2025), cited by NVIDIA's upstream VoiceChat model card.

Complete runtime, M5 Pro 48 GB, Release, 80 ms live cadence: protected-head INT5 peak RSS is about 8.69 GB; three controlled whole-pipeline RTF results were 0.96, 0.96, and 0.99. Optimized INT8 performance has not yet been remeasured.

VoiceChat implements the complete native MLX audio-to-audio path: causal FastConformer perception, the Nemotron-H duplex text/function channels, the standalone text-conditioned EAR-TTS decoder with eight MaskGIT iterations, and a 22.05 kHz neural codec. Streaming retains bounded per-layer attention and causal-convolution caches, while the steady eight-frame live codec window uses a compiled graph. Every 80 ms input frame produces a language decision, 31 RVQ codes, and exactly 1,764 output samples. The loader requires a complete encoder/ + llm/ + tts/ bundle and rejects older understanding-only exports. INT5 now sustains aggregate RTF below 1 on the tested M5 Pro, although per-frame p95 headroom remains narrow.

Inference Backends

Backend	Hardware	Models
MLX	Metal GPU	Qwen3-ASR, Qwen3-TTS, Breeze TTS 2, CosyVoice3, IndexTTS2, Qwen3.5-Chat, PersonaPlex, VoiceChat 11B, Omnilingual ASR (300M / 1B / 3B / 7B), Cohere Transcribe 2B, Voxtral Mini 3B, MOSS Transcribe Diarize, Pyannote, Silero VAD, WeSpeaker, SpeechBrain ECAPA LID, Nemotron 3 Diarization
CoreML	CPU / GPU / Neural Engine (ANE)	Qwen3-ASR encoder (hybrid), Parakeet TDT, Parakeet EOU streaming, Omnilingual ASR 300M, Kokoro-82M, Qwen3.5-Chat (optional), Sortformer diarization, DeepFilterNet3, Silero VAD (optional), WeSpeaker (optional), MOSS Transcribe Diarize, SpeechBrain ECAPA LID, Nemotron 3 Diarization
Accelerate	CPU (SIMD)	Audio preprocessing (STFT, mel, FFT), signal processing

Model Weight Format

MLX models use safetensors format with 4-bit or 8-bit quantization (group size 64). CoreML models use .mlmodelc compiled format. Conversion scripts in scripts/ convert from PyTorch checkpoints.

Cohere and Voxtral also use MLX affine INT5 bundles. MLX does not support affine INT7; INT8 is the supported higher-quality option.

Model	Params	Quantization	Size on Disk
Qwen3-ASR 0.6B (MLX)	~600M	4-bit / 8-bit	680 MB / 1.0 GB
Qwen3-ASR 0.6B (CoreML)	~186M (encoder)	INT8	~180 MB
Qwen3-ASR 1.7B (MLX)	~1.7B	4-bit / 8-bit	2.1 GB / 3.2 GB
Parakeet-TDT 0.6B (CoreML)	~600M	INT8	500 MB
Whisper Large-v3 Turbo (CoreML)	809M	FP16	1.6 GB
MOSS Transcribe Diarize 0.9B (MLX)	908.5M	FP16 audio + INT5 / INT8 decoder	987.0 MiB / 1,200.1 MiB
MOSS Transcribe Diarize 0.9B (CoreML)	~900M	FP16 / INT8 block-32	1.7 GB / 1.2 GB
Parakeet-EOU 120M (CoreML)	~120M	INT8	~120 MB
Omnilingual-ASR-CTC 300M (CoreML)	326M	INT8	312 MB
Omnilingual-ASR-CTC 300M (MLX)	326M	4-bit / 8-bit	193 MB / 342 MB
Omnilingual-ASR-CTC 1B (MLX)	1.01B	4-bit / 8-bit	549 MB / 1006 MB
Omnilingual-ASR-CTC 3B (MLX)	~3B	4-bit / 8-bit	1.71 GB / 3.16 GB
Omnilingual-ASR-CTC 7B (MLX)	~7B	4-bit / 8-bit	3.55 GB / 6.63 GB
Cohere Transcribe 2B (MLX)	2B	FP16 / INT5 / INT8	3.85 / 1.62 / 2.25 GiB
Voxtral Mini 3B 2507 (MLX)	3B	FP16 / INT5 / INT8	8.71 / 3.77 / 5.18 GiB
Qwen3-ForcedAligner 0.6B (MLX)	~600M	4-bit / 8-bit	979 MB / 1.4 GB
Qwen3-ForcedAligner 0.6B (CoreML)	~600M	INT4 / INT8	630 MB / 1.0 GB
Qwen3-TTS 0.6B (MLX)	~600M	4-bit / 8-bit	1.7 GB / 2.4 GB
Qwen3-TTS 1.7B (MLX)	~1.7B	4-bit / 8-bit	3.2 GB / 4.8 GB
CosyVoice3 0.5B (MLX)	~500M	4-bit LLM	~1.2 GB
IndexTTS2 expanded bundle (MLX)	1.5B-class	FP16 + auxiliary models	~4.8 GB
Kokoro-82M (CoreML)	82M	INT8 (1 bucket)	~89 MB
Qwen3.5-Chat 0.8B (MLX)	~800M	INT4	418 MB
Qwen3.5-Chat 0.8B (CoreML)	~800M	INT8	981 MB
PersonaPlex 7B (MLX)	~7B	4-bit / 8-bit	4.9 GB / 9.1 GB
VoiceChat 11B (MLX)	~11B	INT5 / INT8	8.56 GB / 12.11 GB
Pyannote VAD (MLX)	~1.49M	float32	~5.7 MB
Silero VAD v5	~309K	float32	~1.2 MB (MLX & CoreML)
WeSpeaker ResNet34	~6.6M	float32	~25 MB (MLX & CoreML)
SpeechBrain ECAPA VoxLingua107	21.25M	FP16	~40.6 MiB (MLX) / ~40.8 MiB (CoreML)
ReDimNet2-B6	12.3M	float16	~25 MiB (CoreML)
Pyannote Community-1 (CoreML)	8.35M	float32 + native VBx	~32 MB
Nemotron 3 Diarization (CoreML / MLX)	100M	INT8	~100 MB per backend
Sortformer (CoreML)	—	float16	~50 MB
DeepFilterNet3 (CoreML)	~2.1M	FP16	~4.2 MB
LocalVQE v1.4-AEC (CoreML + C++)	200K + 2,742	FP16 + FP32	~732 KB

MLX model initialization

QuantizedLinear.preQuantized creates zero placeholders for checkpoint-loaded quantized linear layers, avoiding random weights and their temporary quantization graphs. withThrowawayRandomState keeps discarded floating-point initialization off the caller’s random-key chain. Composite loaders scope every component: MAGNeT’s T5 and EnCodec, Stable Audio 3’s T5Gemma and SAME-L, and CSM’s Mimi codec. CSM’s quantized embeddings still use the upstream initializer and retain its known initialization-graph limitation.

Performance Optimizations

MLX compilation — Qwen3-TTS compiles the Talker over its growing valid-prefix cache and compiles the fixed 15-group Code Predictor frame as one graph for single-item synthesis.
Metal shader library — Pre-compiled metallib avoids ~5x JIT compilation overhead. Built via scripts/build_mlx_metallib.sh.
Chunked codec decode — TTS decoder processes speech in 25-frame chunks with 10-frame context overlap to avoid GPU timeout.
Batch-doubled CFG — CosyVoice3 DiT halves flow matching passes by batching conditional + unconditional together.
Fused RoPE — Uses MLXNN.RoPE backed by Metal kernel instead of manual rotation.
BN fusion — WeSpeaker batch normalization fused into Conv2d weights at conversion time.

Audio Processing

All audio I/O uses Float32 PCM. Internal resampling handles format conversion:

Model	Expected Rate	Format
Qwen3-ASR	16 kHz	Mono Float32
Cohere Transcribe 2B	16 kHz	Mono Float32
Voxtral Mini 3B	16 kHz	Mono Float32
Qwen3-TTS	24 kHz output	Mono Float32
CosyVoice3	24 kHz output	Mono Float32
Kokoro-82M	24 kHz output	Mono Float32
PersonaPlex	24 kHz I/O	Mono Float32
Pyannote VAD	16 kHz	Mono Float32
Silero VAD	16 kHz	Mono Float32
WeSpeaker	16 kHz	Mono Float32
SpeechBrain ECAPA LID	16 kHz	Mono Float32
ReDimNet2-B6	16 kHz	Mono Float32
DeepFilterNet3	48 kHz	Mono Float32

Source Structure

Sources/
  AudioCommon/            Shared protocols, audio I/O, system-output capture
                          (SystemAudioTap), HuggingFace downloader,
                          SentencePieceModel (protobuf reader)
  MLXCommon/              MLX utilities: weight loading, QuantizedLinear helpers,
                          SDPA multi-head attention helper, metal budget
  Qwen3Common/            Shared model components (KV cache, RoPE, quantization)
  Qwen3ASR/               Qwen3-ASR speech-to-text
  ParakeetASR/            Parakeet TDT speech-to-text (CoreML)
  ParakeetStreamingASR/   Parakeet EOU 120M streaming dictation (CoreML)
  OmnilingualASR/         Meta wav2vec2 + CTC, 1,672 languages
                          (CoreML 300M + MLX 300M / 1B / 3B / 7B)
  CohereTranscribeASR/  Cohere Transcribe 2B speech-to-text (MLX)
  VoxtralASR/            Voxtral Mini 3B speech-to-text (MLX)
  Qwen3TTS/               Qwen3-TTS text-to-speech
  CosyVoiceTTS/           CosyVoice3 text-to-speech
  KokoroTTS/              Kokoro-82M text-to-speech (CoreML)
  Qwen3Chat/              Qwen3.5-0.8B on-device LLM chat (MLX + CoreML)
  PersonaPlex/            PersonaPlex speech-to-speech
  SpeechVAD/              VAD (Silero + Pyannote), diarization, speaker embeddings
  SpeechLanguageID/       Spoken language identification (SpeechBrain ECAPA, MLX + CoreML, 107 labels)
  SpeechEnhancement/      DeepFilterNet3 noise suppression (CoreML)
  SourceSeparation/       Open-Unmix HQ / UMX-L + HTDemucs (Demucs v4) music source separation (MLX, 4 stems)
  MAGNeTMusicGen/         MAGNeT text-to-music generation (MLX, 30 s @ 32 kHz, INT4/INT8)
  VoxCPM2TTS/             VoxCPM2 text-to-speech (MLX, 48 kHz, voice cloning + voice design)
  IndexTTS2TTS/           IndexTTS2 voice cloning (MLX, semantic GPT + emotion/tempo/pause control + S2Mel + BigVGAN)
  VibeVoiceTTS/           VibeVoice long-form / multi-speaker TTS (MLX, EN/ZH)
  MagpieTTS/              NVIDIA Magpie-TTS Multilingual 357M (MLX INT4/INT8,
                          9 languages, 5 baked speakers, streaming)
  MagpieTTSCoreML/        Magpie CoreML backend (4 .mlmodelc + Swift-side
                          FSQ inverse; 8 languages, batch-only; hybrid
                          with MagpieTTS for LocalTransformer + audio
                          embeddings)
  MADLADTranslation/      MADLAD-400 many-to-many translation (MLX, 400+ languages)
  GLiNER/                 GLiNER2.5-Decide classification + entity spans (MLX, INT8/FP16/FP32)
  AudioCLILib/            CLI command implementations
  AudioCLI/               CLI entry point

scripts/              Model conversion (PyTorch → MLX/CoreML), benchmarking
Tests/                Unit and integration tests
Examples/             Demo apps (PersonaPlexDemo, SpeechDemo)

GLiNER2.5-Decide: native MLX classification and entity spans (INT8, FP16 or FP32 weights); no Python at inference time.

Clef-flash: development preview of local text decisions (9B, affine 4-bit MLX), with a joint schema head and a separate output-embedding prior.

MOSS-TTS-Nano

Streaming speech synthesis and voice cloning with CoreML, 48 kHz stereo audio and 80 ms chunks.

MossTTSNanoModel serializes requests and creates fresh CoreML state for each utterance. synthesize and synthesizeStream preserve stereo; SpeechGenerationModel returns a mono downmix. Pass language: nil. Streams end with an empty final marker.

macOS 15+ / iOS 18+; four compiled graphs total 367.6 MB. Prompt, reference and requested frames share a 1,024-row context. The runtime rejects overflow and raises an error if maxFrames is reached before natural end-of-speech. CPU is the default; gpu permits CPU+GPU execution. PCG64 sampling uses NumPy-compatible seeds; backend numerical differences can still change output.

MOSS-TTS-Nano

Local native inference

Optional agentDirectory loads a staged Gemma 4 12B Unified INT4 text model alongside E4B. Use model: "gemma4-12b-it" for chat with native tools, result history, and cancellation. E4B remains available for other requests. Gemma4UnifiedChat uses the pinned MLX text implementation and releases each request’s KV cache. Response parsing removes channel delimiters from returned text. Orphaned tool-call terminators are rejected as malformed responses.

The native service serializes chat, synthesis, and transcription; lightweight VAD has separate stream state. Gemma prefill uses 256-token batches, request KV state is released, and the MLX cache is capped at 128 MiB with a 20 GiB allocator limit.

API details

Optional model ownership

SpeechRuntime provides shared loading, cancellable FIFO admission and serialized execution. Qwen3ASRRuntime selects MLX or full CoreML through explicit descriptors; native APIs and CLI flags remain available.

The audio server holds one execution permit through each complete request, including streamed output, with up to 32 requests waiting. ASR failures produce HTTP errors or WebSocket error events; valid empty transcripts remain successful. Playback preserves overflow samples. Cached models and pending playback audio still occupy memory.

Voxtral TTS

Voxtral 4B TTS 2603 runs locally on Apple Silicon through the VoxtralTTS module: 20 preset voices, nine languages, and 24 kHz mono speech.

VoxtralTTS → 26-layer LM → 3-layer acoustic transformer → 24 kHz PCM

Preset voices only; reference-audio cloning is unavailable. Audio output streams through the SDK, CLI, WebSocket and HTTP PCM. Supply the complete input text up front. Model weights use CC BY-NC 4.0.

Read the Voxtral TTS guide

Fish Audio Drama 3

Experimental native MLX support for Drama 3 is available through FishAudioDrama3Model. A 35B hybrid-attention MoE backbone, temporal audio tower and Fish DAC codec produce 44.1 kHz mono audio. The 4-bit checkpoint passed waveform and independent ASR checks for text-only generation, reference cloning and CLI output on Apple M5 Pro. 8-bit and BF16 remain unvalidated.

The default balanced mode trims idle MLX buffers above 512 MiB at stage boundaries. The shared model-lifetime cache cap also applies, preserving lower caller budgets; neither control limits total process memory. --fish-drama3-memory-mode unrestricted or memoryOptions: .unrestricted loads weights eagerly and skips stage trimming, while retaining the lifetime cap.

Text-only generation defers 794.4 MiB of reference encoder weights. The first reference encoding loads and retains them, so keep codec.safetensors available until then. Precomputed reference codes do not need the encoder.

FishAudioTTS: FishAudioDrama3Model · backbone.model.* · semantic_head.* · tower.* · codec.safetensors

Built with Fish Audio. Bundles retain the Community License and NOTICE.

4-bit benchmark

PCM streaming is available through generateStreaming, generateStream and --stream. The default chunk contains 3 codec frames, about 139 ms of audio. Automatic long-text chunking and streamed text input remain unavailable.

Each tower codebook uses 4,096 token values. Buffered and streaming Fish DAC decoding both clamp residual codes to the codec’s 1,024-entry residual codebooks. Returned codebooks retain the original tower values.

Invalid rotary dimensions are rejected before loading model weights.