Architecture

speech-swift is organized as a modular Swift package with shared protocols, independent model modules, and a unified CLI. All inference runs on-device using MLX (Metal GPU) or CoreML (Neural Engine).

Module Dependency Graph

                    ┌──────────┐
                    │ AudioCLI │  (entry point)
                    └────┬─────┘
                         │
                  ┌──────┴──────┐
                  │ AudioCLILib │  (commands)
                  └──────┬──────┘
                         │
       ┌─────────┬───────┼───────┬──────────┬──────────────┐
       │         │       │       │          │              │
  ┌────┴───┐ ┌──┴──┐ ┌──┴──┐ ┌─┴────┐ ┌───┴────┐ ┌──────┴───────┐
  │Qwen3ASR│ │Qwen3│ │Cosy │ │Perso-│ │Speech- │ │  Speech-     │
  │Parakeet│ │ TTS │ │Voice│ │naPlex│ │  VAD   │ │Enhancement   │
  └────┬───┘ └──┬──┘ └──┬──┘ └──┬───┘ └───┬───┘ └──────┬───────┘
       │        │       │       │         │             │
       └────────┴───────┼───────┴─────────┘             │
                        │                               │
                 ┌──────┴──────┐                        │
                 │ Qwen3Common │  (shared layers)       │
                 └──────┬──────┘                        │
                        │                               │
                 ┌──────┴──────┐                        │
                 │ AudioCommon │ ◄──────────────────────┘
                 └─────────────┘  (protocols, audio I/O)

MOSS Transcribe Diarize

MossTranscribe returns speaker-attributed timestamp segments through two native backends. CoreML uses a fixed 1,024-token MLState for short recordings. MLX uses a dynamically growing 131,072-token context, affine INT5/INT8 decoder weights, and optional FP16/INT8 KV caches for globally contextualized long-form transcription. Both are offline, autoregressive runtimes rather than streaming models.

VoiceChat 11B

Task and reference: duplex speech-to-speech; SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model (Interspeech 2025), cited by NVIDIA's upstream VoiceChat model card.

Complete runtime, M5 Pro 48 GB, Release, 80 ms live cadence: protected-head INT5 peak RSS is about 8.69 GB; three controlled whole-pipeline RTF results were 0.96, 0.96, and 0.99. Optimized INT8 performance has not yet been remeasured.

VoiceChat implements the complete native MLX audio-to-audio path: causal FastConformer perception, the Nemotron-H duplex text/function channels, the standalone text-conditioned EAR-TTS decoder with eight MaskGIT iterations, and a 22.05 kHz neural codec. Streaming retains bounded per-layer attention and causal-convolution caches, while the steady eight-frame live codec window uses a compiled graph. Every 80 ms input frame produces a language decision, 31 RVQ codes, and exactly 1,764 output samples. The loader requires a complete encoder/ + llm/ + tts/ bundle and rejects older understanding-only exports. INT5 now sustains aggregate RTF below 1 on the tested M5 Pro, although per-frame p95 headroom remains narrow.

Inference Backends

Backend	Hardware	Models
MLX	Metal GPU	Qwen3-ASR, Qwen3-TTS, CosyVoice3, IndexTTS2, Qwen3.5-Chat, PersonaPlex, VoiceChat 11B, Omnilingual ASR (300M / 1B / 3B / 7B), Cohere Transcribe 2B, Voxtral Mini 3B, MOSS Transcribe Diarize, Pyannote, Silero VAD, WeSpeaker, SpeechBrain ECAPA LID
CoreML	Neural Engine	Qwen3-ASR encoder (hybrid), Parakeet TDT, Parakeet EOU streaming, Omnilingual ASR 300M, Kokoro-82M, Qwen3.5-Chat (optional), Sortformer diarization, DeepFilterNet3, Silero VAD (optional), WeSpeaker (optional), MOSS Transcribe Diarize, SpeechBrain ECAPA LID
Accelerate	CPU (SIMD)	Audio preprocessing (STFT, mel, FFT), signal processing

Model Weight Format

MLX models use safetensors format with 4-bit or 8-bit quantization (group size 64). CoreML models use .mlmodelc compiled format. Conversion scripts in scripts/ convert from PyTorch checkpoints.

Cohere and Voxtral also use MLX affine INT5 bundles. MLX does not support affine INT7; INT8 is the supported higher-quality option.

Model	Params	Quantization	Size on Disk
Qwen3-ASR 0.6B (MLX)	~600M	4-bit / 8-bit	680 MB / 1.0 GB
Qwen3-ASR 0.6B (CoreML)	~186M (encoder)	INT8	~180 MB
Qwen3-ASR 1.7B (MLX)	~1.7B	4-bit / 8-bit	2.1 GB / 3.2 GB
Parakeet-TDT 0.6B (CoreML)	~600M	INT8	500 MB
Whisper Large-v3 Turbo (CoreML)	809M	FP16	1.6 GB
MOSS Transcribe Diarize 0.9B (MLX)	908.5M	FP16 audio + INT5 / INT8 decoder	987.0 MiB / 1,200.1 MiB
MOSS Transcribe Diarize 0.9B (CoreML)	~900M	FP16 / INT8 block-32	1.7 GB / 1.2 GB
Parakeet-EOU 120M (CoreML)	~120M	INT8	~120 MB
Omnilingual-ASR-CTC 300M (CoreML)	326M	INT8	312 MB
Omnilingual-ASR-CTC 300M (MLX)	326M	4-bit / 8-bit	193 MB / 342 MB
Omnilingual-ASR-CTC 1B (MLX)	1.01B	4-bit / 8-bit	549 MB / 1006 MB
Omnilingual-ASR-CTC 3B (MLX)	~3B	4-bit / 8-bit	1.71 GB / 3.16 GB
Omnilingual-ASR-CTC 7B (MLX)	~7B	4-bit / 8-bit	3.55 GB / 6.63 GB
Cohere Transcribe 2B (MLX)	2B	FP16 / INT5 / INT8	3.85 / 1.62 / 2.25 GiB
Voxtral Mini 3B 2507 (MLX)	3B	FP16 / INT5 / INT8	8.71 / 3.77 / 5.18 GiB
Qwen3-ForcedAligner 0.6B (MLX)	~600M	4-bit / 8-bit	979 MB / 1.4 GB
Qwen3-ForcedAligner 0.6B (CoreML)	~600M	INT4 / INT8	630 MB / 1.0 GB
Qwen3-TTS 0.6B (MLX)	~600M	4-bit / 8-bit	1.7 GB / 2.4 GB
Qwen3-TTS 1.7B (MLX)	~1.7B	4-bit / 8-bit	3.2 GB / 4.8 GB
CosyVoice3 0.5B (MLX)	~500M	4-bit LLM	~1.2 GB
IndexTTS2 expanded bundle (MLX)	1.5B-class	FP16 + auxiliary models	~4.8 GB
Kokoro-82M (CoreML)	82M	INT8 (1 bucket)	~89 MB
Qwen3.5-Chat 0.8B (MLX)	~800M	INT4	418 MB
Qwen3.5-Chat 0.8B (CoreML)	~800M	INT8	981 MB
PersonaPlex 7B (MLX)	~7B	4-bit / 8-bit	4.9 GB / 9.1 GB
VoiceChat 11B (MLX)	~11B	INT5 / INT8	8.56 GB / 12.11 GB
Pyannote VAD (MLX)	~1.49M	float32	~5.7 MB
Silero VAD v5	~309K	float32	~1.2 MB (MLX & CoreML)
WeSpeaker ResNet34	~6.6M	float32	~25 MB (MLX & CoreML)
SpeechBrain ECAPA VoxLingua107	21.25M	FP16	~40.6 MiB (MLX) / ~40.8 MiB (CoreML)
ReDimNet2-B6	12.3M	float16	~25 MiB (CoreML)
Pyannote Community-1 (CoreML)	8.35M	float32 + native VBx	~32 MB
Sortformer (CoreML)	—	float16	~50 MB
DeepFilterNet3 (CoreML)	~2.1M	FP16	~4.2 MB
LocalVQE v1.4-AEC (CoreML + C++)	200K + 2,742	FP16 + FP32	~732 KB

Performance Optimizations

MLX compilation — Qwen3-TTS compiles the Talker over its growing valid-prefix cache and compiles the fixed 15-group Code Predictor frame as one graph for single-item synthesis.
Metal shader library — Pre-compiled metallib avoids ~5x JIT compilation overhead. Built via scripts/build_mlx_metallib.sh.
Chunked codec decode — TTS decoder processes speech in 25-frame chunks with 10-frame context overlap to avoid GPU timeout.
Batch-doubled CFG — CosyVoice3 DiT halves flow matching passes by batching conditional + unconditional together.
Fused RoPE — Uses MLXNN.RoPE backed by Metal kernel instead of manual rotation.
BN fusion — WeSpeaker batch normalization fused into Conv2d weights at conversion time.

Audio Processing

All audio I/O uses Float32 PCM. Internal resampling handles format conversion:

Model	Expected Rate	Format
Qwen3-ASR	16 kHz	Mono Float32
Cohere Transcribe 2B	16 kHz	Mono Float32
Voxtral Mini 3B	16 kHz	Mono Float32
Qwen3-TTS	24 kHz output	Mono Float32
CosyVoice3	24 kHz output	Mono Float32
Kokoro-82M	24 kHz output	Mono Float32
PersonaPlex	24 kHz I/O	Mono Float32
Pyannote VAD	16 kHz	Mono Float32
Silero VAD	16 kHz	Mono Float32
WeSpeaker	16 kHz	Mono Float32
SpeechBrain ECAPA LID	16 kHz	Mono Float32
ReDimNet2-B6	16 kHz	Mono Float32
DeepFilterNet3	48 kHz	Mono Float32

Source Structure

Sources/
  AudioCommon/            Shared protocols, audio I/O, system-output capture
                          (SystemAudioTap), HuggingFace downloader,
                          SentencePieceModel (protobuf reader)
  MLXCommon/              MLX utilities: weight loading, QuantizedLinear helpers,
                          SDPA multi-head attention helper, metal budget
  Qwen3Common/            Shared model components (KV cache, RoPE, quantization)
  Qwen3ASR/               Qwen3-ASR speech-to-text
  ParakeetASR/            Parakeet TDT speech-to-text (CoreML)
  ParakeetStreamingASR/   Parakeet EOU 120M streaming dictation (CoreML)
  OmnilingualASR/         Meta wav2vec2 + CTC, 1,672 languages
                          (CoreML 300M + MLX 300M / 1B / 3B / 7B)
  CohereTranscribeASR/  Cohere Transcribe 2B speech-to-text (MLX)
  VoxtralASR/            Voxtral Mini 3B speech-to-text (MLX)
  Qwen3TTS/               Qwen3-TTS text-to-speech
  CosyVoiceTTS/           CosyVoice3 text-to-speech
  KokoroTTS/              Kokoro-82M text-to-speech (CoreML)
  Qwen3Chat/              Qwen3.5-0.8B on-device LLM chat (MLX + CoreML)
  PersonaPlex/            PersonaPlex speech-to-speech
  SpeechVAD/              VAD (Silero + Pyannote), diarization, speaker embeddings
  SpeechLanguageID/       Spoken language identification (SpeechBrain ECAPA, MLX + CoreML, 107 labels)
  SpeechEnhancement/      DeepFilterNet3 noise suppression (CoreML)
  SourceSeparation/       Open-Unmix HQ / UMX-L + HTDemucs (Demucs v4) music source separation (MLX, 4 stems)
  MAGNeTMusicGen/         MAGNeT text-to-music generation (MLX, 30 s @ 32 kHz, INT4/INT8)
  VoxCPM2TTS/             VoxCPM2 text-to-speech (MLX, 48 kHz, voice cloning + voice design)
  IndexTTS2TTS/           IndexTTS2 voice cloning (MLX, semantic GPT + emotion/tempo/pause control + S2Mel + BigVGAN)
  VibeVoiceTTS/           VibeVoice long-form / multi-speaker TTS (MLX, EN/ZH)
  MagpieTTS/              NVIDIA Magpie-TTS Multilingual 357M (MLX INT4/INT8,
                          9 languages, 5 baked speakers, streaming)
  MagpieTTSCoreML/        Magpie CoreML backend (4 .mlmodelc + Swift-side
                          FSQ inverse; 8 languages, batch-only; hybrid
                          with MagpieTTS for LocalTransformer + audio
                          embeddings)
  MADLADTranslation/      MADLAD-400 many-to-many translation (MLX, 400+ languages)
  AudioCLILib/            CLI command implementations
  AudioCLI/               CLI entry point

scripts/              Model conversion (PyTorch → MLX/CoreML), benchmarking
Tests/                Unit and integration tests
Examples/             Demo apps (PersonaPlexDemo, SpeechDemo)