Qwen3-TTS

Qwen3-TTS is a 12Hz codec language model with a Mimi decoder for high-quality text-to-speech synthesis. The default MLX path now uses the 1.7B Base bf16 bundle; 8-bit bundles remain available for smaller downloads. int4 was decommissioned for TTS quality.

Pipeline

Speech synthesis follows a three-stage pipeline:

1. Talker — 28-layer transformer that converts input text into first codebook tokens at 12.5 Hz
2. Code Predictor — 5-layer transformer that predicts the remaining 15 codebooks from the first codebook hidden states
3. Mimi Codec Decoder — Converts all 16 codebook tokens into a 24 kHz audio waveform

Architecture

Talker

The Talker is the core autoregressive model that generates codec tokens from text input.

Code Predictor

A lightweight 5-layer transformer that takes the hidden states from the first codebook and predicts the remaining 15 codebooks autoregressively. For single-item synthesis, the fixed 15-group loop runs as one compiled graph.

Mimi Codec Decoder

The Mimi decoder converts quantized codec tokens back into audio:

1. RVQ decode (16 codebooks)
2. Pre-convolution (512 to 1024 channels)
3. Pre-transformer (1024 to 512 bottleneck, 8 layers, SwiGLU + LayerScale)
4. Upsample (2x, 2x)
5. SEANet decoder (8x, 5x, 4x, 3x upsample stages)
6. 24 kHz waveform output

Model Variants

CoreML Backend

CoreML runs six compiled components. Compute placement depends on the selected bundle.

1. TextProjector — Projects text token embeddings to the shared hidden space
2. CodeEmbedder — Embeds first-codebook tokens and control tokens
3. MultiCodeEmbedder — Embeds tokens from codebooks 1–15
4. CodeDecoder — 28-layer autoregressive transformer with stateless KV cache (max 256 positions)
5. MultiCodeDecoder — 5-layer code predictor for codebooks 1–15
6. SpeechDecoder — Mimi codec decoder, converts 16 codebook tokens to 24 kHz audio

# CoreML synthesis
.build/release/speech speak "Hello, world!" --engine coreml -o hello.wav

# CoreML uses temperature 0.8 by default (required for quality output)
.build/release/speech speak "Long text here." --engine coreml --temperature 0.9 -o out.wav

speech qwen3-tts-coreml "Hello world" \
  --model aufklarer/Qwen3-TTS-1.7B-CoreML \
  --speaker-embedding speaker-17b.npy --output hello-17b.wav

CLI Usage

Generate speech from text:

.build/release/speech speak "Hello, world!" --output hello.wav

Options

Examples

# Basic synthesis
.build/release/speech speak "The quick brown fox." -o fox.wav

# Streaming output
.build/release/speech speak "Long passage of text..." --stream -o stream.wav

# Batch synthesis from file
.build/release/speech speak --batch-file sentences.txt --batch-size 4 -o output_dir/

Streaming

The --stream flag enables chunked audio output during generation. Instead of waiting for the entire utterance to complete, audio is emitted in chunks as tokens are produced. Use --first-chunk-frames and --chunk-frames to control the size of each chunk.

Batch Mode

For synthesizing multiple utterances, use --batch-file with a text file containing one line per utterance. The --batch-size flag controls how many utterances are processed in parallel.

Performance

On an M2 Max, Qwen3-TTS achieves an RTF (real-time factor) of approximately 0.55, meaning it generates speech faster than real-time. With compile() warmup, each step takes about 37 ms.

Languages

Qwen3-TTS supports multilingual text-to-speech synthesis. The model automatically detects the input language and generates speech accordingly.

Swift API

import Qwen3TTS
import AudioCommon

let model = try await Qwen3TTSModel.fromPretrained()
let audio = model.synthesize(text: "Hello, world!", language: "english")
try WAVWriter.write(samples: audio, sampleRate: 24000, to: URL(filePath: "hello.wav"))

Local-only bundles

Use fromLocal when your app owns model acquisition and storage. Pass the main TTS model and speech-tokenizer directories separately. The call never resolves a cache or contacts the network, and reads the main bundle's config.json before allocating MLX modules.

let model = try Qwen3TTSModel.fromLocal(
    modelDirectory: ttsDirectory,
    tokenizerDirectory: speechTokenizerDirectory
)

Diagnostics

Qwen3-TTS sends model-loading and inference diagnostics to AudioLog.modelLoading and AudioLog.inference instead of stdout or stderr. Counts, shapes, and timings remain visible; caller-provided language and speaker values are private, and input text is never logged.