Omnilingual ASR

Meta's Omnilingual ASR is a language-agnostic speech recognition family covering 1,672 languages across 32 distinct scripts — the broadest language coverage of any on-device ASR model on Apple Silicon. Soniqo ports the CTC variant to both CoreML (Neural Engine) and MLX (Metal GPU), with four model sizes from 300M to 7B parameters.

Architecture

Omnilingual CTC is a supervised fine-tune of Meta's wav2vec 2.0 backbone with a linear CTC head over a shared multilingual vocabulary. The pipeline is parallel and non-autoregressive — one forward pass per utterance, no decoder loop.

raw audio [1, samples]
  → wav2vec2 feature extractor (7 strided CNN layers, ×320 downsample)
  → weight-normalised Conv1d positional encoder
  → N × pre-norm Transformer encoder layers
  → final layer norm
  → linear CTC head → logits [1, T, 10288]

At 16 kHz input, the 320× downsampled encoder produces 50 Hz frame rate. A 10-second clip produces 499 frames of logits. Greedy CTC decoding collapses consecutive duplicates and skips special tokens via the SentencePiece tokenizer.

Model Variants

Ten variants are published under the aufklarer namespace on HuggingFace — two CoreML window sizes plus eight MLX quantisation combinations:

CLI Usage

CoreML (Neural Engine)

# 10 s window (default)
.build/release/audio transcribe recording.wav --engine omnilingual

# 5 s window (lower memory, faster cold start)
.build/release/audio transcribe recording.wav --engine omnilingual --window 5

MLX (Metal GPU)

# 300M @ 4-bit (default MLX variant)
.build/release/audio transcribe recording.wav --engine omnilingual --backend mlx

# 1B @ 4-bit — higher accuracy
.build/release/audio transcribe recording.wav --engine omnilingual --backend mlx --variant 1B

# 3B @ 8-bit — approaching reference quality
.build/release/audio transcribe recording.wav --engine omnilingual --backend mlx --variant 3B --bits 8

# 7B @ 4-bit — largest CTC variant, best accuracy
.build/release/audio transcribe recording.wav --engine omnilingual --backend mlx --variant 7B

Swift API

CoreML backend

import OmnilingualASR
import AudioCommon

let model = try await OmnilingualASRModel.fromPretrained()
let audio = try AudioFileLoader.load(url: url, targetSampleRate: 16000)
let text = try model.transcribeAudio(audio, sampleRate: 16000)
print(text)

MLX backend

import OmnilingualASR

// Default: 300M @ 4-bit
let model = try await OmnilingualASRMLXModel.fromPretrained()

// Larger variants
let big = try await OmnilingualASRMLXModel.fromPretrained(variant: .b1, bits: 4)
let huge = try await OmnilingualASRMLXModel.fromPretrained(variant: .b3, bits: 8)
let max = try await OmnilingualASRMLXModel.fromPretrained(variant: .b7, bits: 4)

let text = try model.transcribeAudio(audio, sampleRate: 16000)

CoreML vs MLX

Both backends produce essentially identical transcripts modulo 1-2 characters (quantisation and runtime differences). Choose based on your deployment target:

Preprocessing Details

Omnilingual requires utterance-level layer-norm on the raw waveform, matching fairseq2's apply_audio_normalization:

normalized = (audio - mean(audio)) / sqrt(variance(audio) + 1e-5)

The Swift port normalises the real audio content before zero-padding to the CoreML fixed window, so sub-window inputs match reference pipeline statistics exactly. This is the single most common port pitfall — HuggingFace's wav2vec2 implementation does group-norm per-feature, not utterance layer-norm.

The reference pipeline enforces a 40-second hard cap (MAX_ALLOWED_AUDIO_SEC) on input audio. The Swift port enforces the same limit — longer inputs throw a clear error pointing to SpeechVAD or ParakeetStreamingASR for segmented processing.

Language Coverage

Omnilingual supports 1,672 languages across 32 scripts, including 500+ low-resource languages added via community data collection. Sample coverage:

Full ISO 639-3 + ISO 15924 code list with English names in the lang_ids.py source, and grouped by script with country hints in the model card.

Verified Output

Transcripts from the Swift port on the FLEURS benchmark, CoreML 300M:

The MLX 300M-4bit variant produces essentially identical output modulo 1-2 characters. Larger variants (1B, 3B, 7B) reduce residual errors progressively.

Reference

• Paper: Omnilingual ASR (arXiv 2511.09690)
• Upstream implementation: facebookresearch/omnilingual-asr (Apache 2.0)
• Swift port source: speech-swift/Sources/OmnilingualASR
• Model card: docs/models/omnilingual-asr.md