Running a voice agent on-device
one pipeline, three memory budgets.
A local voice agent isn’t one model — it’s a pipeline of four: voice activity detection, speech-to-text, a language model, and text-to-speech, wired together by a turn-taking state machine. The pipeline is the same on a phone and on a laptop. What changes is which model goes in each box, and that’s decided by two numbers: how much memory you have, and which accelerator you’re feeding.
The number to hold onto is the whole loop, not one model: speech perception (STT) + answer generation (LLM) + speech synthesis (TTS) all fit inside it — ~1.2 GB on iPhone, ~1.5 GB on a Galaxy S23, and under ~4 GB on a Mac with a real chat brain.
The pipeline is the portable part.
Orchestration lives in speech-core as a pure C++ state machine that is model-agnostic: the four stages are interfaces you plug any backend into — on-device ONNX, LiteRT, CoreML/MLX, or even a cloud API — so the same pipeline runs a 270M tool-caller on a phone and a Gemma 4 brain on a Mac.
One loop, four stages. A wake word, AEC, and enhancement are optional pre-VAD steps; the LLM’s tool calls are what let it act, not just answer.
The orchestrator is VoicePipeline in speech-core. It owns turn detection, interruption, conversation history, the tool-calling loop, and speech queuing — but no models of its own. Every stage is swappable behind an interface (VADInterface, STTInterface, LLMInterface, TTSInterface) that you back with ONNX Runtime, LiteRT, CoreML/MLX (via the speech-swift sibling), or even a cloud API. Lighter modes exist too — Echo (VAD → STT → TTS) and TranscribeOnly (VAD → STT → text).
NPU-first, and the whole loop fits in ~1.5 GB.
On a phone the constraint is memory, battery, and thermals — not raw compute. The right target is the neural processing unit (NPU): Apple's Neural Engine on iOS, NNAPI / Hexagon on Android. It runs quantized transformers at a fraction of the power of the CPU or GPU and leaves the CPU free for audio and UI.
Why the NPU and not the CPU or GPU? An NPU is a fixed-function dataflow engine built for one job — the low-precision multiply-accumulate that neural nets are made of. It keeps weights in on-chip SRAM right next to the compute units and skips the instruction fetch/decode and wide general-purpose datapaths that burn most of a CPU’s or GPU’s energy. Moving data costs far more than doing the math, so keeping both local is the win: the same matmuls run at a fraction of the watts — no thermal throttling, no battery drain, and the CPU stays free for everything else.
So the mobile pipeline is small, quantized, NPU-friendly models: Silero VAD, Parakeet-EOU 120M (lightweight streaming STT with inline end-of-utterance), FunctionGemma 270M (the tool-calling brain), and Kokoro-82M or Supertonic-3 for voice. Measured on device:
| Stage · model | iPhone 16 Pro (CoreML / ANE) | Galaxy S23 (LiteRT / ONNX, CPU) |
|---|---|---|
| STT · Parakeet-EOU 120M | 0.04 RTF · 297 MB | 0.21 RTF · 232 MB |
| STT · Omnilingual 300M | 0.28 RTF · 495 MB | 0.15 RTF · 831 MB |
| TTS · Kokoro-82M | 0.08 RTF · 676 MB | 0.53 RTF · 640 MB |
| TTS · Supertonic-3 99M | 0.15 RTF · 956 MB | 0.34 RTF · 832 MB |
| LLM · FunctionGemma 270M | 128 tok/s · 236 MB | 118 tok/s · 611 MB |
RTF = wall-time ÷ audio-seconds (lower = faster; <1.0 is faster than real time). Peak memory is per-model in isolation.
Every stage clears real time. Loaded together for a live agent, the resident working set is ~1.2 GB on the iPhone and ~1.5 GB on the Galaxy S23 — the gap is almost all LLM: FunctionGemma is 236 MB on the ANE versus 611 MB on the S23 CPU. iOS already runs on the NPU (that’s the 0.04 RTF STT and 128 tok/s LLM); Android is CPU-real-time today, with NNAPI/Hexagon delegates as upside, not a prerequisite.
Try it: Examples/iOSEchoDemo runs Parakeet + Silero + Kokoro as a full echo loop on an iPhone; the Android side lives in speech-android, wrapping the same pipeline. (Examples/iOSBenchmark is the harness behind the iPhone numbers above.)
A real LLM that drives the machine — still under 4 GB.
On a Mac the constraint relaxes: unified memory shared between CPU and a Metal GPU. Same four stages, bigger models, and crucially a real chat brain instead of a 270M tool-caller — and the whole loop still fits in ~4 GB.
The LLM is the brain, and it’s where model size stops being optional. After testing, two models earned the job: FunctionGemma 270M on mobile (enough to emit a tool call, all a phone-side agent needs) and Gemma 4 E2B/E4B (4-bit MLX) on desktop (genuine multi-turn conversation and reasoning that holds up, while staying light). The tempting middle — a compact sub-1B chat head — looks free on footprint but wobbles on real multi-turn, so it didn’t make the cut. The jump that matters isn’t 0.27B → 0.8B; it’s FunctionGemma’s tool-calls → Gemma 4’s conversation.
The rest of the stack steps up too: desktop swaps the lightweight Parakeet-EOU for Parakeet-TDT (full multilingual), and TTS stays on Supertonic-3 everywhere — fast, 44.1 kHz, stable. We skip the heavier zero-shot-cloning TTS models on purpose; cloning isn’t reliable enough to sit inside an always-on agent.
| Stage | Mobile | Desktop (Mac / MLX) |
|---|---|---|
| VAD | Silero v6.2.1 | Silero v6.2.1 |
| STT | Parakeet-EOU (streaming) | Parakeet-TDT (multilingual) |
| LLM | FunctionGemma 270M | Gemma 4 E2B/E4B |
| TTS | Kokoro-82M / Supertonic-3 | Supertonic-3 |
| Resident set | iPhone ~1.2 GB · S23 ~1.5 GB | ~4 GB |
That desktop path is what the Runner Agent ships — a notarized macOS app running Mic → VAD → Parakeet-TDT → Gemma 4 → Supertonic TTS entirely locally, a 47 MB DMG with the whole loop around ~4 GB on Apple Silicon. And because the brain emits tool calls, Runner doesn’t just talk back — it drives the machine by voice: open a browser and run DOM operations, open Telegram or WhatsApp to read incoming messages and type replies, launch and control apps — all from spoken commands, all local. That’s the real reason desktop wants Gemma 4 and not a 270M tool-caller: acting on the machine takes actual reasoning, not just slot-filling.
One pipeline, matched to the budget.
VAD → STT → LLM → TTS is identical everywhere. Everything interesting is matching each stage to the device’s accelerator and memory.
Battery and thermals pick the models. The ANE makes iOS effectively free CPU-wise; Android is CPU-real-time today with NPU headroom to come.
The whole loop fits in ~4 GB, and that budget buys Gemma 4 reasoning that drives the machine by voice, plus multilingual Parakeet-TDT.
Audio and conversation state never leave the device — perception, reasoning, and synthesis all run on the metal in front of you.