Speech Server

Native Apple runtime · PocketTTS

PocketTTS adds native CoreML speech synthesis to speech-swift. All stages default to CPU-only execution. The English Alba preset streams 1,920-sample frames at 24 kHz (80 ms per frame).

Swift and CLI usage

Use pocket-tts with voice alba. Streaming HTTP requires PCM; WebSocket emits audio deltas. The existing default models are unchanged.

curl http://localhost:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"pocket-tts","voice":"alba","input":"Hello world.","response_format":"pcm","stream":true}' \
  -o hello.pcm

speech-server is a local HTTP + WebSocket server that exposes every Soniqo model through a simple REST API, plus an OpenAI Realtime API-compatible WebSocket at /v1/realtime. It ships in the same Homebrew bottle as the speech CLI — brew install speech drops both on your PATH.

Install and run

brew install speech

speech-server --port 8080
# Starting server on http://127.0.0.1:8080
# Endpoints:
#   POST /transcribe  - Speech-to-text (WAV body or JSON with audio_base64)
#   POST /speak       - Text-to-speech (JSON: {text, engine?, language?})
#   POST /respond     - Speech-to-speech (WAV body, voice/max_steps via query)
#   POST /enhance     - Speech enhancement (WAV body)
#   GET  /health      - Health check
#   WS   /v1/realtime - OpenAI Realtime API (JSON events, base64 PCM16 audio)

Command-line flags

FlagDefaultDescription
--host127.0.0.1Bind address. Change to 0.0.0.0 to expose on LAN.
--port8080TCP port.
--preloadoffLoad all models eagerly at startup. Slower boot (~30–60 s) but zero first-request latency.

Models are downloaded on first use and cached in ~/Library/Caches/qwen3-speech/. The first request for a given model pays the download + load cost (30 s – 2 min depending on model size); subsequent requests are warm.

REST endpoints

POST /transcribe — Speech-to-text

Accepts either a raw WAV body or a JSON envelope with base64-encoded audio.

# WAV body (preferred — lower overhead)
curl -X POST http://127.0.0.1:8080/transcribe \
  -H "Content-Type: audio/wav" \
  --data-binary @recording.wav

# JSON with base64
curl -X POST http://127.0.0.1:8080/transcribe \
  -H "Content-Type: application/json" \
  -d '{"audio_base64":"'"$(base64 -i recording.wav)"'","language":"en"}'

Response: {"text": "…", "language": "en", "confidence": 0.93}.

POST /speak — Text-to-speech

curl -X POST http://127.0.0.1:8080/speak \
  -H "Content-Type: application/json" \
  -d '{"text":"Hello, world!","engine":"kokoro","language":"en"}' \
  --output hello.wav

Response body is a WAV blob. Supported engine values: qwen3 (default), cosyvoice, kokoro.

POST /respond — Speech-to-speech

curl -X POST "http://127.0.0.1:8080/respond?voice=en_female_calm&max_steps=256" \
  -H "Content-Type: audio/wav" \
  --data-binary @question.wav \
  --output answer.wav

Runs PersonaPlex 7B — speech in, speech out. Transcript is returned as an X-Response-Text header. See PersonaPlex guide for voice preset names.

POST /enhance — Speech enhancement

curl -X POST http://127.0.0.1:8080/enhance \
  -H "Content-Type: audio/wav" \
  --data-binary @noisy.wav \
  --output clean.wav

DeepFilterNet3 at 48 kHz. Input is resampled if needed.

GET /health — Liveness probe

curl http://127.0.0.1:8080/health
# {"status":"ok"}

The HTTP and realtime WebSocket tests wait for /health to return HTTP 200 with {"status":"ok"}, with a 10-second startup deadline. Startup errors fail test setup, and teardown awaits server shutdown before reusing its port.

WebSocket: /v1/realtime

Drop-in compatible with the OpenAI Realtime API — the same JSON event schema (session.update, input_audio_buffer.append, response.create, response.audio.delta, …) with base64-encoded PCM16 audio at 24 kHz. Clients written against OpenAI's Realtime SDK work against speech-server without code changes (just switch the WebSocket URL).

Automatic turns with server VAD

Manual input_audio_buffer.commit remains the default. Set session.turn_detection to server_vad to run Silero VAD on the server and automatically transcribe each completed speech turn. Prefix audio is retained, idle buffering stays bounded, and a continuously active turn is force-closed at the configured maximum.

ws.send(JSON.stringify({
  type: "session.update",
  session: {
    turn_detection: {
      type: "server_vad",
      threshold: 0.5,
      prefix_padding_ms: 300,
      silence_duration_ms: 500,
      max_turn_duration_ms: 120000
    }
  }
}));

// Event order for each detected turn:
// input_audio_buffer.speech_started
// input_audio_buffer.speech_stopped
// input_audio_buffer.committed
// conversation.item.input_audio_transcription.completed

// Set turn_detection to null (or { type: "none" }) for manual commits.

JavaScript example

const ws = new WebSocket("ws://127.0.0.1:8080/v1/realtime");

ws.onopen = () => {
  ws.send(JSON.stringify({
    type: "session.update",
    session: { modalities: ["audio", "text"] }
  }));

  // Stream PCM16 mono 24kHz audio from the mic:
  const audioBase64 = await capturePCM16Chunk();
  ws.send(JSON.stringify({
    type: "input_audio_buffer.append",
    audio: audioBase64
  }));

  ws.send(JSON.stringify({ type: "response.create" }));
};

ws.onmessage = ev => {
  const msg = JSON.parse(ev.data);
  if (msg.type === "response.audio.delta") {
    playPCM16Base64(msg.delta);
  }
};

Python example

import asyncio, base64, json, wave, websockets

async def main():
    async with websockets.connect("ws://127.0.0.1:8080/v1/realtime") as ws:
        await ws.send(json.dumps({"type": "session.update",
                                  "session": {"modalities": ["audio", "text"]}}))

        with wave.open("question.wav", "rb") as wav:
            pcm16 = wav.readframes(wav.getnframes())
        await ws.send(json.dumps({"type": "input_audio_buffer.append",
                                  "audio": base64.b64encode(pcm16).decode()}))
        await ws.send(json.dumps({"type": "response.create"}))

        async for raw in ws:
            msg = json.loads(raw)
            if msg["type"] == "response.audio.delta":
                open("answer.pcm", "ab").write(base64.b64decode(msg["delta"]))
            elif msg["type"] == "response.done":
                break

asyncio.run(main())

Deployment notes

Source

Optional model ownership

SpeechRuntime provides shared loading, cancellable FIFO admission and serialized execution. Qwen3ASRRuntime selects MLX or full CoreML through explicit descriptors; native APIs and CLI flags remain available.

The audio server holds one execution permit through each complete request, including streamed output, with up to 32 requests waiting. ASR failures produce HTTP errors or WebSocket error events; valid empty transcripts remain successful. Playback preserves overflow samples. Cached models and pending playback audio still occupy memory.

Voxtral TTS

Voxtral 4B TTS 2603 runs locally on Apple Silicon through the VoxtralTTS module: 20 preset voices, nine languages, and 24 kHz mono speech.

curl http://localhost:8080/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"model":"voxtral-tts","input":"Hello, welcome to Voxtral.","voice":"casual_male","response_format":"wav"}' \
  -o hello.wav

Preset voices only; reference-audio cloning is unavailable. Audio output streams through the SDK, CLI, WebSocket and HTTP PCM. Supply the complete input text up front. Model weights use CC BY-NC 4.0.

Read the Voxtral TTS guide