macvoice

A local voice agent loop for Apple Silicon in under 2 GB, with no PyTorch.

The write-up: A Local Voice Agent on a MacBook in Under 2 GB.

 mic ──▶ Silero VAD ──▶ Smart Turn ──▶ Parakeet ──▶ your LLM ──▶ Phonon ──▶ speaker

On an M5 MacBook Pro, the voice pipeline uses 1.4 GB (physical footprint, GPU memory included) and starts speaking ~350 ms after you stop talking, before LLM time. With a local Qwen3-4B as the brain, the first word of the answer arrives in 0.6–0.8 s.

Stage	Model	Runtime
Voice activity	Silero VAD v6	ONNX Runtime (CPU)
End of turn	Smart Turn v3.2	ONNX Runtime (CPU)
Speech to text	Parakeet TDT 0.6B v2, 8-bit	MLX
Text to speech	Phonon 31M English	ptts (Rust, Metal)
Brain	any OpenAI-compatible server, or a coding agent over ACP

New: talk to Claude Code while it works. Point macvoice at a project folder and drive Claude Code (or any ACP coding agent) by voice. It asks out loud before running commands, you answer "yes" or "no", and you can talk over it to change direction, or type when that's easier:

uv run macvoice run --llm acp --cwd ~/demo-site

See Talk to a coding agent while it works.

Quick start

Needs an Apple Silicon Mac and uv.

uv sync
uv run macvoice download             # the models, 0.85 GB, once

# In another terminal, a local LLM (optional; ~2.6 GB, fully offline):
uvx --from mlx-lm mlx_lm.server --model mlx-community/Qwen3-4B-Instruct-2507-4bit --prompt-cache-size 1

uv run macvoice run                  # talk to it; talk over it to cut it short

After the download the voice pipeline runs offline. (run downloads anything still missing itself, so download is only there to get the wait out of the way first.) While it runs, the bottom line of the terminal shows what it's doing and how loud your mic is:

● hearing you  ▕██████████████▉         ▏

It goes listening → hearing you → thinking → speaking, plus allow? while a coding agent waits for your answer to a permission question. Each finished turn prints above it (--no-status turns it off).

Type or talk

run also gives you an input box, so you can type a prompt instead of saying it. Typed and spoken prompts go to the same conversation, and replies are rendered as markdown (code blocks and markdown symbols are left out of what's spoken).

Key
Enter	send
Shift+Enter, Option+Enter, Ctrl+J, or \ then Enter	new line
Up / Down, Ctrl+R	earlier prompts (kept across runs), search them
Ctrl+C	clear the line; or stop the reply and cancel the agent's turn; twice to quit
Esc	stop the reply and cancel the agent's turn
y / n / a	answer a permission question (when the line is empty)

Commands: /mute (replies as text only), /modes and /mode NAME (an ACP agent's modes), /model [NAME] and /effort [LEVEL] (its model and reasoning effort), /help, /quit. Shift+Enter works in terminals that support the kitty keyboard protocol (Ghostty, kitty, WezTerm, iTerm2); in Terminal.app use Option+Enter (with "Use Option as Meta key" on) or Ctrl+J. --no-console goes back to voice only.

At startup it prints which brain it's using. By default that's the LLM server on localhost:8080 if one is running; otherwise an "echo" brain that repeats what you say, so you can try the audio loop with no LLM at all.

New to running a local LLM? docs/llm-setup.md walks through it step by step (Qwen3-4B on MLX, about 2 minutes plus a 2.1 GB download), along with Ollama, LM Studio, hosted APIs and Claude Code over ACP.

The first time, macOS asks for microphone access for your terminal.

Echo cancellation (on by default)

Laptop speakers leak into the laptop mic, so a voice agent hears itself. macvoice runs the mic and speaker through one AVAudioEngine with Apple's voice processing turned on, the same echo canceller FaceTime uses and Chrome uses for echoCancellation: "all" (audio_mac.py, PyObjC; no model, about 70 MB). On an M5's speakers it brings the agent's own voice at the mic from −21 dBFS down to −78 dBFS, below the room's noise, so you can interrupt it by talking. A few things to know:

• It cancels everything the Mac plays, not just the agent: a video playing in the background won't trigger it either.
• Talking over the reply stops it speaking after 250 ms of speech, so a cough doesn't. Speech in the first 0.5 s of a reply is ignored: the first reply of a session leaks a syllable or two of echo while the canceller learns the room.
• It uses the system default mic and speaker, and turns other apps' audio down only slightly while it runs. Choosing a device with --input-device or --output-device turns it off, and if voice processing won't start on your audio hardware macvoice says so and runs half duplex.
• --no-aec turns it off: the loop is then half duplex, the mic muted while it speaks. With headphones, --no-aec --barge-in lets you interrupt without echo cancellation.

The ready: line at startup says which mode you're in.

Pick a brain

Setup details for each option are in docs/llm-setup.md.

# Default: the local server on port 8080 (mlx_lm.server above), else echo
uv run macvoice run

# Any other OpenAI-compatible endpoint: Ollama, LM Studio, llama.cpp, or a hosted API
uv run macvoice run --base-url http://localhost:11434/v1 --model llama3.2

# Just the audio loop, even if a server is running
uv run macvoice run --llm echo

# Claude Code over ACP (npm i -g @agentclientprotocol/claude-agent-acp)
uv run macvoice run --llm acp

# Any other ACP agent
uv run macvoice run --llm acp --acp-command "gemini --experimental-acp"

ACP agents keep their own conversation. Without --cwd, the agent gets a scratch folder and every tool request is declined, so it acts as a chat brain.

Talk to a coding agent while it works

Give the agent a project folder and it can read, edit and run things there. It tells you what it did in a sentence or two instead of reading code aloud.

uv run macvoice run --llm acp --cwd ~/demo-site                     # the folder is created if needed
uv run macvoice run --llm acp --cwd ~/demo-site --mode acceptEdits  # edits go through; commands still ask
uv run macvoice run --llm acp --cwd ~/demo-site --model sonnet --effort low  # the agent's own model and effort settings
uv run macvoice modes                                               # the agent's modes, models and effort levels

When the agent wants to run a tool, it asks out loud ("Claude wants to create the site folder. Allow?") and prints the exact command. Say yes or no, or press y, n or a (allow always). Anything unclear gets asked once more, then declined, and so does silence for 20 seconds. Nothing heard while the question is playing counts as an answer.

--permissions allow approves every request without asking, and --permissions deny declines them all. --mode takes any mode the agent advertises, by id or name, so it works with any ACP agent. --model and --effort work the same way for agents that offer those settings (Claude Code: opus, sonnet, haiku, ... and low to max). While a slow reply hasn't started, a soft tone plays once a second so you know it's working (--no-thinking-tone turns it off).

Talking over it stops the voice, not the agent. When you talk over a reply, or while the agent is still working, it goes quiet for the rest of that turn but keeps working. The whole reply still lands on screen, and what you said goes to the agent as the next prompt once the turn ends (it prints queued: ...). If nothing is queued, two soft blips tell you when the agent is free again (--no-thinking-tone turns these off too). To actually cancel the agent's turn, say just "stop" (or "cancel", "stop that"), or press Ctrl+C or Esc. Permission questions are always spoken, even after you've talked over the reply.

Caveats:

• The agent really runs things. With --permissions allow, or a mode such as bypassPermissions, it runs commands and edits files without asking. Use those only in a folder you don't mind losing. In acceptEdits mode, file edits go through without asking; commands still ask.
• Coding agents are slow to start talking. The voice pipeline answers in about 350 ms, but Claude Code took 3–4.5 s to start replying to simple requests in testing, and longer when it's reading files or running commands. That's what the thinking tone is for.
• One-word answers are near the floor. A turn needs 250 ms of speech, and a quick "no" can fall under it. A missed answer is asked again, and silence declines. The keys always work.
• "Always" is up to the agent. macvoice passes the agent's own "allow always" option back; how long that lasts (this session, this folder) is the agent's rule, not macvoice's.
• Tested with Claude Code (claude-agent-acp). Other ACP agents should work, since modes and permission requests are part of the protocol, but their modes and request details differ.
• What you say waits for the turn to finish. Talking over the agent doesn't cancel its turn, so your next prompt waits until it's done, which can take a while if it's in the middle of a long change. Say "stop" to cancel instead.
• "Stop" can cut it off mid-change. Cancelling (session/cancel) can land partway through a multi-step change. Ask it what state things are in.
• The agent doesn't know where you stopped listening. Its history has the whole reply, so it may refer to something you didn't hear. It's on screen.

Other commands

uv run macvoice simulate recording.wav -o reply.wav   # run a recording through the loop, no devices
uv run macvoice transcribe a.wav b.wav                 # Parakeet only
uv run macvoice say "Hello there" [-o hello.wav]       # Phonon only
uv run macvoice run --voice Harper                     # a Phonon catalog voice
uv run macvoice run --turns 3                          # exit after three replies (voice only)
uv run macvoice modes [--acp-command "..."]            # an ACP agent's modes, models and effort levels

Each turn prints where the time went:

  you : How many days are in a leap year?
  bot : A leap year has 366 days.
        endpoint 301 ms (detector, p=0.99) | asr 54 ms | llm 404 ms | tts 10 ms | => first audio 769 ms after you stopped

How turns end

After 300 ms of silence, the turn's audio, trimmed to where speech stopped, goes to Smart Turn. If it says the speaker is done (P ≥ 0.5) the turn ends immediately; otherwise the loop keeps listening for up to 3 s, and speech that resumes continues the same turn. --no-turn-detector falls back to a plain 1.5 s silence timer. The logic is a pure class, endpointing.py, driven chunk by chunk and unit-tested with synthetic VAD probabilities.

Benchmarks

uv run macvoice bench stack --stack new --out results/new.json
uv run macvoice bench turns --out results/turns.json

# The PyTorch baseline (Whisper large-v3 + Kokoro), in its own environment
UV_PROJECT_ENVIRONMENT=.venv-baseline uv sync --extra baseline
.venv-baseline/bin/macvoice bench stack --stack baseline --out results/baseline.json
uv run macvoice bench compare results/baseline.json results/new.json

The results from the article (M5, 32 GB, macOS 26.6, on AC power) are in results/.

Tests

uv run pytest                 # everything (~20 s on an M5)
uv run pytest -m "not models" # unit tests only, no model downloads

Tests marked devices play sound and use the real mic and speaker to check echo cancellation, including a full live loop that is interrupted mid-reply; they're opt-in: MACVOICE_DEVICE_TESTS=1 uv run pytest -m devices (in a quiet room). macOS cancels any sound the Mac itself plays, so the "person" interrupting is mixed into the echo-cancelled mic signal; MACVOICE_HUMAN_TESTS=1 ... -s asks you to do it for real.

Unit tests cover the model list and first-run download, the status line, the endpointer, sentence splitting, the OpenAI-compatible client and brain selection (against a local SSE server), the ACP client, permission questions and modes (against a scripted fake agent), spoken and keyboard answers, the thinking tone, the console's keys and commands (through a pipe), what the voice leaves out of markdown, the live loop's half duplex, echo guard and barge-in rules, the streaming resampler for the voice-processing mic, and the NumPy feature ports (pinned to the PyTorch/transformers reference outputs). Tests marked models run the real models: Parakeet accuracy, the published 8-bit Parakeet matching a local conversion (when the original checkpoint is cached), Phonon → Parakeet round trips, the full loop on 32 spoken prompts, the 2 GB ceiling with no PyTorch, and Claude Code over ACP (skipped if claude-agent-acp isn't installed).

Models

Every model is pinned to a commit (Silero to a release tag plus its SHA-256). The published Parakeet checkpoint is fp32 and 2.3 GB; macvoice runs it in bf16 with the encoder quantized to 8 bits, so it downloads that conversion (0.7 GB) instead. It is made by scripts/make_parakeet_q8.py with the same code macvoice falls back to if the copy can't be downloaded, and a models test checks the two are identical. Other --precision values download the original and convert it at load.

Licenses

Code: Apache-2.0. Models and test data have their own licenses; see NOTICE.