Skip to content

voice-mode.md

Latest commit

 

History

History
113 lines (91 loc) · 5.93 KB

voice-mode.md

File metadata and controls

113 lines (91 loc) · 5.93 KB
Raw

Voice in OpenMausBot

Decision doc, 2026-08-14. How bots speak, and how you hold a conversation with one.

Shape

Renderer (src/)                        Harness (server/)
├── lib/tts/index.ts   the speaker     ├── tts/speech-text.ts  markdown → speakable
│     queue · prefetch · interrupt     └── tts/elevenlabs.ts   verify · voices · synthesize
└── components/CallView.tsx
      Apple STT endpointing            POST /api/tts/prepare → utterances
                                       POST /api/tts/speak   → mp3 bytes

One voice provider: ElevenLabs, bring your own key. No local model, no second provider, no fallback ladder — if there is no key, voice is off and the buttons say so.

Why the key stays on the harness

The renderer never talks to ElevenLabs. GET /api/config reports configured-or-not booleans and nothing else, which is the same rule every other credential follows, and it is worth more than a saved round trip. So the app asks the harness for audio and the harness holds the key.

Two states worth distinguishing, because they need different instructions: configured (a key is saved) and ready (a key and a chosen voice). Speaking without either throws NoVoiceConfigured, which the route turns into a 409 — "you haven't set this up" is not a provider failure and should not look like one.

The spoken register

The half that decides whether this is pleasant. Agents write for a screen: fenced code, file paths, tables, link soup. Read aloud verbatim, a diff is four minutes of punctuation and server/drivers/acp/core.ts is "server slash drivers slash a c p slash core dot t s".

speech-text.ts says the prose, names the artifacts, drops the syntax. It also splits into utterances, because that is the unit of work — one request, one clip, and the client fetches the next while the current one plays. One request per utterance rather than the streaming-input WebSocket: same perceived latency, far fewer moving parts, and no socket to leak when a turn is interrupted.

Call mode

Half-duplex, on purpose. The dictation helper is SFSpeechRecognizer on raw AVAudioEngine input with no acoustic echo cancellation. A microphone left open through playback transcribes the bot's own voice back into the conversation and the two of them talk forever. So the mic is live only when the bot is not speaking, and interrupting is a tap, the Space bar, or Escape. Full-duplex barge-in needs AEC on the capture path — a real follow-up, not a footnote.

Turn detection stays native and local. A buffer-backed SFSpeechRecognizer does not emit isFinal just because the speaker becomes quiet; it finalizes only after its audio stream ends. Call mode therefore starts the native helper with a silence timeout. Once a non-empty transcript stops changing for 850ms, the helper stops capture and calls endAudio(), which produces the final transcript sent to the renderer. Composer dictation omits the timeout and keeps its press-to-stop behavior. No cloud STT or bundled VAD model is involved.

Narration is what makes it bearable. An agent turn is 5–60 seconds of tool calls, and silence that long reads as a dropped call. Every activity chip the harness narrates is read aloud as it happens. The phrase is computed once, server-side, into tool.spoken at fold time — so the chip you see and the phrase you hear cannot drift apart.

Approvals are spoken. A request.opened card is read out and answered with "yes"/"no". Anything that is not clearly a decision is refused and re-asked: consent must never be inferred from a sentence that merely contained the word "sure". Non-permission questions are read too, and the next complete spoken turn is returned as the answer, so an agent asking for input does not strand the call behind an invisible card.

Latency, honestly. Endpointing is 300–700ms and time-to-first-byte is ~100–250ms, against an agent turn of 5–60s. The agent dominates by 50–100x, so voice choice is a quality decision, not a latency one. The way to make a call feel conversational is to put the bot you call on a fast model and let it delegate real work to specialists over ask_bot — no new machinery required.

Rejected

Option Why not
OS voices (macOS/Windows) Audibly synthetic; would cheapen the feature
Piper Same complaint, one tier up
Kokoro-82M in the renderer Genuinely good and free, but it is a second provider, a 2.2MB chunk, an ONNX runtime and a first-run model download. Simplicity won.
Cartesia Cheaper and faster to first byte, but a second provider earns its keep only once one is not enough
ElevenLabs Agents Its custom-LLM cascade_timeout_seconds maxes at 15s and agent turns exceed that; it also wants to own turn-taking and tool calls, which is what the harness owns
OpenAI Realtime / Gemini Live (speech-to-speech) They replace the brain, and the brain being Claude Code on your own machine is the product

Known gaps

  • Calls are macOS-only, because dictation is. The voice half works everywhere.
  • Rooms don't speak yet, though per-bot voices already exist (bot.voice).
  • No spend meter. ElevenLabs bills per character. Auto-speak is off by default partly for that reason, but the app should eventually show usage.
  • No voice barge-in — see half-duplex above.

Failure boundaries

  • Intentional microphone stops (playback, hang-up, or replacement) do not emit a natural speech:end; otherwise the renderer could reopen capture during the bot's audio.
  • Call phases are updated synchronously alongside React state, so a helper exit in the same event-loop turn as a final transcript cannot observe a stale listening phase.
  • Leaving the bot view owns and ends its call. A hidden overlay cannot leave a microphone session or a stale currentCall behind.
  • Synthesis requests are abortable from the renderer and individual utterances are capped server-side to bound accidental hosted-voice spend.