Skip to content

feat: enhance audio backend detection for Linux/WLS2 and input devi…#10

Open
crixue wants to merge 3 commits into
renjfk:mainfrom
crixue:main
Open

feat: enhance audio backend detection for Linux/WLS2 and input devi…#10
crixue wants to merge 3 commits into
renjfk:mainfrom
crixue:main

Conversation

@crixue

@crixue crixue commented Jul 21, 2026

Copy link
Copy Markdown

feat: enhance audio backend detection for Linux/WSL2 and input device handling

Why

The STT pipeline previously assumed macOS CoreAudio for both recording and device listing. This prevented Linux and WSL2 users from selecting the correct microphone, and caused sox to fail or record from the wrong device when no CoreAudio backend was available. A single cross-platform audio backend abstraction was needed.

What Changes

  • README.md: rewrote the Speech-to-text section into macOS and Linux/WSL2 subsections. Added microphone verification steps, Linux CPU/CUDA build instructions with a GPU architecture table, a smoke-test section, and updated /stt-mic and STT pipeline descriptions.
  • lib/stt.js:
    • Added detectAudioBackend() to choose coreaudio on macOS, pulseaudio when pactl exists, or the sox default otherwise.
    • Added parsePactlSources() and parsePactlSourcesShort() to parse pactl output and exclude .monitor sources.
    • Refactored listInputDevices() to return { name, label } descriptors per backend.
    • Added buildRecordArgs(backend, mic) to generate the correct sox -t arguments for each backend.
    • Added backend logging at recording start and registerSTT time.
    • Added medium-q5_0 to the model registry.
  • test/stt.test.js: added tests for parsePactlSources, parsePactlSourcesShort, and buildRecordArgs.

Capabilities

  • macOS users continue to use CoreAudio input devices with no behavior change.
  • Linux and WSL2 users can now select real PulseAudio input sources via /stt-mic.
  • Recording automatically uses the pulseaudio sox driver on Linux when pactl is present, falling back to the default sox device otherwise.
  • Users can verify the active whisper.cpp backend (METAL / CUDA / CPU) and expected performance from the README.

Impact

  • Improved first-run experience on Linux/WSL2 with clear setup and verification steps.
  • More reliable microphone capture across platforms by matching the sox driver to the OS audio stack.
  • Better diagnostics via backend-aware logging.
  • No breaking changes for existing macOS users; stored stt.mic values remain valid.

feat: add support for selecting transcription language in STT

Why:

whisper-cli defaults to -l en, so the plugin silently forced English decoding for all users — non-English speech was transcribed poorly with no way to fix it.

What Changes:

  • New exported buildWhisperArgs() pure function assembles whisper-cli args with a language flag
  • Language resolution follows the existing pattern: kv("stt.language") overrides options.sttLanguage, falling back to auto
  • New /stt-language command with a curated language picker (auto + 11 common languages)
  • Unknown language codes are detected from stderr and reported as errors (whisper-cli exits 0 even on invalid languages)
  • Tests added for buildWhisperArgs; README and command docs updated

Capabilities:

  • Users can set any whisper.cpp language code via the sttLanguage plugin option, or switch at runtime among common languages via /stt-language
  • Out of the box, auto-detection now works for any spoken language
  • Invalid language codes surface a clear error instead of a misleading "no speech detected"

Impact:

  • Non-English users get correct transcription with zero configuration
  • Existing English users see a minor behavior change (default is now auto instead of en): a small language-detection overhead and occasional misdetection on very short clips, recoverable via /stt-language or sttLanguage
  • STT API transcription path is unaffected (local whisper-cli only)

feat: add WSL audio server diagnostics and hints for recording failures

Why

On WSL2, WSLg's PulseAudio server can wedge: its listen backlog (hardcoded to 5 in PulseAudio) fills up with orphaned connections that are never accept()ed, so every new client gets Connection refused. The plugin made this failure invisible at three points:

  • detectAudioBackend() probes with pactl --version, which never connects to the server and reports "pulseaudio" even when the server is dead
  • /stt-mic swallowed listing errors and toasted a misleading "No input devices found"
  • startRecording surfaced only sox's raw, cryptic stderr (FAIL formats: can't open input...)
    Users on WSL had no way to tell "no microphone exists" apart from "the audio server is stuck, restart WSL".

What Changes

  • lib/stt.js — new diagnostics block:
  • isWSL() (exported): detects WSL via WSL_DISTRO_NAME / WSL_INTEROP
  • pulseServerHealth(): probes with pactl info, which actually connects to the server (unlike pactl --version)
  • buildAudioHint() (exported, pure): builds a user-facing hint per backend / server state / WSL-or-not; on WSL it explicitly advises wsl --shutdown on Windows
  • audioFailureSuffix() (internal): appends the hint to recording-failure toasts when the PulseAudio server is unreachable
  • Wiring: /stt-mic's empty-device branch now health-checks first and toasts/logs the hint; both sox error and non-zero exit paths in startRecording append the same hint. detectAudioBackend() is unchanged — "pulse stack exists" and "server is healthy" stay separate concerns
  • test/stt.test.js — 2 new tests: buildAudioHint branch coverage (WSL / non-WSL / healthy / non-pulse) and isWSL() env detection with env restoration
  • README.md — new "WSL2 audio troubleshooting" note: no /dev/snd is normal, Connection refused → wsl --shutdown, Windows microphone privacy settings, wsl --update (plus oxfmt whitespace-only reformat of two existing tables)

Capabilities

  • Users hitting a dead WSLg audio server now see an actionable message — "Audio server unreachable. On WSL, WSLg's PulseAudio may be stuck - run wsl --shutdown on Windows, then reopen Ubuntu" — instead of "No input devices found" or raw sox stderr
  • Non-WSL Linux users get a corresponding PipeWire/PulseAudio service hint; a reachable server with zero sources gets a "check your audio input source configuration" hint
  • Diagnosis runs lazily (only on failure paths), with a 3 s timeout, so the happy path is untouched

Impact

  • Behavior change is additive: only error/empty-result paths change; recording, device listing, and backend detection on healthy systems are byte-for-byte identical
  • No new dependencies, no config or API surface changes; 16/16 tests pass, npm run check clean
  • Verified live on a wedged WSLg instance: the hint correctly triggers with isWSL: true, serverOk: false
  • Full audio recovery itself still requires the user to run wsl --shutdown on Windows — the plugin now tells them exactly that

@crixue crixue changed the title feat: enhance audio backend detection for different OS and input devi… feat: enhance audio backend detection for Linux/WLS2 and input devi… Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant