Add local Whisper speech input and managed model setup

.github/workflows/ci.yml

@@ -15,8 +15,10 @@ jobs:
         with:
           node-version: '26.5.1'
           cache: npm
+      - run: sudo apt-get update && sudo apt-get install -y ffmpeg
       - run: npm ci
       - run: npm run typecheck
       - run: npm test
+      - run: python3 scripts/test_local_whisper.py
       - run: npx playwright install --with-deps chromium
       - run: npm run test:e2e

README.md

@@ -115,3 +115,23 @@ screenshots. For sensitive reports, follow [SECURITY.md](SECURITY.md).
 ## License
 
 Agentflow is licensed under the [MIT License](LICENSE).
+
+## Fully local transcription
+
+Select **Local Whisper** under Speech & settings → Listening. It runs speech recognition on your Mac with no API key or 9Router. Speaking still uses the independently selected TTS provider, and Codex/Claude retain their own model routing; local transcription does not make an agent conversation fully offline.
+
+Install `ffmpeg`, Python 3, and the pinned `whisper-server` runtime described in [local runtime setup](docs/local-whisper-runtime.md). Then explicitly prepare and start the engine:
+
+```sh
+agentflow local-whisper setup
+agentflow local-whisper start
+agentflow local-whisper status
+```
+
+The default URL is `http://127.0.0.1:17864`. To use an existing engine, enter its loopback HTTP origin in **Local Whisper URL**. The model setting is a label for the server's loaded model; changing it does not load a different model. Models are never downloaded while recording or during login. `agentflow local-whisper stop` only stops the engine owned by Agentflow. The helper accepts `import-model PATH --sha256 HASH` for an explicitly trusted local model; use `--help` for options. Agentflow's helper state lives beneath `$AGENTFLOW_HOME/local-whisper` (default `~/.config/agentflow/local-whisper`).
+
+Browser WebM/MP4/WAV recordings are converted in memory by ffmpeg to 16 kHz mono PCM WAV, with a 90-second duration limit, 20 MB input limit and 15-second conversion deadline. Set `AGENTFLOW_FFMPEG` to an absolute executable path if needed. No conversion files are retained. The local endpoint cannot contain remote hosts, credentials, paths, queries or redirects, and its transport bypasses global HTTP proxies. Silent/near-silent recordings do not reach Whisper. Very quiet speech may need a closer microphone.
+
+Each recording captures its listening settings before microphone permission is requested. Canceling or changing sessions discards late transcription and cannot submit it to the agent. Canceling closes local transport and terminates conversion, but whisper-server may continue inference internally; this is not a claim that HTTP disconnect preempts GPU computation. Inference has a 60-second timeout; check `agentflow local-whisper status`, the server's loaded model and the configured URL after engine errors.
+
+Tests and TDD evidence: [local transcription validation](docs/local-whisper-validation.md). Real microphone permission, audible playback and sleep/wake still require device acceptance.

docs/local-whisper-qa.json

@@ -0,0 +1,292 @@
+{
+  "date": "2026-10-15",
+  "engine": "whisper.cpp 1.9.1",
+  "model": "base.en",
+  "model_sha256": "a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002",
+  "hardware": "Apple M3, 24 GiB unified memory",
+  "isolation": "macOS sandbox-exec denied external network for both server and clients; loopback allowed. External hostname and 1.1.1.1 curl probes failed.",
+  "benchmark": {
+    "description": "Raw engine synthetic TTS fixtures, not human microphone accuracy",
+    "clean_wer": 0.005988023952095809,
+    "clean_warm_p95_seconds": 0.138,
+    "results": [
+      {
+        "fixture": "clean1",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.116,
+        "reference": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+        "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+        "word_errors": 0,
+        "reference_words": 13
+      },
+      {
+        "fixture": "clean2",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.12,
+        "reference": "The weather is clear today. I will walk to the store and buy some coffee before lunch.",
+        "text": "The weather is clear today. I will walk to the store and buy some coffee before lunch.",
+        "word_errors": 0,
+        "reference_words": 17
+      },
+      {
+        "fixture": "clean3",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.13,
+        "reference": "We finished the first draft yesterday. The next step is to review the changes and prepare the final report.",
+        "text": "We finished the first draft yesterday. The next step is to review the changes and prepare the final report.",
+        "word_errors": 0,
+        "reference_words": 19
+      },
+      {
+        "fixture": "clean4",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.132,
+        "reference": "Could you check the schedule and tell me whether we have enough time to finish this task today?",
+        "text": "Could you check the schedule and tell me whether we have enough time to finish this task today?",
+        "word_errors": 0,
+        "reference_words": 18
+      },
+      {
+        "fixture": "clean5",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.128,
+        "reference": "This recording is a test of local speech recognition. No internet connection should be needed to understand these words.",
+        "text": "This recording is a test of local speech recognition. No internet connection should be needed to understand these words.",
+        "word_errors": 0,
+        "reference_words": 19
+      },
+      {
+        "fixture": "clean6",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.135,
+        "reference": "After breakfast we will take the train into the city and meet our friends near the museum.",
+        "text": "After breakfast we will take the train into the city and meet our friends near the museum.",
+        "word_errors": 0,
+        "reference_words": 17
+      },
+      {
+        "fixture": "clean7",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.125,
+        "reference": "Remember to close the window before leaving the house. The forecast says it will rain later this afternoon.",
+        "text": "Remember to close the window before leaving the house. The forecast says it will rain later this afternoon.",
+        "word_errors": 0,
+        "reference_words": 18
+      },
+      {
+        "fixture": "clean8",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.128,
+        "reference": "Please save a copy of the document and put it in the folder for next week.",
+        "text": "Please save a copy of the document and put it in the folder for next week.",
+        "word_errors": 0,
+        "reference_words": 16
+      },
+      {
+        "fixture": "clean9",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.126,
+        "reference": "The package arrived this morning. Everything was packed carefully and nothing appears to be missing.",
+        "text": "The package arrived this morning, everything was packed carefully, and nothing appears to be missing.",
+        "word_errors": 0,
+        "reference_words": 15
+      },
+      {
+        "fixture": "clean10",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.12,
+        "reference": "Let us start with the simplest example and make sure it works before adding any more features.",
+        "text": "Let us start with the simplest example and make sure it works before adding any more features.",
+        "word_errors": 0,
+        "reference_words": 17
+      },
+      {
+        "fixture": "clean11",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.13,
+        "reference": "I would like a table for four people on Saturday evening, preferably beside a window.",
+        "text": "I would like a table for four people on Saturday evening, preferably beside a window.",
+        "word_errors": 0,
+        "reference_words": 15
+      },
+      {
+        "fixture": "clean12",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.123,
+        "reference": "Thank you for your help with the project. Your comments made the instructions much easier to follow.",
+        "text": "Thank you for your help with the project. Your comments made the instructions much easier to follow.",
+        "word_errors": 0,
+        "reference_words": 17
+      },
+      {
+        "fixture": "clean13",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.131,
+        "reference": "Turn left at the next street and continue for two blocks. The library is on the right.",
+        "text": "Turn left at the next street and continue for two blocks. The library is on the right.",
+        "word_errors": 0,
+        "reference_words": 17
+      },
+      {
+        "fixture": "clean14",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.112,
+        "reference": "We should test the new settings before the meeting starts so that everyone can hear clearly.",
+        "text": "We should test the new settings before the meeting starts so that everyone can hear clearly.",
+        "word_errors": 0,
+        "reference_words": 16
+      },
+      {
+        "fixture": "clean15",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.12,
+        "reference": "The team agreed to fix the current problems before beginning work on the next version.",
+        "text": "The team agreed to fix the current problems before beginning work on the next version.",
+        "word_errors": 0,
+        "reference_words": 15
+      },
+      {
+        "fixture": "clean16",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.12,
+        "reference": "There are several ways to solve this problem. Let us compare them carefully and choose one.",
+        "text": "There are several ways to solve this problem. Let us compare them carefully and choose one.",
+        "word_errors": 0,
+        "reference_words": 16
+      },
+      {
+        "fixture": "clean17",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.132,
+        "reference": "I left my notebook on the kitchen table. Please bring it with you when you come over.",
+        "text": "I left my notebook on the kitchen table. Please bring it with you when you come over.",
+        "word_errors": 0,
+        "reference_words": 17
+      },
+      {
+        "fixture": "clean18",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.138,
+        "reference": "The train leaves at nine tomorrow morning. We should arrive at the station at least twenty minutes early.",
+        "text": "The train leaves at 9 tomorrow morning. We should arrive at the station at least 20 minutes early.",
+        "word_errors": 2,
+        "reference_words": 18
+      },
+      {
+        "fixture": "clean19",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.128,
+        "reference": "Put the blue box on the shelf and leave the smaller green box beside the door.",
+        "text": "Put the blue box on the shelf and leave the smaller green box beside the door.",
+        "word_errors": 0,
+        "reference_words": 16
+      },
+      {
+        "fixture": "clean20",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.136,
+        "reference": "The lights are working again. We can finish cleaning the room and move the chairs back into place.",
+        "text": "The lights are working again. We can finish cleaning the room and move the chairs back into place.",
+        "word_errors": 0,
+        "reference_words": 18
+      },
+      {
+        "fixture": "technical",
+        "duration_seconds": 10,
+        "elapsed_seconds": 0.15,
+        "reference": "Run the unit tests, review the pull request, and restart the local Whisper server. Check the TypeScript compiler and the Rust build.",
+        "text": "Run the unit tests, review the pull request, and restart the local whisper server. Check the TypeScript compiler and the REST build.",
+        "word_errors": 1,
+        "reference_words": 22
+      }
+    ]
+  },
+  "agentflow_adapter": [
+    {
+      "name": "clean1.wav",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.407278791
+    },
+    {
+      "name": "clean1.webm",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.13789725
+    },
+    {
+      "name": "clean1.mp4",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.12846329200000003
+    },
+    {
+      "name": "noise.wav",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.12652508299999998
+    },
+    {
+      "name": "silence.wav",
+      "text": "",
+      "elapsed_seconds": 0.02996124999999995
+    }
+  ],
+  "managed_runtime": {
+    "two_engines_simultaneously": true,
+    "idempotent_start": true,
+    "stop_isolated_in_both_directions": true,
+    "restart_without_external_network": true,
+    "state_paths_with_spaces": true,
+    "observed_start_seconds": [
+      0.441,
+      7.837,
+      0.335
+    ]
+  },
+  "murmur_adapter": {
+    "clean_wav_expected_transcript": true,
+    "digital_silence_empty": true,
+    "varied_120_second_audio_transcribed": true
+  },
+  "agentflow_limits": {
+    "varied_90_second_audio_transcribed": true,
+    "seconds": 1.099,
+    "120_second_audio_rejected": true
+  },
+  "memory": {
+    "server_rss_kib_snapshot": 311136,
+    "note": "Single ps RSS snapshot after workload; not peak usage and does not fully account for Metal/unified GPU allocations."
+  },
+  "limitations": [
+    "Synthetic macOS say voice only, not a representative human/accent/microphone corpus.",
+    "Raw engine hallucinated you on digital silence; app guards returned empty. Noisy silence may still hallucinate.",
+    "Technical fixture misrecognized Rust as REST.",
+    "Aborting HTTP discards output but may not immediately stop engine CPU/GPU work.",
+    "Native microphone, actual hotkeys/focus/paste, sleep/wake and real browser device acceptance remain manual gates.",
+    "Native-agent reasoning and TTS are independent; this validates offline STT, not fully offline Codex/Claude conversations."
+  ],
+  "agentflow_adapter_final_transport": [
+    {
+      "name": "clean1.wav",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.161127833
+    },
+    {
+      "name": "clean1.webm",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.127854666
+    },
+    {
+      "name": "clean1.mp4",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.125231583
+    },
+    {
+      "name": "noise.wav",
+      "text": "Please move the meeting to Thursday morning and send everyone an updated invitation.",
+      "elapsed_seconds": 0.12394433300000003
+    },
+    {
+      "name": "silence.wav",
+      "text": "",
+      "elapsed_seconds": 0.026634292000000073
+    }
+  ]
+}

docs/local-whisper-qa.md

@@ -0,0 +1,68 @@
+# Local Whisper QA — 2026-10-15
+
+This evidence covers the new local STT adapters and managed runtime. No personal
+recordings or native agent sessions were used. Detailed synthetic references,
+transcripts, timings and limitations are in [the JSON report](local-whisper-qa.json).
+
+## Environment and results
+
+- Apple M3, 24 GiB unified memory; whisper.cpp 1.9.1 with Metal; pinned base.en
+  SHA-256 `a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002`.
+- Both server and test clients ran under the included macOS sandbox profile,
+  which denies external networking and permits loopback. External hostname and
+  numeric-IP probes failed; local inference succeeded.
+- Twenty synthetic clean-English utterances, each padded to exactly 10 seconds:
+  normalized WER **0.60%**; raw-engine warm p95 **0.138 seconds** across 19 calls
+  after the first request. These meet the proposed 10%/3-second gates for this
+  narrow corpus only. Audio generation/conversion is outside raw inference timing.
+- Technical fixture: one error, “Rust” → “REST”. This is not a guarantee of
+  human microphone, accent, vocabulary, or noisy-room accuracy.
+- Murmur's compiled adapter transcribed clean WAV and varied 120-second speech;
+  digital silence returned empty. Its capture tests separately verify draining,
+  resampling and completion through the local adapter.
+- Agentflow's compiled adapter transcribed WAV, WebM, MP4 and noisy WAV locally;
+  silence returned empty. Observed short-input adapter times were 0.128–0.407s,
+  including local format conversion. After the direct-HTTP transport hardening,
+  repeated speech checks passed in 0.124–0.161s and silence in 0.027s.
+  A 90-second fixture transcribed in 1.099s;
+  120 seconds failed with the documented 90-second limit.
+- Two independently managed engines ran simultaneously. Repeated start was
+  idempotent. Stopping either preserved the other; offline restart passed.
+  State directories contained spaces. Startup ranged 0.335–7.837s in these runs.
+- Server RSS snapshot: 311,136 KiB. This is not a peak or full Metal memory metric.
+
+## Reproduce the raw-engine benchmark
+
+Install runtime/model explicitly using the runtime guide first. Start an engine
+with networking restricted (use a free port and your model path):
+
+```sh
+sandbox-exec -f scripts/local-whisper-offline.sb whisper-server \
+  --host 127.0.0.1 --port 17865 -m /absolute/path/to/model.bin
+```
+
+In another terminal:
+
+```sh
+sandbox-exec -f scripts/local-whisper-offline.sb python3 \
+  scripts/benchmark-local-whisper.py --url http://127.0.0.1:17865 \
+  --output /tmp/local-whisper-benchmark.json
+```
+
+The script uses macOS `say` and `ffmpeg`, reads committed synthetic reference
+text, generates temporary audio, and deletes that audio when it finishes. It
+never downloads a model or submits anything to a native agent. This benchmark
+tests the raw engine; the app-specific adapter tests remain necessary.
+
+## Remaining gates and limits
+
+Physical microphone/permission checks, real Control hold/release and focused
+app insertion, sleep/wake, and browser hardware acceptance have **not** been
+performed. Keep those QA issue checkboxes open. Automated capture, browser and
+session fixtures do not prove physical-device behavior.
+
+Raw Whisper produced “you” on digital silence. Both adapters prevent this for
+their silence thresholds; noisy silence can still hallucinate. Aborting an HTTP
+request prevents insertion/submission but does not promise immediate engine
+compute interruption. Local STT does not make separately configured TTS or
+native Codex/Claude reasoning local.