AI auto scripture: local transcription + AI reference detection (opt-in) - #3579
AI auto scripture: local transcription + AI reference detection (opt-in)#3579josephomills wants to merge 252 commits into
Conversation
3083d2f to
f187e0c
Compare
|
Pushed a round of updates from live field testing at a conference, plus a fully-local AI option:
141 unit tests passing. The PR description's update section has the full list. |
3aca772 to
0e03d7d
Compare
|
Left a longer note on #3547 about complementary fully local streaming work I have been testing (no API keys / no LLM for the core path). Happy to collaborate or wait for maintainer preference so we do not step on this review. |
|
@AlloDel your driver is merged and wired up — engine selector in settings, resumable model download, packaging config. Thank you, the VAD choice and the Electron notes were spot on. One change to your driver, after live testing. The decoder deafness you documented goes deeper than stream reuse: any chunked feeding (fresh, reused, or persistent stream) intermittently decodes short utterances like "next verse" to nothing. The same samples in one batch acceptWaveform on a fresh stream decode correctly every time. So the driver now buffers each utterance (1s pre-roll) and batch-decodes when the VAD closes, waiting 0.5s of real tail audio first so a soft last word isn't cut mid-word. VAD retuned: threshold 0.3, min-silence 0.8s, min-speech 0.15s. If your live-testing notes contradict any of this, I'd like to know. What I'd like you to review: downloading the sherpa runtime on demand — finishing your zero-setup goal. Right now the npm natives ship in the installer (~16 MB/platform) for everyone. The plan, mirroring our ffmpeg/whisper downloaders: Exact-pin sherpa-onnx-node@1.13.4; exclude the platform packages from electron-builder, keep the JS wrapper bundled Does the sibling-layout require() hold across sherpa-onnx-node versions you've used? |
5c74a02 to
5204dc3
Compare
|
I will review this soon. But just a couple things for now I would like changed:
|
|
Alright @vassbo |
New AI_SCRIPTURE_* channels (typed in Main/ToMain), audio data filtered from IPC logging like TIMECODE_AUDIO_DATA, and the orchestrator that owns the transcriber + detection lifecycle. API keys live in the ACCESS store (non-portable, never cloud-synced) and only reach the renderer as booleans.
- whisperManager: binary/model lifecycle following the ffmpegManager pattern (Windows auto-download, system PATH probe, custom path, Hugging Face ggml model downloads with progress) - transcriber: 16kHz PCM ring buffer, RMS silence gate, 7s windows with 1s overlap, whisper-cli per window or persistent whisper-server, noise/low-confidence segment filtering - detection: tier-1 local reference matching (spoken word-numbers, ordinal book prefixes, cue-gated confidence) and tier-2 LLM detection with dedupe, cooldowns and single-flight scheduling - providers: Anthropic / OpenAI / Gemini over plain REST (axios) with structured JSON output, normalized error codes and key test calls - 66 colocated vitest tests
AudioWorklet mirrors the LTC timecode listener (48k to 16k mono Int16, 1s chunks over IPC). The dispatcher validates detections against the selected bibles (collection-aware), verifies quoted verses against actual verse text across translations, clamps ranges, and projects via activeScripture + playScripture so outLocked, history and templates all apply. Auto mode is confidence-gated with a min-display queue, and a manual override watcher pauses auto-projection whenever the operator changes the output slide themselves.
Collapsible panel in the scripture drawer with live transcript view and one-click suggestion cards; setup popup covering whisper install per platform (with custom binary and model paths), provider API keys with test, search-bible multi-select with select all, display translation and behavior settings. Feature is off by default behind a toggle in Settings > Other with a persistent privacy note.
- Anthropic: select the text content block instead of content[0] (thinking models lead with a thinking block) and raise detection max_tokens to 4096 - never abort an in-flight LLM call for a new segment; mark a rerun and refresh with the newest transcript once it settles (a >6s provider previously starved tier-2 completely during continuous speech) - unify book numbering on the 66-book canon via a canonNumber book-table field so tier-1/tier-2 dedupe can't miss across domains - pause tier-2 on permanent errors (model_not_found/invalid_request) - redact API-key-shaped strings from provider error messages - replace retired/nonexistent Gemini model ids with gemini-2.5-flash/pro
- 30s watchdog on whisper-cli runs (SIGTERM then SIGKILL) - overlap dedupe trims segment text, not just timestamps - no child can spawn after stop() (re-checked after every await) - successful server respawn resets the failure counter - server health check verifies the responder is whisper-server before any audio is posted; respawns on a fresh port otherwise - temp dir 0700 / WAV files 0600
|
@josephomills The plan is to review this soon, and get it merged as a hidden/experimental feature before the next beta release in a few weeks. |
|
Great! Looking forward to a lot of feedback from different speakers/preachers, different languages & different computers. I'm also looking up ways to simplify this feature: |
|
🎉 Upstream win: all four sherpa-onnx PRs I sent for this feature are merged and released in sherpa-onnx-node@1.13.7:
One more still in review: k2-fsa/sherpa-onnx#3895 adds beam search + hotwords — correctly spells rare biblical names like Melchizedek and Zerubbabel in testing. Will definitely implement these in the STT rework (1.13.6 → 1.13.7 is purely additive) and share the measured improvements after more testing. @vassbo @AlloDel |
|
Hey @vassbo — the rework reads much cleaner, especially detection moving to the renderer and the generic suggestion model. A few things went out along the way and I'd rather ask than undo something deliberate. Granularity — the one I'd most like to pin down. In August you asked for the AI code split into more files, so I broke scripture into ~10 modules; the recent cleanups go the other way (ten quote-match modules into one 1,660-line engine, whisper manager/transcriber/wrapper into one file). No strong preference either way — I'd just like a rule I can apply so I stop churning your files. Is it "one file per cohesive subsystem, split only by ownership"? And does that extend to the remaining scripture modules, or leave those as they are? Vocabulary biasing — LLM error codes — redactSecrets — went out with the same commit. It stripped Nemotron partials — the partial re-decode is gone, so text only leaves the driver when the utterance closes. Those partials emitted real segments, so detection saw words ~1.2s in; now mid-flow speech waits for a pause, up to the 12s VAD cap. Short utterances are fine — it's continuous preaching where it shows. Complexity, or did it misbehave? Can measure before/after if that helps. Voice commands — happy to go LLM-first, but two things I can't see how they land. Today commands need no key and no setup; LLM-only means an API key, or installing Ollama and pulling a model. And tier 2 is gated on Happy to pick up whichever of these you want and leave the rest. |
|
@josephomills I'm still not done. But taking a break.
|
|
@vassbo Thanks, that's helpful. No rush.
|
|
Voice commands: Right, I remember now. But still only accounts for some languages. I like the idea, but would still not be good to maintain. Let's get the LLM to work well with voice commands at least. Maybe another time... |
|
Voice commands: Alright, LLM will work to an extent. My only concern is the latency and dependence on network connection it'll introduce. Will think of how to mitigate that. STT rework: Sure. Just let me know when to push |
|
Also wanted to ask if you'll like to maintain having a choice of STT model for scripture specifically (ie. Nemotron & whisper) as it is now, or you'd want to have one moving forward. I started with the 2 because whisper was multilingual but not a streaming model as Nemotron. But I found Nemotron 3.5 ASR multilingual and that's what my STT rework uses. And the I think the 'interpretation mode' feature I had is a bit pre-mature, needs more work. I can see whisper being useful in non-scripture aspects like on the Audio tab for audio files (where Nemotron won't be ideal). But that'll be an architectural call. |
7 participants
What
AI auto scripture (opt-in, off by default): FreeShow listens to the sermon, transcribes locally with whisper.cpp, detects scripture references — both spoken ("John chapter 3 verse 16", instant local matching) and quoted ("for God so loved the world…", via a bring-your-own-key LLM) — and either projects them automatically or offers one-click suggestion cards in the scripture drawer.
https://github.com/ChurchApps/FreeShow — demo video to follow.
Features
Guarantees (please review these first)
safeStoragefollow-upPlatform matrix
brew install whisper-cppdetection or custom pathTesting
Happy to split, adjust scope, or rework anything — I kept every touched upstream file minimal and matched existing patterns (LTC worklet, ffmpegManager, ACCESS store, Material inputs, i18n) throughout.
Updates since opening
Test suite is now at 141.