Skip to content

Contextual streaming voice, client telemetry and interrupted queue recovery - #612

Merged
witbrock merged 1 commit into
mainfrom
feat/von-speech-709eb2df
Sep 11, 2026
Merged

witbrock merged 1 commit into
mainfrom
feat/von-speech-709eb2df

Conversation

@witbrock

@witbrock witbrock commented Sep 11, 2026 •

Copy link
Copy Markdown
Member

Merge decision: ready — the supported provider path and affected lifecycle/queue paths pass proportionate checks. This publishes the capability; it does not deploy or enable audio models on the DGX.

User outcome

Von gains contextual live dictation and an explicit conversational voice mode across mobile and desktop browsers, building on the recorded-dictation repairs in #601/#602. Browser/device evidence is retained with turns and failed input attempts. Legacy interrupted queue entries can acquire a real server execution owner rather than remain labelled Next up without a runner.

Current decision surface: native Von task #V#task_deliver_contextual_streaming_dictation_and_709eb2df, Deliver contextual streaming dictation and conversational speech on mobile and desktop. Michael explicitly requested this native programme task; existing Jira migration authority is unchanged.

Material changes

  • Authenticated transcription-only WebRTC, bounded visible conversation/focal vocabulary, provisional captions, ordered/deduplicated finals, draft preservation, and recorded fallback. Provider verification established that gpt-live-transcribe requires manual commits: its server-side VAD option is rejected. Browser acoustic pause detection supplies commits and barge-in signals.
  • Start/End voice, ordinary Von queue/agent/presenter ownership, streamed PCM playback, stable utterance submission IDs, scope fences, interruption and explicit reconnection. Audio-only models stay outside chat routing. No production model settings change.
  • Bounded browser/client snapshots in request envelopes, history debug and turn diagnostics; actor/organisation-scoped input-attempt storage and read-back with 30-day retention. No raw audio, transcript or full user-agent retention in diagnostics.
  • Legacy restart reconciliation reads exact existing answers/tool/effect observations, reconstructs missing conversation bindings through the current authenticated scope, and uses the existing idempotent server handoff. Unknown effects remain unknown. Server-owned Next up and client-owned Ready to resume are distinguished.

Evidence

  • Targeted frontend lifecycle, playback, queue integration and speech tests: 55 passed.
  • Backend speech, telemetry, model selection, history and turn-record checks: 105 passed before the final legacy-row case; final queue service/route cohort: 70 passed. Post-rebase affected backend cohort: 44 passed; frontend cohort: 55 passed again.
  • Minimum Python 3.11 grammar check passed across 1,490 files. Frontend static lint has no errors (one pre-existing unused-disable warning).
  • Real OpenAI media calls through production speech routes, using a loopback synthetic actor/model pool and synthetic WAV microphone. Chrome 152 at 1280×900 and 390×844 recognised “Von uses Vontology and Wikidata for research.” intact, submitted voice once, streamed a fixture-supplied reply, and released all microphone tracks on End. Neither composer viewport overflowed.
  • A real synthesis stream delivered its first 8,192 PCM bytes in 1,111 ms; this is a single measurement, not an ordinary-agent latency claim. Provider configuration rejected server VAD and accepted the corrected transcription-only configuration.

Ship boundary

  • Minimum: working configured media protocol, preserved drafts/scope, one canonical submission per utterance, prompt interruption and microphone release, scoped diagnostic read-back, and an executable idempotent legacy queue successor.
  • Stop ship: candidate-caused lost drafts, cross-conversation writes, duplicate submissions/effects, a retained microphone or obsolete audio after End/interruption, diagnostic scope leakage, or renewed false Next up ownership. The bounded checks found none after repair.
  • Remaining limits: physical Pixel/Android and iOS acoustic, autoplay and lifecycle acceptance has not been performed. Acoustic pause detection is not semantic endpointing; soft speech, long natural pauses and noisy rooms need device evaluation. Existing agent response latency is retained. A partial uncommitted streaming utterance may need repeating after network loss; the recorded mode retains its retry buffer.
  • Non-goals: DGX deployment/activation, changing enabled model pools, blind replay of interrupted external writes, general effect-finality certification, a separate voice agent, or project migration.

@witbrock
witbrock merged commit 36a5320 into main Sep 11, 2026
2 checks passed
@witbrock
witbrock deleted the feat/von-speech-709eb2df branch September 11, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant