Repair contextual dictation and mobile speech flow

docs/design_index.md

@@ -95,6 +95,7 @@ live.
 | Evaluation, benchmarks, or research-sensitive architecture | [agent evaluation and research uptake](engineering/agent_evaluation_and_research_uptake.md) |
 | Minimal imposition, elicitation, or write policy | [minimal-imposition principle](engineering/minimal_imposition_design_principle.md) |
 | Frontend or browser acceptance | [Frontend browser validation](engineering/frontend_browser_user_view_validation.md), at the validation tier justified by the claim |
+| Dictation, audio transcription or mobile speech playback | [Speech I/O](engineering/speech_io.md); current conversational delivery plan remains in its linked Jira issue |
 | Live user-visible behaviour or telemetry diagnosis | Applicable sections of the [real-path replay and telemetry loop](engineering/real_path_server_replay_and_telemetry_loop.md), using the risk tier in `AGENTS.md` |
 | Substantial Jira task definition or pull-request merge boundary | The decision discipline in [`AGENTS.md`](../AGENTS.md#5-capability-slice-planning), the practical [Jira guidance](engineering/operational_engineering_guide.md#53-jira) and [merge-decision procedure](engineering/operational_engineering_guide.md#62-make-new-evidence-change-a-decision), and the [pull-request template](../.github/PULL_REQUEST_TEMPLATE.md) |
 | Practical repository operation | [operational engineering guide](engineering/operational_engineering_guide.md), [authority-alignment scan](engineering/maintaining_global_design_constraints_and_authority_alignment_with_coding_agents.md) |

docs/engineering/speech_io.md

@@ -0,0 +1,111 @@
+# Recorded dictation and speech playback
+
+- **Kind:** Bounded implementation and operating contract
+- **Reviewed:** 10 September 2026
+- **Owner / review trigger:** Speech workstream; review on audio-provider,
+  authentication, browser capture or presenter changes.
+- **Current decisions:** [JVNAUTOSCI-888](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-888)
+  owns the recorded-dictation repair;
+  [JVNAUTOSCI-812](https://naoinstitute.atlassian.net/browse/JVNAUTOSCI-812)
+  owns the current conversational speech plan. Streaming voice is planned,
+  not provided by the block-dictation implementation.
+
+## User flow
+
+Select **Dictate**, speak, then **Finish dictation**. The transcript is inserted
+at the original selection if the draft is unchanged; otherwise it is appended
+to the edited draft. It remains editable. **Send Prompt** during recording
+finishes transcription first and only sends on success. **Cancel dictation**
+releases the microphone and prevents late permission or transcription results
+from changing the draft. A failed upload retains audio in the active page and
+offers **Retry recording** without another microphone request.
+
+Changing conversation or organisation cancels the capture. Leaving the page
+releases it; backgrounding finishes a recording. Raw audio is not saved by Von.
+Retry audio is lost when the page is closed or the capture is cancelled.
+
+**More actions → Dictation** also offers browser recognition explicitly. Its
+support and contextual biasing vary by browser. Recognition results are treated
+as complete, revisable snapshots so replayed final results do not accumulate
+duplicates. Interim text is shown in the status; only final text enters the draft.
+
+## Runtime configuration
+
+The server needs its existing configured OpenAI key and an enabled audio model
+in the authenticated user's or organisation's existing model pool. Supported
+transcription models, in selection order, are `gpt-transcribe`,
+`gpt-4o-transcribe` and `gpt-4o-mini-transcribe`. Optional
+`VON_TRANSCRIPTION_MODEL` selects exactly one of these; it must still be enabled
+in the scoped model pool. Add it as an additional model, not the primary
+conversational model. Audio models are excluded from automatic chat fallback
+and selection candidates. This does not change the main conversational model.
+No credentials or enabled-model settings are changed by this feature.
+
+`GET /api/speech/capabilities` reports actor-specific availability. The composer
+refreshes it on initialisation, focus and preference changes. A configured key
+and enabled model do not prove provider entitlement or live recognition quality;
+provider rejection is a visible retryable failure.
+
+`POST /api/speech/transcribe` accepts multipart `audio`, `context`, JSON
+`vocabulary` and optional BCP 47 `language`. It derives identity and organisation
+from existing trusted context, including the owned window-session header. Legacy
+identity headers alone cannot authorise an external audio request. No request
+field selects another actor's history or credentials.
+
+Audio is bounded to 24 MB, below the provider's 25 MB limit. Multipart parsing
+also has a request bound. The recorder negotiates MP4 or WebM and stops near the
+size limit; formats are validated again on the server. The provider HTTP request
+has a 120-second transport bound to limit occupied workers, with no automatic
+SDK retry. A failed request can be retried from retained browser audio. This is
+an initial operational bound, not a speech-turn duration or a latency claim;
+revisit it against actual successful recording/transcription durations.
+
+Context is the recent visible conversation and current draft (up to 6,000
+characters), plus up to 80 visible literal concept names. The service does not
+scan Vontology or fetch private conversations. Context/keywords are recognition
+hints, not required output or tool instructions. Transcription uses one audio
+model call and no semantic rewriting pass. `gpt-transcribe` receives plural
+`languages`/`keywords`; the GPT-4o audio models receive the compatible prompt
+and singular language parameter.
+
+## Playback repair
+
+Starting dictation stops current narration. Browser speech continues to use
+the existing presenter/spoken channel and playback telemetry. The long-utterance
+pause/resume workaround is restricted to desktop Chromium; it must not run on
+Android or iOS/iPadOS. Mobile playback is invoked synchronously with the user
+gesture, and playback errors offer a visible retry. Automatic narration can
+still be blocked by device autoplay policy; tap **Speak** in that case.
+
+## Evidence boundary
+
+Baseline `59cbeb2f` had browser-only dictation, no vocabulary context,
+append-only final handling and console-only input errors. The DGX conversation **Von Task Smoke Test**, session
+`9b49b86d-26f1-49ac-9027-58b8864ca7e5`, supplies exact source evidence:
+user entries 2 and 4 contain repeated progressively longer prefixes; entry 8
+reports correct desktop behaviour; entry 10 begins by replaying entry 8 before
+new dictation. The native issue is
+`#V#task_agent_8fbedbe81f4951431e8ff417abf402f8`.
+The accumulation bug reproduces this class of output, but raw audio and browser
+recognition events were not retained, so independent acoustic hallucination or
+physical-device behaviour cannot be reconstructed from transcript text alone.
+
+Targeted tests cover MP4/WebM forwarding, contextual language/vocabulary,
+authenticated session binding, model denial, invalid uploads and safe provider
+errors; browser lifecycle tests cover retained-audio retry, permission/cancel
+races, conversation fences, revised recognition snapshots and preservation of
+edits. Speech tests cover mobile gesture timing and exclusion of pause/resume.
+
+On 10 September 2026, Chrome at desktop and 390 × 844 viewports exercised the
+actual `chat_tab.html` composer and `dictation.js` controller at
+`http://127.0.0.1:5018/` using a temporary simulated microphone/provider fixture.
+Failure → edit draft → retry produced one transcript while preserving the edit;
+cancel preserved the draft; the mobile composer had no horizontal overflow.
+This is layout/lifecycle evidence, not acoustic, physical-device, provider-quality
+or live DGX acceptance. The DGX transcript was read through its own canonical
+history service after temporarily reconnecting this Mac's existing Tailscale
+profile. Its observed pre-repair runtime was `12ae94c0`. Runtime deployment and
+physical iOS/Android behaviour require separate verification receipts.
+
+Provider reference: [OpenAI file transcription](https://developers.openai.com/api/docs/guides/speech-to-text).
+Browser reference: [MDN SpeechRecognition](https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition).

src/backend/integrations/internal_mcp/orchestrator.py

@@ -14310,7 +14310,9 @@ def _stage_model_candidates(
                 str(entry.get("model") or "").strip() or None
             )
             host = str(entry.get("host") or "").strip() or None
-            if not provider or not model:
+            from ...languagemodels.model_defaults import is_transcription_model
+
+            if not provider or not model or is_transcription_model(provider, model):
                 continue
             model_parameters = (
                 normalise_model_parameters_for_storage(

src/backend/languagemodels/model_defaults.py

@@ -8,6 +8,16 @@
 
 import os
 
+# Audio/transcriptions models do not implement the chat/tool interface. They
+# may be authorised in the shared model pool without becoming chat fallbacks.
+TRANSCRIPTION_MODELS = ("gpt-transcribe", "gpt-4o-transcribe", "gpt-4o-mini-transcribe")
+
+
+def is_transcription_model(provider: object, model: object) -> bool:
+    return str(provider or "").strip().lower() == "openai" and str(
+        model or ""
+    ).strip().removeprefix("openai:") in TRANSCRIPTION_MODELS
+
 # ---------------------------------------------------------------------------
 # Ollama
 # ---------------------------------------------------------------------------

src/backend/server/routes/speech_routes.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import json
 from datetime import datetime, timezone
 from typing import Any, Dict, Optional
 
@@ -10,10 +11,98 @@
     update_llm_debug_data_for_request_id,
 )
 
-
 speech_bp = Blueprint("speech_bp", __name__, url_prefix="/api/speech")
 
 
+def _speech_actor():
+    from ...security.access_control import (
+        get_effective_organisation_concept_id,
+        get_effective_user_concept_id_with_source,
+    )
+
+    actor, source = get_effective_user_concept_id_with_source()
+    if not actor or source not in {
+        "authenticated_session",
+        "authenticated_session_derived",
+        "trusted_in_process_actor",
+    }:
+        return None, None
+    return actor, get_effective_organisation_concept_id()
+
+
+@speech_bp.route("/capabilities", methods=["GET"])
+def speech_capabilities():
+    from ...services.speech_transcription_service import (
+        SpeechUnavailable,
+        transcription_capability,
+    )
+
+    actor, organisation = _speech_actor()
+    if not actor:
+        return jsonify(error="authentication_required"), 401
+    try:
+        data = transcription_capability(actor, organisation)
+    except SpeechUnavailable as exc:
+        data = {"available": False, "reason": str(exc)}
+    except Exception:
+        data = {
+            "available": False,
+            "reason": "Speech configuration is temporarily unavailable.",
+        }
+    response = jsonify(transcription=data)
+    response.headers["Cache-Control"] = "private, no-store"
+    return response
+
+
+@speech_bp.route("/transcribe", methods=["POST"])
+def transcribe_speech():
+    from ...languagemodels.llm_interface import ModelExecutionEligibilityError
+    from ...services.speech_transcription_service import (
+        MAX_AUDIO_BYTES,
+        SpeechUnavailable,
+        transcribe_audio,
+    )
+
+    actor, organisation = _speech_actor()
+    if not actor:
+        return jsonify(error="authentication_required"), 401
+    # Bound multipart parsing as well as the file read; audio is never retained.
+    request.max_content_length = MAX_AUDIO_BYTES + 65536
+    uploaded = request.files.get("audio")
+    if uploaded is None:
+        return jsonify(error="missing_audio", message="No recording was received."), 400
+    try:
+        vocabulary = json.loads(request.form.get("vocabulary", "[]"))
+        result = transcribe_audio(
+            actor=actor,
+            organisation=organisation,
+            audio=uploaded.read(MAX_AUDIO_BYTES + 1),
+            mime_type=uploaded.mimetype or "",
+            context=request.form.get("context", ""),
+            vocabulary=vocabulary,
+            language=request.form.get("language", ""),
+        )
+    except (ValueError, json.JSONDecodeError) as exc:
+        return jsonify(error="invalid_audio_request", message=str(exc)), 400
+    except (SpeechUnavailable, ModelExecutionEligibilityError) as exc:
+        return jsonify(error="speech_unavailable", message=str(exc)), 503
+    except Exception as exc:
+        # Provider exceptions can contain credentials, URLs and submitted text.
+        current_app.logger.warning(
+            "Speech transcription failed (%s)", type(exc).__name__
+        )
+        return (
+            jsonify(
+                error="transcription_failed",
+                message="Transcription failed. Your recording is available to retry.",
+            ),
+            502,
+        )
+    response = jsonify(success=True, **result)
+    response.headers["Cache-Control"] = "private, no-store"
+    return response
+
+
 def _coerce_float(value: Any) -> Optional[float]:
     if isinstance(value, bool):
         return None
@@ -70,8 +159,10 @@ def _normalise_timestamp(value: Any) -> Optional[str]:
             return value
     if isinstance(value, (int, float)):
         try:
-            return datetime.fromtimestamp(float(value) / 1000.0, tz=timezone.utc).isoformat().replace(
-                "+00:00", "Z"
+            return (
+                datetime.fromtimestamp(float(value) / 1000.0, tz=timezone.utc)
+                .isoformat()
+                .replace("+00:00", "Z")
             )
         except Exception:
             return None
@@ -87,7 +178,9 @@ def _sanitise_playback(payload: Dict[str, Any]) -> Dict[str, Any]:
     settings = settings if isinstance(settings, dict) else {}
 
     return {
-        "request_id": _coerce_text(payload.get("request_id") or playback.get("request_id")),
+        "request_id": _coerce_text(
+            payload.get("request_id") or playback.get("request_id")
+        ),
         "turn_id": _coerce_text(payload.get("turn_id")),
         "conversation_id": _coerce_text(payload.get("conversation_id")),
         "started_at": _normalise_timestamp(playback.get("started_at")),

src/backend/services/model_registry_service.py

@@ -2,18 +2,19 @@
 
 from __future__ import annotations
 
-from datetime import datetime, timezone
 import hashlib
 import json
 import logging
 import os
-from pathlib import Path
 import tempfile
 import threading
 import time
+from datetime import datetime, timezone
+from pathlib import Path
 from typing import Any, Mapping, Optional, Sequence
 from urllib.parse import parse_qsl, unquote, urlsplit
 
+from ..languagemodels.model_defaults import is_transcription_model
 from .settings_service import resolve_enabled_llm_settings, resolve_llm_setting
 
 logger = logging.getLogger(__name__)
@@ -1280,7 +1281,7 @@ def _build_registry_from_settings() -> Mapping[str, Any]:
     user_concept_id = None
     org_concept_id = None
     try:
-        from flask import session, has_request_context
+        from flask import has_request_context, session
 
         if has_request_context():
             user_concept_id = session.get("user_concept_id")
@@ -1298,6 +1299,8 @@ def _build_registry_from_settings() -> Mapping[str, Any]:
             continue
         provider = entry.get("provider")
         model = entry.get("model")
+        if is_transcription_model(provider, model):
+            continue
         locality = "external"
         if isinstance(provider, str) and provider.lower() == "ollama":
             locality = "local"