Repository navigation
feat(stt): offer Codex as an experimental dictation provider - #26076
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the provider is opt-in, the default stays Notes
[DESIGN-REVIEWED] 5c9aa7a Description read[DESCRIPTION-READ] 577ad760d1e9a1b81c31d089274148a947ae110bc2bca1494a9e59edc374817d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing blocks. The blind reader correctly identified the new "Codex (OpenAI, experimental)" provider option and was willing to pick it. The screenshots show the hidden language picker and the hosted-audio notice. Notes
[UX-REVIEWED] 5c9aa7a Description read[DESCRIPTION-READ] 4d1dcd07730bc8fd26e87afe3c1edb19f7e3922bd0044f0b615d126134978326 That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Opt-in addition with the Notes
What this change shipsInventory (10 items) — 8 justifiedIntent: let people already signed in to Codex use it for dictation and voice-message transcription without setting up another speech backend — ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 5c9aa7a Description read[DESCRIPTION-READ] 9003ad23f73026b303cfbf5c7b76bd8a065ac1d28babe097b4e4f1b2f02b1338 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] 5c9aa7a Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
91fcdae to
f100401
Compare
|
f100401 to
5b0a01e
Compare
|
Sensitive Change Review
|
Sensitive change review
|
5b0a01e to
63b8fcb
Compare
Sensitive change review
|
|
63b8fcb to
3c268b4
Compare
Sensitive change review
|
718e744 to
4d4755c
Compare
Sensitive change review
|
4d4755c to
0a3d5e8
Compare
Sensitive change review
|
Sensitive change review
|
|
Sensitive change review
|
Sensitive change review
|
Sensitive change review
|
Sensitive change review
|
|
Hold for maintainer review at 34a896e. This PR adds a sandbox mode and a new spawned process, so it is reserved for a human:
In the sandbox files I read, For the maintainer: with AI findings judged non-blocking:
Not read line by line: To clear: a maintainer reviews the sandbox and spawn changes and rules on the sandbox-off behaviour. — crew-pr-reviewer (r147) |
Sensitive change review
|
bolichen97
left a comment
There was a problem hiding this comment.
Requesting changes at 1bc9a6f for one point. CI and the AI lanes pass, Codex stays opt-in (test_codex_is_opt_in_and_preserves_default), and the default provider stays local.
stt/codex_rpc.py adds a regex in product code. This repository asks new product code not to parse text with a regex:
_VERSION_RE = re.compile(r"kiro_crew_dictation/(\d+)\.(\d+)\.(\d+)(?:\s|$)")
and _VERSION_RE.match(str(initialized.get("userAgent", ""))) reads the Codex version from that string. Please parse it with exact token handling instead:
- Take the first whitespace token of
userAgent(str.split()), and require it to start with the literal prefixkiro_crew_dictation/(str.startswith). - Split the rest on
., require exactly three parts, eachisascii()andisdigit(), and convert them withint(). - Treat anything else as no version, as the
matchmiss does today.
A test for a valid value, one with a suffix or a fourth part, and one with the wrong prefix would cover it.
On the sandbox change: require_confinement defaults to False, and only the new Codex dictation spawn sets it, so no existing spawn changes. That is fine as it is.
|
@bolichen97 note: replaced the regex in The new
This head is also a rebase onto current main. Its one conflict, in |
Sensitive change review
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Thanks for this. The Codex wire boundary is careful work: a five-method allowlist, every server request denied, and bounded transcript storage. The review of head 849efb3 found two things to fix before this can merge.
-
No evidence for the new error states (
website/src/i18n/locales/en.manual.json:9480-9486).- What happens: the PR adds seven
stt_codex_*setup-error messages to the Voice page. The UX review on this head says it "cannot evaluate: missing a screenshot of any Codex setup-error state". Both screenshots in the body show only the ready state, and they come from an older build (3c268b4). No test renders an error state. - Smallest fix: attach screenshots taken at this head of Codex with the Voice provider set to it, in at least two states. Show one Settings state, either Codex not installed (
stt_codex_missing) or sandbox unavailable. Show one signed-out error (stt_codex_login_required); the error at record time is fine. Then re-run the UX review.
- What happens: the PR adds seven
-
A second confinement check duplicates one that main already has (
sandbox_wrap.py:518,stt/codex.py:124,stt/codex_rpc.py:62and:153).- What happens:
no_backend_refuses_spawnand the newrequire_confinementflag check the same thing assandbox_spawn.credential_mask_applies(mode)on main. The Codex model provider already relies on that check throughenforce_sandbox_floor. The new predicate equalsnot credential_mask_applies(mode)in every case but one: a gateway that itself runs inside a Kiro Crew sandbox. In that case dictation starts, andwrap_argvtakes the nested passthrough withoutextra_hidden_dirs. So the child does not get the credential mask that the body and thecodex_rpc.pycomment promise. The provider refuses that case. For this one caller, the only reachable branch ofrequire_confinementis a case the preflight check already refuses. - Smallest fix: delete
no_backend_refuses_spawn, therequire_confinementparameter and its branches insandbox_wrap.pyandsandbox_spawn.py, and thesandbox.pyexport. Then callcredential_mask_applies(...)(orenforce_sandbox_floor(ACP_BACKEND_CODEX, ...)) incodex.sandbox_refuses()and before the spawn inCodexRpc.start. After this change the PR touches no sandbox file. A maintainer still has to decide whetheragent.sandbox=offshould refuse or run atstandard. This fix works with either answer.
- What happens:
The earlier change request on 1bc9a6f asked to replace the regex in codex_version. That looks done at this head, but the request stays open until its reviewer reviews again.
Optional:
- The only test that reaches the Codex spawn stubs
codex_credential_maskand checks onlywrapped[:5]. You can remove the mask,--disable shell_toolorstrip_python_env=Trueand every test still passes. Add a test that uses the real mask and asserts the full argv andstrip_python_env. - The dictation child gets no private
CODEX_SQLITE_HOME, and_ENV_NAMESdrops one that an operator sets. The Codex provider gives each runtime its own because concurrent app-servers fail withdatabase is locked. We have not reproduced this, so treat it as a likely risk. - Open PR #29221 changes which
codexbinary the provider runs and adds its own bundled-Codex lookup. Name the merge order in the body so dictation and the provider keep running the same binary.
Add Codex (experimental) to the speech-to-text providers. It uses an existing Codex ChatGPT login through Codex's app-server realtime transport, for live dictation and recorded clips. The local provider stays the default. Only the user's input transcript becomes text. The adapter sends five RPC methods, sets client-managed handoffs and declines every server request, so dictated words never start a Codex task, and assistant audio is discarded. The codex binary comes from the provider-CLI trust walk, never a bare PATH lookup, and the app-server child runs inside the agent sandbox at the operator's tier or standard, whichever is stricter. Sessions finish with session.close and read late transcript text until session.closed.
|
Answers to the review of 849efb3, both fixed in 5c9aa7a. 1. note: screenshots of two error states are now in the body, taken in a Dev Fleet pod built at 066b2ed400 with the provider set to Codex. Neither state is stubbed.
The UX review runs again on the new head. 2. note: the second confinement check is removed, and the PR now touches no sandbox file.
Optional items.
|
|
Thanks for the quick turnaround. I re-reviewed head 5c9aa7a. I found nothing blocking, but this PR changes a design doc ( What my earlier review of 849efb3 asked for:
What I verified on this head: the child starts only through Still open for maintainers:
Optional:
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved at a maintainer's direction, on head 5c9aa7a.
Every item from the earlier reviews is resolved on this head: the error-state screenshots are attached and the UX review passes, the duplicate confinement check is gone (both entry points use sandbox.credential_mask_applies), and the version check now parses the first userAgent token with an exact prefix and three digit groups instead of a regex. The maintainer signed off on the feature-map README change and accepted the agent.sandbox=off difference as intended: dictation still runs Codex confined at standard, which is stricter than the model provider, not looser.
Optional follow-ups, not merge conditions: map a non-CodexError start failure in stt/codex.py:286 to stt_codex_failed with a logged cause, guard the voice-codex install in packaging/build-desktop.sh:583 like the recogniser step below it, and pin the transcript caps and event-queue limits with at-limit tests.
Problem / Motivation
Goal: Offer Codex as an optional, experimental speech-to-text provider that uses an existing Codex ChatGPT login on a voice-capable Codex install.
People who already sign in to Codex can't use it for dictation or recorded voice messages today. They have to set up a separate speech backend.
Why it matters
Codex 0.163.0 and later recognises speech for ChatGPT accounts that have voice through its experimental app-server realtime transport. Offering it as a provider adds dictation for those users without a new key, and the selected chat agent stays whatever it was.
Not a goal
What changed (motivation → approach → change)
Voice settings now offer Codex (OpenAI, experimental). The page says that audio goes to OpenAI, explains each setup problem, and hides the language picker because Codex detects the language itself.
pages/settings/SttSettings.tsx,lib/sttProviders.ts,website/src/i18nconfig/sections_stt.py,docs/configuration.mdcodexis a valid stored provider. The default stayslocal.stt/codex.py,stt/codex_rpc.py,stt/codex_media.py,dashboard/stt_adapted_stream.py,acp/client_spawn_env.py,agent_sdk/__init__.py,agent_sdk/backends_launch.pykiro_crew.agent_sdk.transcribe.py,transcribe_local.pytest/test_codex_stt*.py,test/fixtures/codex_voice.json,setup.cfg,extras.pyvoice-codexmedia extra, registered with its install hint.config-baseline.jsoncodexprovider value.packaging/build-desktop.sh,website/electron/test/packaging.test.jsOnly the user's input transcript becomes text. The adapter sends five RPC methods and nothing else. It sets
clientManagedHandoffs, declines every server request, and refuses Codex builds older than 0.163.0, so dictated words never start a Codex task.Dictation runs the same
codexthe Codex model provider runs. A new helper next to_resolve_codex_acp_bin()inacp/client_spawn_env.pyreturns the command the codex-acp adapter would start:CODEX_PATHwhen it is set, made absolute and checked, otherwise the@openai/codexscript the adapter depends on, found from the adapter's real install path with the Node it runs under. There is no second search ladder and no provenance check, because Crew runs as the user, the provider is opt-in, and this is the provider's own trust model. An adapter that resolves to a launcher shim with no locatable Codex CLI reportsstt_codex_cli_unresolvedand namesCODEX_PATH; nothing installed reportsstt_codex_missing.The app-server child runs inside the agent sandbox through
sandboxed_spawn_argv_async, through the samewrap_argvlauncher and with the same credential mask the provider uses for codex-acp. The decision to run it is the provider's own predicate,sandbox.credential_mask_applies(mode), whichenforce_sandbox_flooralso uses. Settings reads it before any install step, andCodexRpc.startreads it again before the spawn. Where it is False (no sandbox backend, or a gateway already inside a Crew sandbox, whose nested passthrough would drop the mask), dictation fails withstt_codex_sandbox_unavailable. This PR changes no sandbox file. One difference from the provider remains: withagent.sandbox=offthe provider refuses to start, while dictation still runs Codex confined atstandard(it takes the operator's tier when that isccorstrict). The child gets no session-bus variables. Windows has no agent sandbox, so the Windows desktop bundle does not ship thevoice-codexextra.If the Codex child exits, sends a malformed or oversized line, fails a write, does not answer a request, or the connection does not come up in time, the user gets the retryable
stt_codex_failed.CodexRpc.write()classifies every write failure,call()andstart()classify their timeouts, and the reader queues a privatekiro_crew/transport_failedevent that the child cannot forge. Only a real Codex error maps tostt_codex_voice_unavailable. The stream sends a Codex session's error code to the client only when it is one of the sevenstt_codex_*codes instt.codex.SESSION_ERROR_CODES, whichrun_codex_sessionpasses to_drive_helper_sessionasrelayed_codes. Any other code is sent asstt_session_failed. A registered provider keeps main'sRELAYED_SESSION_ERROR_CODESlist.To end a session the adapter sends
session.closeand keeps reading late transcript text untilsession.closed. The owning speech spec explains the wire contract and how to mock it.Backwards compatibility
Compatible: every existing provider value and the
localdefault stay valid, andvoice-codexis optional. Existing decoder callers keep their duration bound.The new
extra_hidden_dirson the Codex dictation spawn hides the same credential paths from the Codex child that the Codex model provider hides from codex-acp. The spawn is new in this PR and has one caller:Reader: src/kiro_crew/stt/codex_rpc.py:CodexRpc.start -- chat -- the dictation child keeps
~/.codexvisible, so the ChatGPT login still works (test/test_codex_stt.py::test_rpc_login_env_and_process_cleanup)Removes nothing. The language picker is hidden only while Codex is selected.
Tests
test/test_codex_media.pycovers the WebRTC peer, track frames and SDP answer with stand-inaiortcandavmodules, and the paced PCM buffer.test/test_codex_stt.pyandtest/test_codex_stt_stream.pymock Codex at the process, sandbox and peer boundaries. They need no Codex binary, WebRTC package, microphone or login.session.closehandshake with late transcript text, recorded audio, and process cleanup.CODEX_PATH(relative, non-executable, script), the adapter's bundled Codex CLI, a shim-resolved adapter and nothing installed. Spawn tests cover the sandbox tier the child gets, the refusal before any spawn where the credential mask would not apply (no backend, or inside a Crew sandbox), and removal of the launcher file. One spawn test uses the provider's real credential mask and checks the full argv andstrip_python_env.test/test_spawn_audit.pynow sees the Codex spawn routed through the sandbox, so it no longer needs a benign-spawn entry.website/src/testcovers the label, the setup errors and the hosted-audio notice in the provider-vocabulary, settings-surface and retention-note tests.mainworktree on this host. They are a launch-copy steering test, an auto-improvement private-store test and a CI-kernel JUnit test, all caused by this host's scratch paths and Btrfs lock identities. Semgrep on the diff and the flake-pattern ratchet pass.Manual verification
not installed(stt_codex_missing) while no codex-acp adapter was reachable. After the adapter was installed into the pod's worktree, clicking the composer microphone in Chromium with a fake audio device made/api/ws/sttanswerstt_codex_login_required, because this host's Codex login is not a ChatGPT login. Both states are real. The screenshots below come from this run./api/ws/sttstarted the sandboxed app-server, which used Codex 0.165.0 on a Bedrock-backed login. The dashboard receivedstt_codex_login_required("Sign in to Codex with ChatGPT using codex login."), and the launcher file and temp directory were removed afterwards.session.closed. It also showed that silence and a user turn don't finalize a session, which is why the client sendssession.closeand waits forsession.closed.Screenshots / video
Taken in a Dev Fleet pod built at 066b2ed400, with the provider set to Codex. Neither state was stubbed.
Settings with Codex not installed (
stt_codex_missing):Dictation started while Codex is not signed in with ChatGPT (
stt_codex_login_required):Taken in a Dev Fleet pod built at 3c268b4. Voice settings with Codex selected and ready, showing the "Codex (OpenAI, experimental)" label and the retention notice:
The provider picker open:
Related Issues
Closes #26075
Merge order with #29221, which changes which
codexthe provider's app-server runs: either PR can merge first. Dictation must keep running the provider's binary, so the PR that merges second makesresolve_codex_cli()follow the provider's lookup. If #29221 merges first, I will do that on this PR.Checklist