Repository navigation
feat(voice): optionally tell the agent a message was dictated - #32016
Conversation
bb899d9 to
50ab995
Compare
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the setting is opt-in and off by default, and the description explains why the note goes in the message text instead of a request flag. Notes
[DESIGN-REVIEWED] fa16558 Description read[DESCRIPTION-READ] 2cd9aee2f0e372c018fb50ca779656c9725d8cc9abff630455ebd55e82ff06e5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The new setting is off by default, so nobody's messages change unless they turn it on. Each part of the change has a real problem to solve, code that uses it, and no existing mechanism that already does it. What this change shipsInventory (7 items) — 7 justifiedIntent: a person who dictates wants the agent to read speech-to-text errors generously instead of literally. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] fa16558 Description read[DESCRIPTION-READ] e7d325f8ab49b2c2960cc65a098db77fb90bb08c8bfbf74146ece7e465fdc8c8 That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing blocks. The blind reader understood the new "Tell the agent this was dictated" toggle and said they would use it. It matches the toggles beside it under Fine-tuning. Both of its states (setting on, note in a sent message) appear in screenshots. Notes
[UX-REVIEWED] fa16558 Description read[DESCRIPTION-READ] fd473e142be6e88b0b7cb8771b4eb0a26850eca97f297f70bd96962d4c754266 That sha256 names everything this verdict read: the PR description, plus the 5 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
50ab995 to
93eeb97
Compare
93eeb97 to
3ddbb5c
Compare
|
Rebased: main moved in config-baseline.json, docs/feature-map/README.md, docs/system-specs/modules/config.md, src/kiro_crew/config/section_builders.py, src/kiro_crew/docs/configuration.md, the settings registry files, all 12 locale catalogs, CrewChatWindow.tsx, crewWindowStore.ts and CrewChatWindow.test.tsx since this head went green (fe2d88c -> fcfba68); CI re-running. |
|
3ddbb5c to
04cec85
Compare
|
@bolichen97 thanks for the review. issue: Fixed locally in 72cd65bc28: the line is now note (non-blocking): the App chunk over its budget I built both trees with |
8901908 to
2c9354a
Compare
|
@bolichen97 the |
2c9354a to
f5761cd
Compare
|
Human judgment recorded@Pearcekieser recorded a false-positive release (class
This decision applies only to this commit. A new push requires a new judgment. |
Add stt.dictation_note, an off-by-default checkbox under Settings, Voice,
Fine-tuning ("Tell the agent this was dictated"). When it is on, a message
that holds dictated words ends with one line telling the agent it may
contain transcription errors. Typed-only messages never get it.
The Voice atom records which session's composer or parked draft received
spoken words. Each send path (main chat send and steer, pane send and
steer) takes that record when it consumes the composer, and every path
that hands the text back (a refused or failed send, a returned steer, a
cancelled queued card) puts it back. Emptying the composer clears it, and
a discarded dictation takes back the mark it set.
The note is part of the message text, like the gateway's own provenance
lines, so it reaches the agent on every later path (queue, steer,
regenerate, resume) and the transcript shows what the agent was told.
f5761cd to
fa16558
Compare
|
Thanks for this. The review on this head found no blocking problems. Two things still need a maintainer before this can merge:
What the review checked on fa16558: the setting Done well: the code never guesses from the words whether a message was dictated, so typed text that happens to end with the note line comes back unchanged. A test checks exactly this. Optional:
|
bolichen97
left a comment
There was a problem hiding this comment.
Approve on fa16558. I read the whole diff (2844 lines). Both points from our change request are fixed. This fixes #32015.
- Regex:
withDictationNotenow usestext.trimEnd(). I searched the diff and found noreplace(/,new RegExp,re.compileor other regex literal in product code. - Bundle: Bundle Size Gate passes on this head. The App chunk is 2,296.43 kB, within its budget.
- The setting
stt.dictation_noteis off by default. The PUT accepts only a real boolean, the same asdictation_panelnext to it. - Provenance is per session and per composer scope (main page, pane, crew-window draft and rewind box). It is never guessed from the text. Every hand-back path (refused send, steer, queued-card cancel, delivery-unconfirmed restore) puts the record back and restores the note-free text.
- The note is never added to a command (
isCommandText), andwithDictationNoteis idempotent. - All 97 checks are green, and nobody else has an open change request.
Problem / Motivation
Goal: Let someone who dictates turn on a setting that tells the agent a message was dictated, so the agent reads transcription errors charitably.
A dictated message reaches the agent as plain text, indistinguishable from typing. Speech-to-text produces misheard words, odd punctuation and words in the wrong place, and the agent takes them literally: it acts on the wrong word or asks about an obvious transcription error.
Why it matters
People who dictate most of their messages hit this on nearly every turn. The agent already knows how to read a sloppy transcript generously; it just has no way to know the text was spoken.
Not a goal
What changed (motivation → approach → change)
A new checkbox, "Tell the agent this was dictated", sits under Settings → Voice → Fine-tuning. It is stored as
stt.dictation_note, off by default. When it is on, a message that holds words the composer's dictation wrote ends with exactly one line,[This message was dictated. It may contain transcription errors, odd typos, misplaced words, or punctuation.]. A typed-only message and a slash command never get it.The note is part of the message text, like the gateway's own provenance lines (
[sent by session …]). A request flag the gateway appends for the agent only would have to ride the queue, a steer, the sub-agent hold, regenerate and resume, or lose the note. In the text it reaches the agent on every path.The voice hook records that a composer holds dictated words whenever it writes them (streaming, batch, an off-screen delivery, a retried recording); the words themselves are never inspected. The record is keyed by session and composer, because the main chat, a split pane and a crew window each keep their own draft of a session. It lives in session storage (
mc-dictated-drafts, at most 200 entries), so it survives a reload as long as the shortest-lived draft does. It never moves to a new chat that takes over text.Each send and steer takes the record and decides the note. The command check runs on the finished outgoing text, after any quote, with the gateway's rule (starts with
/after leading whitespace). A send that hands its text back (refused, failed, a returned steer, a cancelled queued card, the "not confirmed" restore strip) puts the record back from a flag it carries. Emptying the composer, a command or a discard clears it.The rule is per composer, not per word: a composer counts as dictated from its first dictated word until it is sent or emptied, even if an edit deletes every dictated word. Per-span tracking needs a text model the composer lacks, and a hand-fixed message still started as speech; the module doc says so.
Where it lives:
src/kiro_crew/config/sections_stt.py,section_builders.py, and the STT config GET/PUT insrc/kiro_crew/dashboard/handlers/core_stt_settings.py; regeneratedconfig-baseline.json, thesrc/kiro_crew/docs/registry and UI index, and the command palette registry, plusconfiguration.md. The checkbox is inwebsite/src/pages/settings/SttSettings.tsx, with its label and description in every catalog underwebsite/src/i18n/locales/.website/src/chat-core/composer/dictationProvenance.ts, with the note text indictationNote.prompt.ts; marked fromuseComposerVoice.ts, appended inoutgoingTurn.ts.website/src/pages/chat/page/composerSend.ts,busyTurnControls.tsandturnActions.tsfor the main chat, withcomposerDrafts.tsclearing on an emptied composer and never moving it to a new chat;website/src/components/useChatPaneSend.tsandChatPane.tsxfor panes;website/src/pages/chat/crew-window/CrewChatWindow.tsxandcrewWindowStore.tsfor crew windows;website/src/hooks/useQueuedMessageActions.tsfor a cancelled queued card.Known limits, left for a follow-up:
Backwards compatibility
Compatible: the setting is new and off by default, so no message changes until someone turns it on.
Removes nothing: the PR adds a setting and a suffix; no key, path, validator or record is removed or narrowed.
Tests
dictationProvenance.test.tspins the record's rules (per session and composer, taken by a send, cleared by an empty composer, kept across a reload) and the note (exact text, once, at the end, never on a command), and that the setting is read live from the shared STT config.outgoingTurn.test.tsrows put the note at the very end of both texts, after session links and around a quote or expanded paste, on send and steer, without changingtypedOnly. A command gets no note, a quoted message whose own text starts with a slash still gets one, and text that already ends with the note gets no second one.useComposerVoiceDictationNote.test.tsxdrives the voice hook: batch, streaming and off-screen deliveries mark the right session and composer, a slot switch keeps the mark where it was, and a discard or a hypothesis that retracts to nothing removes only the mark it set.ChatPage.dictationNote.test.tsxcovers the main composer: one note when dictated and on, none when typed or off, none on the next typed message, a refused send and a refused steer handing the words back and adding the note once on retry, typed text ending with the note line coming back whole, emptying the composer, a command sent from a dictated composer, a composer with no chat yet dropping its record once it shows a chat, a slot switch, and a new chat that takes over text entered while it was created, which starts unmarked while a dictated draft left behind keeps its note.ChatPane.dictationNote.test.tsxcovers the pane: send, steer, a steer demoted to the queue stashing its provenance, rebinding to another member, and words dictated into the main chat that do not mark a message typed in the pane for the same session.ChatPane.slashIntercept.test.tsxadds a/sidethat clears the mark.CrewChatWindow.test.tsxcovers a crew window: a dictated send ends with the note, a refused one comes back without it and the retry carries it once, and the next typed message carries nothing.crewWindowTranscript.test.tschecks that an off-screen delivery, including one into a rewind box, names the draft box it marks.useQueuedMessageActions.test.tsxrestores provenance from a stash hit and never infers it from a card's text.SttSettings.surface.test.tsxchecks the toggle lives under Fine-tuning, is off, and saves{ dictation_note: true }.test/test_stt_dictation_note.pycovers the default, the loader's bool-only rule, and the PUT/GET round trip.Determinism: every new test is synchronous or awaits a rendered state with
waitFor; none sleeps, uses timers or depends on order. The new and touched vitest files (250 tests) passed 3 repeated runs and a--sequence.shufflerun;test_flake_pattern_ratchet.pypasses.Manual verification
A throwaway pod ran this branch's built dashboard against its own gateway, driven by Playwright in headless Chromium. That run was on an earlier version of the branch, before it was rebased onto current main and before the per-composer scope, the crew window, reload persistence and the command check were added. Those later changes are covered by the unit tests above, which ran on the final head. The microphone was a synthetic Web Audio stream with a recorder stub, because headless Chromium has no audio input. The recogniser was faked: batch transcription answered a fixed transcript, and the streaming socket sent scripted
ready,partial,finalandendpointframes. Only the STT availability fields of the config read were overridden, because the pod has no speech model; thedictation_notevalue, settings writes, chat sends and the stored transcript were real. Results:Local review: two model reviewers and a verifier ran four rounds on this branch. The fourth answered the CI findings on the create carry, and the fix for them was written in this session, not by a separate model-pinned repair agent. The GPT reviewer ran on
gpt-5.6-solbecause the profile'sgpt-6.1-solis not served on this account, so that lane ran at lower fidelity than the profile asks for.Screenshots / video
The checkbox under Fine-tuning, on:
A dictated message as sent and stored, with the note on its own line; the earlier message above it was sent with the setting off:
Related Issues
Fixes #32015
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)