Skip to content

feat(stt): play a sound when dictation stops on its own - #17923

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
amadsalmon:feat/dictation-stop-sound
Oct 10, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
amadsalmon:feat/dictation-stop-sound

Conversation

@amadsalmon

@amadsalmon amadsalmon commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Problem / Motivation

Goal: A dictation that stops before the user ends it plays a short sound, so someone not looking at the screen knows to stop talking.

When dictation stops on its own (the stream fails, the 5-minute cap is reached, the connection drops), the composer clears the Listening state and shows an error. People often dictate while reading from another screen and never see that. They keep talking, and everything after the stop is lost.

Why it matters

The failure is silent exactly when the user is not watching the screen, which is a common way to dictate. The first they hear of it is after minutes of lost speech.

Not a goal

What changed (motivation → approach → change)

Area Change
hooks/useStreamingStt.ts raises the dictation sound when capture ends without the user
hooks/notificationEvent.ts, hooks/useNotificationSound.ts the new DICTATION_STOPPED_KIND and its sound category
pages/settings/NotificationsPanel.tsx the Interrupted dictation row
i18n/locales/en.manual.json and the 11 translations the row's label and description in 12 languages; i18n/locales/en-XA.json regenerated
components/commandPalette/settingsManual.ts, settingsRegistry.gen.ts, src/kiro_crew/docs/settings-registry.generated.json the row in settings search, regenerated with npm run gen:settings
docs/system-specs/modules/app-notifications.md the new kind

The dashboard already has a sound system with one setting per kind of event, and two kinds the frontend raises itself (turn, approval). This adds a third, dictation, raised by useStreamingStt when capture ends while the user is still dictating: an error frame or a socket close during capture, or the 60-second readiness buffer filling up. A stop, a cancel, an unmount and a superseded session stay silent, and the sound plays once per stop.

Like turn, the kind is sound-only: no feed entry, toast or badge. It gets its own Interrupted dictation row in Settings > Notifications > Per-category sounds. With no override it plays the all-categories sound, like every other row except the existing approval one, so "Use default" means the same thing on it as on its neighbours; it can be given its own sound or silenced, and all = none silences it too. Its description says it plays a sound, so a settings-search hit explains itself.

Backwards compatibility

Compatible: a new sound category and settings row. Stored settings without a dictation key resolve to the built-in default, and every existing category resolves as before.

Removes nothing: no setting, kind, key or reader goes away.

Tests

  • useStreamingStt.stopBeforeReady.test.tsx, new block: the sound fires once for an error frame followed by a close, a dropped connection, a socket error followed by a close, and a full readiness buffer; it does not fire after a user stop (even when the drain then fails) or on cancel.
  • useNotificationSound.test.ts: dictation resolves to the all-categories sound with no override, honours its own override and none, and is silenced by all = none.
  • NotificationsPanel.test.tsx: with the all-categories sound set to Blip, the new row reads "Use default" and its Test button plays Blip. The approval row's position-based lookup is updated for the inserted row, and the two override checks read their own row instead of matching "Ding" anywhere on the page.

Four of the new hook tests fail with the hook change reverted. The related frontend set passes, plus tsc, eslint, npm run i18n:check and every locale's src/i18n/style/ register tests with I18N_BASE_REF set, so the changed-value checks run on the new strings.

Manual verification

Ran the branch's own build on an isolated gateway (fresh data home, minimal seed) and opened Settings > Notifications > Per-category sounds in light and dark mode. The Interrupted dictation row renders between Conversation handoffs and Proactive agent messages, with its info tooltip and Test button, and reads "Use default" like its neighbours.

Screenshots / video

Light:

Settings, Per-category sounds, light: Interrupted dictation row

Dark:

Settings, Per-category sounds, dark: Interrupted dictation row

recorded from the branch head · isolated gateway, minimal seed · real server, no fixtures

Related Issues

Part of #17842

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@amadsalmon
amadsalmon requested a review from a team as a code owner October 7, 2026 23:47
@amadsalmon
amadsalmon requested a review from bolichen97 October 7, 2026 23:47
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — 🟡 CONCERNS

UX-level review of 6475e95f264c80f64a11a1e738fc9b49db4cbd3d via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

No first-time reader has seen the new "Interrupted dictation" row; this lane can't judge whether a newcomer understands it.

Watch

  • Description copy not shown. The row's info tooltip and its settings-search entry both carry "Plays a sound when dictation stops before you end it". Neither screenshot shows that text, so its wording and wrapping haven't been checked in the rendered UI. Severity is low: the copy is only seen when someone hovers the icon or searches. To close it, attach a screenshot of the tooltip open, or of a settings-search hit for "dictation".
  • Row placement. In both screenshots (attachment-01 dark, attachment-02 light), the row sits between "Conversation handoffs" and "Proactive agent messages", reads "Use default", and has a Test button like its neighbours. Both themes render cleanly. The verdict stops at CONCERNS only because no blind reader has read the row in this fork lane. A maintainer can get that read by pushing the branch to this repository.

[UX-REVIEWED] 6475e95

Description read

[DESCRIPTION-READ] 8cf5fb27d0a615d48e9c28966427864b0cf53e50331c86b321af8260c7a0f91c

That sha256 names everything this verdict read: the PR description, plus the 5 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 6475e95f264c80f64a11a1e738fc9b49db4cbd3d via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the 300 ms throttle in useNotificationSound's onNotification can't swallow the dictation cue when a turn or approval sound fires just before it.

[DESIGN-REVIEWED] 6475e95

Description read

[DESCRIPTION-READ] 611f4ede9d3af89038157dd965d27709593cb7690c5d933ddf333cf77b168194

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 6475e95f264c80f64a11a1e738fc9b49db4cbd3d via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6475e95

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 6475e95f264c80f64a11a1e738fc9b49db4cbd3d via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The new dictation sound is on by default: every dictation user hears the all-categories chime (enabled: true, all: 'chime') when dictation stops by itself.

What this change ships

Inventory (6 items) — 6 justified

Intent: let someone dictating without watching the screen know, by sound, that dictation stopped without them, so they stop talking. This is an ADDITION.

  1. A sound plays when dictation stops on its own: an error frame, a dropped socket, or a full readiness buffer — justified
  2. Stopping, cancelling, unmounting or a superseded session stays silent, and the sound plays once per stop — justified
  3. New "Interrupted dictation" row in Settings > Notifications > Per-category sounds, placed between Conversation handoffs and Proactive agent messages — justified
  4. With no override, the row uses the all-categories sound, and all = none silences it — justified
  5. The row shows up in settings search, with its label and description in 12 languages — justified
  6. Light and dark screenshots committed under temp-screenshots/17842-dictation-sound/ — justified

[FIRST-PRINCIPLES-REVIEWED] 6475e95

Description read

[DESCRIPTION-READ] e1b98a945591d69b5a2a5ac971554e21b6bc10e9e70fe17032d309f56a239ebc

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 6475e95f264c80f64a11a1e738fc9b49db4cbd3d via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6475e95

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@amadsalmon
amadsalmon force-pushed the feat/dictation-stop-sound branch from 602d5ed to 65c922a Compare October 8, 2026 00:34
@amadsalmon

Copy link
Copy Markdown
Member Author

self-added: no
mechanism: each per-category "Use default" option names the sound it inherits, resolved through presetForKind

  • "Use default" doesn't match what plays — fixed in 65c922a

Each per-category row's "Use default" option now names the sound it plays without an override, from the same presetForKind resolution playback uses: Use default (Ding) on Interrupted dictation, Use default (Pulse) on Approvals and questions (the pre-existing mismatch), Use default (Chime (C6-E6-G6)) elsewhere.
Choosing "Use default" previews that same sound; it previously played the all-categories sound regardless of a built-in default.
Covered by NotificationsPanel.test.tsx: one test reads each row's select, one picks "Use default" on the dictation row and asserts ding. Each fails with its half of the change reverted.
Screenshots in the PR body show the labels in light and dark mode.

@amadsalmon

Copy link
Copy Markdown
Member Author

self-added: no

  • No one-time reader has seen the new row — rebutted (fork-lane ceiling)

On a fork PR the UX lane cannot run the blind first-time-reader pass, so it caps at CONCERNS regardless of the diff. Nothing in this PR can change that.
The label and description follow the neighbouring rows' pattern ("Conversation handoffs" / "When a conversation finishes or pauses for your input"): "Interrupted dictation" / "When dictation stops before you end it".
A maintainer can run the blind read by pushing this branch into the main repository, or record the judgment with /ai-review override ux <head-sha>: <reason>.

@amadsalmon

Copy link
Copy Markdown
Member Author

The workflow runs for the current head are waiting for maintainer approval (fork PR, all at action_required). Could a maintainer approve them? This push answers the UX Review's "Use default doesn't match what plays" item: each row's "Use default" now names the sound it plays (dispositions above).

@amadsalmon

amadsalmon commented Oct 8, 2026 •

Copy link
Copy Markdown
Member Author

Status for head 65c922afb: every red check is an infrastructure failure, not a finding on the diff.

  • Sensitive Change Review, Screenshot Evidence, Fork GPT 6.1 Review, Fork Opus 5.5 Review, Fork Internal Content Scan, Fork Security Scope Review: each died on API rate limit exceeded for installation between 00:44 and 00:47 UTC. Every fork review lane in the repo failed in that window.
  • Dependency License Gate: Dependency review is not supported on this repository (dependency graph off). It is failing on every PR right now, same-repo ones included.

A fork author cannot rerun jobs. Once the rate limit has reset, could a maintainer run:

gh run rerun 37708513239 --failed --repo kirodotdev/KiroCrew
gh run rerun 37708513173 --failed --repo kirodotdev/KiroCrew
gh run rerun 37709453525 --failed --repo kirodotdev/KiroCrew
gh run rerun 37709453534 --failed --repo kirodotdev/KiroCrew
gh run rerun 37709453519 --failed --repo kirodotdev/KiroCrew
gh run rerun 37709453594 --failed --repo kirodotdev/KiroCrew

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 8, 2026
@amadsalmon
amadsalmon force-pushed the feat/dictation-stop-sound branch from 65c922a to d92a713 Compare October 8, 2026 01:44
@amadsalmon

Copy link
Copy Markdown
Member Author

Pushed d92a713876, which supersedes the rerun list above for this PR: the new head runs every lane from scratch.

The fork runs for the new head will need approval again.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@amadsalmon
amadsalmon force-pushed the feat/dictation-stop-sound branch from d92a713 to 0460f45 Compare October 8, 2026 02:16
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@amadsalmon

Copy link
Copy Markdown
Member Author
  • Placement span=ba50182f09e5 — rebutted

Not changing: every per-category sound lives in Notifications > Per-category sounds, and the user changes them all together through "Default (all categories)". A dictation sound picker on the Voice tab would be the only sound control outside that panel, and would not follow the all-categories select the way users expect.
Discovery from the dictation side is covered: the settings registry entry notifications.sound-category-dictation is labelled "Interrupted dictation", so searching settings for "dictation" lists it next to the Voice-tab dictation entries and deep-links to the highlighted row.

@amadsalmon

Copy link
Copy Markdown
Member Author
  • category_dictation_description wording span=685f3ff46aac — rebutted

Not changing: "so you know to stop talking" names the wrong action. When capture ends on its own, what the user usually needs to do is start dictation again, not stop talking. Being told to stop could lose the rest of their sentence.
"Plays a sound when dictation stops before you end it" states the trigger plainly. The error shown with the cue gives the reason and the next step, for example lib.sttProviders.stream_error_max_duration: "Dictation reached its length limit. Start a new recording to continue."

@amadsalmon

amadsalmon commented Oct 8, 2026 •

Copy link
Copy Markdown
Member Author

Ready for maintainer review and merge on 6475e95f2: all 95 checks pass, including Integration now that #17964 is fixed (#18224).

This head is a rebase onto main 41afcb850 that also resolves an import-line conflict with #18377 in useStreamingStt.ts. On it, GPT 6.1 and Opus 5.5 report no blocking findings, Design and First Principles pass, and UX's two advisory notes have a disposition above.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@amadsalmon
amadsalmon force-pushed the feat/dictation-stop-sound branch from 0460f45 to 1b4bd13 Compare October 9, 2026 18:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Oct 9, 2026
A dictation can end without the user stopping it: the stream fails,
reaches its length cap, or loses its connection. The composer clears
the Listening state and shows an error, but someone reading from
another screen while dictating does not see it and keeps talking.

useStreamingStt now raises a new "dictation" sound kind whenever
capture ends while the user is still dictating. A stop or cancel the
user makes stays silent. The kind is sound-only, like "turn", and gets
its own "Interrupted dictation" row in Settings > Notifications, with
"ding" as its built-in default.

Part of kirodotdev#17842
@amadsalmon
amadsalmon force-pushed the feat/dictation-stop-sound branch from 1b4bd13 to 6475e95 Compare October 9, 2026 19:01
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 9, 2026
@amadsalmon

Copy link
Copy Markdown
Member Author
  • Description copy not shown span=c0f6a759dc91 — acknowledged, no change

The tooltip and search entry render category_dictation_description through the same component as every other row's description on this panel, so the wording is the only new thing, and it is checked by i18n:check and the 12 locale style suites. Adding a tooltip screenshot would mean a new head and another round of fork run approvals for a low-severity copy check. I can add one if a maintainer wants it.

  • Row placement span=514e96a446e0 — acknowledged, no change

As the lane notes, both themes render cleanly. The remaining gap is the blind first-time read, which the fork lane cannot run; only a maintainer push to this repository can supply it.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 9, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This plays the notification sound, under a new dictation category, when dictation stops without the user ending it. The three cases are an error frame, a dropped socket, and a full readiness buffer, and they map directly onto the three places the hook already knew capture had ended unexpectedly. I read the hook diff to check it plays only once. The error-frame path calls cleanup(), and the buffer path calls stop(), before the close handler runs, so the !captureStoppedRef.current check stops a second sound. The two new tests ("once, even when the close follows") pin that, and user stop, cancel and a superseded session stay silent. The settings row, registry and 12 catalogs are consistent. Design, First Principles, Opus and GPT pass, every UX item has a disposition, and CI is green. The screenshots are committed from a fork, which is allowed. — crew-pr-reviewer

@iamwhatever
iamwhatever enabled auto-merge (squash) October 10, 2026 07:56
@iamwhatever
iamwhatever merged commit 179b98f into kirodotdev:main Oct 10, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 10, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

This PR appears to have left src/kiro_crew/docs/ui-index.generated.json stale on main, which is now failing the Build Wheel / "Build frontend" step (and the three Build Desktop jobs) on head d3f462f23.

The new notifications.sound-category-dictation setting is in the registry and locales but not in the committed UI index, so node scripts/gen-ui-index.mjs --check fails:

gen-ui-index: src/kiro_crew/docs/ui-index.generated.json is stale. Run `npm run gen:ui` in website/ and commit the result.

Failing job: https://github.com/kirodotdev/KiroCrew/actions/runs/38036318549/job/114167446391

Fix is to run npm run gen:ui in website/ and commit the regenerated ui-index.generated.json (the only change is the one dictation entry across the 12 locales).

@amadsalmon

Copy link
Copy Markdown
Member Author

Thanks for catching this. Fix is up in #18803: it regenerates ui-index.generated.json on current main (adds the dictation entry across the 12 locales; gen-ui-index --check passes). The --check build step landed with #18443 after this PR's last CI run, which is why its checks stayed green. The same regeneration also covers the stale digest #18753 fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants