Skip to content

fix(kas): show the telemetry-off notice once per chat - #35788

Merged
bolichen97 merged 1 commit into
mainfrom
crew/slack/kas-telemetry-notice-once
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/slack/kas-telemetry-notice-once

Conversation

@buluoray

@buluoray buluoray commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A KAS chat tells a user who turned kiro-cli telemetry off about it once, not on almost every message.

A user turned kiro-cli telemetry off (telemetry.enabled: false) and chatted on KAS. The chat kept showing the same notice: "ℹ Your kiro-cli telemetry is off, so Kiro Crew asked the KAS engine to turn its telemetry off too. KAS does not confirm this, so Kiro Crew cannot check that it took effect." In a long dashboard chat it showed on almost every message.

Why it matters

Every user who opted out of kiro-cli telemetry and chats on KAS gets this row over and over. Nothing they do in the chat makes it stop. The rows bury the conversation, and they teach users to skip notices.

Not a goal

  • Changing how often the other session-start notices show. Each of them names a step the user can take, and it stops once they take it.
  • A dismiss button or a setting for this notice.
  • Changing what the notice says, or when KAS is told to turn telemetry off.

What changed (motivation → approach → change)

A chat now shows the telemetry notice once. It shows a telemetry notice again only when the text is a different one.

The chat's next KAS session carries The chat shows
the notice it showed last nothing
the other telemetry notice that notice, once
no notice nothing

Symptom: the notice repeats. Cause: a chat goes through many KAS sessions. It gets one after it sat idle past session.timeout_secs (60 minutes by default), after a gateway restart, and after a model, project or harness switch. Every session the chat creates or reloads carries the notice, and the dashboard added a row on the first turn of each one. Nothing remembered that the chat was already told.

Change:

  • _notice_steer_upgrade in src/kiro_crew/dashboard/chat_turn/turn_spec_hooks.py keeps a short mark of the last telemetry notice text on the chat's slot (_telemetry_notice_shown, in state_parts/chat_slot.py and state_parts/slot_session.py). It adds the row only when the text differs from that mark.
  • The mark is saved on the chat's metadata line as telemetry_notice_shown (slot_persistence/metadata_codec.py), so a gateway restart keeps it. A value on that line that is not a mark is ignored, and the chat is told again.
  • docs/system-specs/modules/acp-client.md describes the new rule.

Two choices to note:

  • The mark is of the text, not a yes/no flag. There are two telemetry notices: one says the opt-out was sent, the other says it was not applied to this chat. A chat that saw the first is still told the second.
  • The mark is saved on the metadata line, not looked up in the chat's rows. After a restart a chat loads only its newest 500 rows, so a long chat would be told again.

kiro-cli and the kiro harness show no such notice, so they set no rate to match. The change that added the notice (#33026) meant one row per chat.

Backwards compatibility

Compatible: the metadata line gains one optional key. A line without it reads as "not told yet", so an existing chat shows the notice one more time and then stays quiet.

Removes nothing: the diff adds a slot field, a metadata key and a check before one notice row. No path, key, record or function is taken away, and the notice still shows once per chat.

Tests

  • test/test_kas_cli_settings.py::test_a_chat_shows_the_telemetry_notice_once: three session starts of one chat give one row; the other notice shows once; a second chat is told on its own.
  • test/test_kas_cli_settings.py::test_a_restart_does_not_repeat_the_telemetry_notice: a chat is saved with the real save, dropped, and restored from disk; its next session start adds no row. A line whose value is not a mark tells the chat again.
  • test/test_slot_metadata_codec.py: the new key in the round trip for every read purpose, and in both key orders.
  • test/test_state_parts_facade.py: the new slot field's default.

Proof: PROVEN test/test_kas_cli_settings.py::test_a_chat_shows_the_telemetry_notice_once, test/test_kas_cli_settings.py::test_a_restart_does_not_repeat_the_telemetry_notice, test/test_state_parts_facade.py::test_a_fresh_slot_has_every_base_default (head a018133)

Determinism of the new and changed tests: 20 runs at -n0, 10 runs at -n 4 --dist loadgroup --max-worker-restart=2 with test/test_kas_agent_prompt_floor.py and test/test_kas_withheld_credentials_notice.py as co-tenants, and one shuffled order of the three test files. All green.

Manual verification

Run on an isolated preview gateway built from this change, in a real KAS chat. The run was made before the last rebase onto main; the rebase changed no line of the diff. The chat's project folder had a .kiro/settings/cli.json with "telemetry.enabled": false.

  1. Sent one message. The chat showed the notice once.
  2. Restarted the gateway. The chat came back with its three rows.
  3. Sent a second message in the same chat. The crew log shows a second session/opened for the chat. No new notice row.

Result: five rows (user, notice, assistant, user, assistant), one telemetry notice. The chat's metadata line carried telemetry_notice_shown.

Related Issues

Closes #35781

Sibling sweep

Swept: git grep -n "if is_new" -- src/kiro_crew/dashboard/chat_turn/
Sibling: src/kiro_crew/dashboard/chat_turn/turn_spec_hooks.py -- not affected -- its other session-start rows either name a fix the user can apply (the six other rows of _notice_steer_upgrade: an update command, a setting, the model picker, a file to remove) or state a fault of that session (Crew's tools are off; a field of the agent spec has no effect on this backend), so they end when the user acts
Sibling: src/kiro_crew/dashboard/chat_turn/mcp_session.py:236 -- not affected -- the row says this session lacks one of Crew's own servers, a fault of that session that its next session may not have

Pattern harvest

Not generalizable: the defect is one notice with no step the user can take, drawn by a loop built for notices that end when the user acts.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 12, 2026 08:34
@buluoray
buluoray requested a review from mbriones98 November 12, 2026 08:35
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a018133939e1586f4e490486b589e19bc2580dc4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a018133939e1586f4e490486b589e19bc2580dc4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the per-chat mark targets the real cause, repeated session starts, and the optional metadata key is backward-compatible and easy to revert.

Notes

  • When a user clears or resets a chat, the mark stays (it is "never cleared"). A cleared chat can lose the notice row for good while the opt-out still applies. Decide whether a reset should clear _telemetry_notice_shown.

[DESIGN-REVIEWED] a018133

Description read

[DESCRIPTION-READ] a9a4d489205950c170ea7ec6452d4a644e0454f4fc07e6667ad8327513b55a5a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a018133939e1586f4e490486b589e19bc2580dc4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each item traces to the defect in #35781 and is covered by tests that fail on base. The per-chat mark sits at the right level: it records that the chat was already told.

What this change ships

Inventory (5 items) — 5 justified

Intent: a user who opted out of kiro-cli telemetry sees the KAS telemetry notice once per chat, not on almost every message. This is a FIX.

  1. A dashboard chat shows the telemetry notice once, however many KAS sessions it goes through. — justified
  2. The chat shows the notice again only when the text changes (sent becomes "not applied", or back). — justified
  3. A gateway restart keeps the "already told" state, through the new telemetry_notice_shown key on the metadata line. — justified
  4. A telemetry_notice_shown value that is not a mark is ignored, so the chat is told again. — justified
  5. acp-client.md describes the once-per-chat rule. — justified

[FIRST-PRINCIPLES-REVIEWED] a018133

Description read

[DESCRIPTION-READ] 27ec7abd4524c6ab8c94bf2371dd410df2d22612e6d36f0542ca7a40837b9947

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a018133939e1586f4e490486b589e19bc2580dc4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a018133

Verdict parsed from the review's SHA-scoped output markers for commit a018133939e1586f4e490486b589e19bc2580dc4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a018133939e1586f4e490486b589e19bc2580dc4 false-positive: <reason, 20 to 500 characters>

@buluoray
buluoray enabled auto-merge (squash) November 12, 2026 08:38
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a018133939e1586f4e490486b589e19bc2580dc4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a018133

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a018133939e1586f4e490486b589e19bc2580dc4 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 12, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on one point. The rest looks right to me.

New regex in product code. dashboard/slot_persistence/metadata_codec.py adds:

_TELEMETRY_NOTICE_MARK_RE = re.compile(r"\A[0-9a-f]{16}\Z")

We keep new regex out of product code. A plain check does the same job:

_HEX = frozenset("0123456789abcdef")

def _read_telemetry_notice_shown(r: _Read) -> None:
    mark = r.meta.get("telemetry_notice_shown")
    if isinstance(mark, str) and len(mark) == 16 and set(mark) <= _HEX:
        r.slot._telemetry_notice_shown = mark

The existing "no mark" case in test_a_restart_does_not_repeat_the_telemetry_notice still covers the reject path.

The rest I read in full and have no concerns. _telemetry_notice_is_new dedupes only telemetry_unconfirmed_notice, stores a 16-char sha256 mark, and shows the notice again when its text changes. The metadata field is wired like human_seen, and the tests cover the once-per-chat case, a changed answer, a second chat and a restart.

Not checked: I ran no tests locally and relied on CI (96 green).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 12, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (8 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: a dashboard chat shows the KAS telemetry notice once, keyed by a mark of its text saved on the chat's metadata line, with tests for repeat sessions and restart. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/acp-client.md.

@KiroCrewCaptain
KiroCrewCaptain dismissed their stale review November 12, 2026 10:06

Withdrawn: a maintainer requested changes on this head shortly before this approval was posted, so its no-requested-changes criterion no longer holds. Auto-merge is off; this PR now waits for a maintainer.

A KAS chat whose user turned kiro-cli telemetry off showed the "telemetry
is off" notice on the first turn of every session the chat created or
reloaded: after an idle timeout, a gateway restart, or a model, project or
harness switch. In a long chat that was almost every message.

The chat now keeps a mark of the last telemetry notice it showed and adds
a row only when the notice text differs from it. The mark is saved on the
chat's metadata line (telemetry_notice_shown), so a restart keeps it.

Closes #35781
@buluoray
buluoray force-pushed the crew/slack/kas-telemetry-notice-once branch from 051fcd7 to a018133 Compare November 12, 2026 10:20
@buluoray

Copy link
Copy Markdown
Contributor Author

@bolichen97 I dropped _TELEMETRY_NOTICE_MARK_RE, and _read_telemetry_notice_shown now uses your _HEX frozenset check, at src/kiro_crew/dashboard/slot_persistence/metadata_codec.py:750-756 in a018133. The reject case is still covered by the "no mark" line of test_a_restart_does_not_repeat_the_telemetry_notice (test/test_kas_cli_settings.py:202), which passes locally on this head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 12, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

The PR adds a per-chat mark (a 16-character hex digest of the last telemetry notice text) and adds the notice row only when the text changes. The mark is saved as an unowned, set-and-never-cleared metadata key, which the full save keeps from the disk copy (carry_unowned_metadata). The read accepts only exactly 16 hex characters, so a forged value just means the chat is told again. The only production caller passes a real slot that always has the field. Existing tests that pass object() never set the telemetry notice. The new tests cover several session starts, switching between the two notice texts, separate chats, a restart from disk, a forged value and the codec round trip. No regression on sibling notices.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit a018133939 · 143 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 12, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on a018133. My regex point is fixed: _TELEMETRY_NOTICE_MARK_RE is gone, and _read_telemetry_notice_shown now checks len(mark) == 16 and set(mark) <= _HEX. I compared this diff with the one I reviewed on 051fcd7. Apart from that swap, only hunk offsets from main moved. There is no re.compile or other regex in the diff.

Not checked: I ran no tests locally and relied on CI (97 green).

@bolichen97
bolichen97 merged commit 2905c5c into main Nov 12, 2026
97 of 98 checks passed
@bolichen97
bolichen97 deleted the crew/slack/kas-telemetry-notice-once branch November 12, 2026 12:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

KAS chat repeats the "telemetry is off" notice each time its session restarts

3 participants