Repository navigation
feat(metrics): send product metrics to Kiro telemetry from the public build - #37183
Conversation
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS No primary control was misread. The reader understood "Keep on", "Turn off" and the "Record metrics" switch, including that turning recording on sends data to Kiro telemetry. Notes
[UX-REVIEWED] 9de526d Description read[DESCRIPTION-READ] 74983a3fa491b3d40ddfa9a2807338526d9c417761b04fd3aecf31769a4cfd23 That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ human override accepted@chenmingwei23 overrode this lane for |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the owner made the default-export decision, the notice gate closes on failure, LTS, CI and dev-home builds return no destination, and the rendered evidence is attached. Notes
[DESIGN-REVIEWED] 9de526d Description read[DESCRIPTION-READ] 1c2ce2d0ea90bc5c9d7809a92f89df3be0c9e0ebc117c81892e4380c67631123 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] 9de526d Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider raised 2 possible problems and estimates a 62% chance that this change introduces a defect; a person should check them. I'm requesting changes. The PR makes the public build send product metrics and a hashed install id to a Kiro endpoint whenever recording is on, which it is by default, after a notice instead of a written opt-in. That reverses an explicit pin in the telemetry RFC: "the on default alone never exports, and neither Keep on nor the CLI notice counts as the opt-in". The only authority for the reversal is an "owner decision" claimed in the description and in the PR's own edit to a draft RFC. There is no accepted RFC on the base, no docs/decisions entry and no maintainer override, which the repository's process requires for this kind of default and trust-boundary change. The author also says a real accepted export to the endpoint should be shown before merge, and it hasn't been. The backend, frontend and E2E suites were still pending at decision time. The implementation itself looks careful: notice gating fails closed, and LTS builds, CI runners and non-default homes are excluded. Once a maintainer records the decision and real-endpoint export is shown, it may be mergeable. Things to check (possible problems the decider raised; at this confidence many turn out to be fine):
An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
3dc4fdd to
7e1d7dd
Compare
7e1d7dd to
7c1a451
Compare
b41caf3 to
1294923
Compare
341f436 to
decb1de
Compare
… build The public telemetry provider now returns one built-in destination, kiro-telemetry, after the operator's own entries. It sends the reviewed PRODUCT_METRICS list to the Kiro telemetry endpoint with ClientType=kiro-crew on every point, a reduced resource (no install id, pid or host.* keys), delta temporality, no OS type, and none of the process-wide OTel header, credential or certificate variables. Its session adds only x-kiro-machineid (a SHA-256 of the existing install id, never created here); no sign-in token or other credential is sent. It sends to no other URL and follows no redirect. It is the one destination that exports under the recording default (owner decision 2026-11-14): while local recording is on, by default or by choice, once the privacy notice that names it was shown. The notice is version 3: PRIVACY_NOTICE_VERSION goes from 2 to 3 in beacon.py and privacyNotice.ts, so the first-run Privacy chapter shows once more and gains a line naming Kiro telemetry, and the CLI one-line notice names it and stamps the version it printed. Keep on or `kirocrew telemetry enable` (both record a new dashboard.kiro_telemetry_ack_version, written only when the text shown named Kiro telemetry, and by the CLI only after its line reached a terminal; it is read-only in the all-settings page like privacy_ack_version) count, and so does the Kiro line printed on a terminal by a start with no dashboard (gateway --no-dashboard, run, chat), which writes its own stamp. A line that went to a log or a pipe, privacy_ack_version alone, and a telemetry.enabled: true written under an older notice do not count. The shown state is part of the rebuild fingerprint, read from the same config load the build's export decision used, so a Keep on attaches the reader at the next consent recheck. Operator entries still need the written opt-in (outbound_consented). Every off switch still stops it: heartbeat-off upgrades, a written false or non-bool, DO_NOT_TRACK, KIROCREW_TELEMETRY=0, telemetry disable. A policy otlp_hosts list can refuse its host. It is never returned in an LTS build, with or without a policy collector, on a CI runner, or under a non-default KIROCREW_HOME. Because the notice names it, it no longer makes the Settings switch or `kirocrew telemetry enable` refuse to turn recording on: both ask kiro_export.egress_needs_written_opt_in, which counts only the other destinations, and the collection status reports that as enable_refused. The collection status keeps export_permitted for the operator opt-in, and a recorder kept running while consent cannot be read still reports its Kiro export. The Privacy chapter says metrics are recorded here and names Kiro telemetry once, on its own line; Settings names the Kiro export apart from a collector of the user's own that still waits for the opt-in (which it calls "your own collectors"), and so does `kirocrew telemetry enable`. The Kiro stamp is written only when the line reached the terminal, and a headless start prints it even after a dashboard start stamped the generic line or a redirected `telemetry enable` wrote the generic acknowledgement; once printed to a log it is repeated only to a terminal. While the Kiro line is still owed, the theme boot payload holds the reported notice version back so the dashboard chapter shows once more. The Privacy chapter's Turn off line names the Kiro export, and the Telemetry panel footer no longer says "no egress". The OTLP/HTTP exporter becomes a base dependency (same pin as the otlp extra, which stays). Docs: the export guide, metrics.md, README privacy text and the crew-log telemetry RFC's export rule. The test suite turns the destination off in test/conftest.py so no test reaches the network.
decb1de to
9de526d
Compare
|
Product decision (owner) for head 9de526d: Public non-LTS builds export the reviewed The RFC text in this PR is updated to match; a maintainer review is still requested before merge. |
|
/ai-review override first-principles 9de526d owner-decision: owner decided default export of PRODUCT_METRICS to the built-in Kiro destination after the naming notice, LTS never, all opt-outs kept: #37183 (comment) |
Human judgment recorded@chenmingwei23 recorded a owner-decision release (class
This decision applies only to this commit. A new push requires a new judgment. |
|
This adds a new concept, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/). The First Principles finding was overridden by the PR author. That finding says the export consent gate now lets product metrics and a hashed install id leave a default public install, which main refuses. So a maintainer who is not the author has to decide this. Linked issue #21270 also asks for a maintainer sign-off on data scope before its pull requests merge. Neither this PR nor that issue has one yet. What the review found on this head:
Optional:
|
bolichen97
left a comment
There was a problem hiding this comment.
Direction signed by maintainer (bolichen97): default on, per owner decision 11-15.
I read the whole diff (code, tests, frontend, all 13 locales). It does what the checklist for default-on export asks.
Export conditions
- Export runs only when local recording is on and privacy notice v3 was seen.
notice_shown()readsdashboard.kiro_telemetry_ack_version >= 3or the notice stamp, and fails closed when neither is there. destinations()returns nothing for LTS, CI, a non-defaultKIROCREW_HOME, and the test suite (autouse fixture inconftest.py).- Operator destinations still need
outbound_consented. The enable refusals ignore only the built-in destination.
Off switches
_build_recorderreturns a null recorder when consent is false, before any destination is built. Sotelemetry.enabledfalse or non-bool,KIROCREW_TELEMETRY=0,DO_NOT_TRACK,kirocrew telemetry disable, and an upgrade with the heartbeat off all stay off. Tests pin each one.
What leaves the machine
- The endpoint is the constant
https://telemetry.us-east-1.kiro.dev/v1/metrics. The session sends only to that URL, with redirects off and TLS verification on. Any other URL gets a local fake 401 and nothing is sent. - Only the metrics in
PRODUCT_METRICS(counters and gauges with closed enum attributes) go out, plusClientTypeand six resource keys: service name and version, runtime name and version, harness, process role. No install id, pid, host or OS fields. - The only header added is
x-kiro-machineid, a SHA-256 of a versioned prefix plus the install id.install_id(create=False)never creates an id. No credentials, paths or prompt text are sent, andprocess_env_credentials=False.
Notice
PRIVACY_NOTICE_VERSIONgoes 2 to 3 inbeacon.pyandprivacyNotice.ts. The CLI and headless line stamps the notice only when it printed to a terminal. The dashboard writes the ack only when the chapter showed the Kiro line, and shows the chapter again while the Kiro notice is owed.
Other
- Export failures do not block startup.
- No regex in product code. The new dependency
opentelemetry-exporter-otlp-proto-http==1.44.0is pinned ininstall_requires.
Follow-ups, not blocking
- The PR says it was not run against the real endpoint. A real accepted export should be shown before or soon after merge.
- The wire test checks metric names, resource keys, the header and
ClientType. It does not check point attribute keys or values. That rests on the reviewedPRODUCT_METRICSdefinitions on main andtest_product_metrics.py.
I did not run the tests locally.
… to Kiro telemetry The help text for telemetry.enabled said default recording had no network egress, and the beacon_enabled help said recording egresses only to a configured destination after an explicit opt-in. With the Kiro telemetry destination from #37183, a default public install sends the reviewed product metrics to Kiro telemetry once the notice that names it has been shown. Say so in both help strings and in the TelemetryConfig docstring, and regenerate config-baseline.json.
…y destination (#22397, #37183) Backport to release/0.13.0 of three main commits, picked with git cherry-pick -x in main's merge order and squashed into one commit: - #22397 (e58c25a): record local usage metrics by default, with disk caps and a CLI notice. Existing opt-outs are kept: heartbeat off with telemetry.enabled unset, telemetry.enabled false (quoted and non-bool values included), KIROCREW_TELEMETRY=0, DO_NOT_TRACK=1. - #33422 (ffd2339): RFC update, telemetry.enabled now defaults to true. Docs only. - #37183 (c1222b9): the public build sends the reviewed PRODUCT_METRICS list to Kiro telemetry once privacy notice v3 has been shown; notice v3 names it; the OTLP exporter becomes a base dependency. Hand-resolved for 0.13: in docs/guides/telemetry-otlp-export.md the intro, the telemetry.enabled row and the collection/egress paragraph take the pick's text, and 0.13's own text is kept elsewhere. The pick's one-line additions to config_all_settings.py and its test are left out: the All settings page is not on 0.13. Also fixes the settings help for telemetry.enabled and beacon_enabled and the TelemetryConfig docstring, which said default recording sends nothing off the machine, and regenerates config-baseline.json. Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com> (cherry picked from commit e58c25a) (cherry picked from commit ffd2339) (cherry picked from commit c1222b9)
…y destination (#22397, #37183) (#37471) Backport to release/0.13.0 of three main commits, picked with git cherry-pick -x in main's merge order and squashed into one commit: - #22397 (e58c25a): record local usage metrics by default, with disk caps and a CLI notice. Existing opt-outs are kept: heartbeat off with telemetry.enabled unset, telemetry.enabled false (quoted and non-bool values included), KIROCREW_TELEMETRY=0, DO_NOT_TRACK=1. - #33422 (ffd2339): RFC update, telemetry.enabled now defaults to true. Docs only. - #37183 (c1222b9): the public build sends the reviewed PRODUCT_METRICS list to Kiro telemetry once privacy notice v3 has been shown; notice v3 names it; the OTLP exporter becomes a base dependency. Hand-resolved for 0.13: in docs/guides/telemetry-otlp-export.md the intro, the telemetry.enabled row and the collection/egress paragraph take the pick's text, and 0.13's own text is kept elsewhere. The pick's one-line additions to config_all_settings.py and its test are left out: the All settings page is not on 0.13. Also fixes the settings help for telemetry.enabled and beacon_enabled and the TelemetryConfig docstring, which said default recording sends nothing off the machine, and regenerates config-baseline.json. (cherry picked from commit e58c25a) (cherry picked from commit ffd2339) (cherry picked from commit c1222b9) Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com>
…f-form offer (#37460, #37463, #37545) Backport to release/0.13.0 of three main commits, picked with git cherry-pick -x in main's merge order and squashed into one commit. They follow up #37183, which #37471 already put on 0.13: - #37460 (2a679d6): the built-in kiro-telemetry destination is also returned under a non-default KIROCREW_HOME. kiro_export.destinations() drops only the is_default_home() check; the LTS, CI and test-suite gates and every consent gate stay. The heartbeat keeps its own default-home rule. - #37463 (8de422d): installs that are off only because an older build wrote telemetry.enabled: false into their config are shown the Privacy chapter once, in an off form that names Kiro telemetry, with Turn on performance metrics / Keep off. kiro_export.off_offer_due decides that case; Keep off records only the notice version so it is not asked again. - #37545 (09b0ddc): test only. The TestOffOffer tests fake the default data home, since the offer is asked only there and the test home is a private KIROCREW_HOME. All three picks applied without conflicts; each backported patch is identical to its main commit (git range-diff shows no change). (cherry picked from commit 2a679d6) (cherry picked from commit 8de422d) (cherry picked from commit 09b0ddc) Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com>
…f-form offer (#37460, #37463, #37545) Backport to release/0.13.0 of three main commits, picked with git cherry-pick -x in main's merge order and squashed into one commit. They follow up #37183, which #37471 already put on 0.13: - #37460 (2a679d6): the built-in kiro-telemetry destination is also returned under a non-default KIROCREW_HOME. kiro_export.destinations() drops only the is_default_home() check; the LTS, CI and test-suite gates and every consent gate stay. The heartbeat keeps its own default-home rule. - #37463 (8de422d): installs that are off only because an older build wrote telemetry.enabled: false into their config are shown the Privacy chapter once, in an off form that names Kiro telemetry, with Turn on performance metrics / Keep off. kiro_export.off_offer_due decides that case; Keep off records only the notice version so it is not asked again. - #37545 (09b0ddc): test only. The TestOffOffer tests fake the default data home, since the offer is asked only there and the test home is a private KIROCREW_HOME. All three picks applied without conflicts; each backported patch is identical to its main commit (git range-diff shows no change). (cherry picked from commit 2a679d6) (cherry picked from commit 8de422d) (cherry picked from commit 09b0ddc) Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com>
Problem / Motivation
Goal: The public build sends the reviewed product metrics to Kiro telemetry while local recording is on, after the user has been shown a privacy notice that names it; LTS builds never send there.
Today the public
DefaultTelemetryProvider.otlp_destinationsreturns only the operator's owntelemetry.otlp_endpoint(s)entries, so the public build sends product metrics nowhere. The seam for a build-shipped destination (narrowing fields from #26480,label_sources, thePRODUCT_METRICSpreset) is already on main; nothing in the public build uses it.(Goal and Not a goal updated on the owner's instruction of 2026-11-14: default recording also exports to this destination, after a new notice; LTS never.)
Why it matters
Maintainers cannot see how the public build is used beyond the daily heartbeat count. The product metrics list was reviewed for export.
Not a goal
telemetry.enabled: truewritten orKIROCREW_TELEMETRY=1.What changed (motivation -> approach -> change)
Owner decision (2026-11-14): in the public, non-LTS build, the built-in Kiro telemetry destination exports wherever local recording is on, including by default, but only after the user was shown a notice that names it (the dashboard's Privacy chapter once, with Keep on / Turn off, or the one-line CLI/headless notice). Before that nothing goes out. Every existing off switch stays off. LTS never sends there.
Approach: a destination built in code by the public provider, using the existing seam, plus one per-destination export rule in the recorder build and a notice version bump.
src/kiro_crew/metrics/kiro_export.py:kiro-telemetry, endpoint constanthttps://telemetry.us-east-1.kiro.dev/v1/metrics,allow=PRODUCT_METRICS,point_attributes={"ClientType": "kiro-crew"},resource_allowwithoutservice.instance.id,process.pid,os.type(Settings says the operating system is never sent) orhost.*,temporality="delta",process_env_credentials=False.DefaultTelemetryProvider.otlp_destinationsreturns config entries first, then it.x-kiro-machineid(SHA-256 of the existing install id; never creates the id). No request without it, none to any other URL, no redirects, standard TLS verification, no client certificate._build_recorderkeepskiro_export.exported(destinations, cfg, outbound): this destination whenkiro_export.notice_shown(), every other one only whenoutbound_consented.PRIVACY_NOTICE_VERSION2 -> 3 (beacon.py,privacyNotice.ts), so the Privacy chapter shows once more to users who saw v2, and it gets awhereBodyKiroline naming Kiro telemetry (all 13 catalogs + pseudolocale). The CLI/headless line (metrics/notice.py) names it and now stamps the version it printed, so v2 stamps print once more.notice_shown=dashboard.privacy_ack_version >= 3(Keep on, orkirocrew telemetry enable) or the CLI stamp at 3. Atelemetry.enabled: truewritten under an older notice does not count.notice_shown=dashboard.privacy_ack_version >= 3, or a separatekiro_telemetry_notice_shownstamp written only when a start with no dashboard (gateway --no-dashboard,run,chat) printed the Kiro line and the print succeeded. A headless start prints it even after a dashboard start stamped the generic line. The generic CLI stamp, the local-only line and a dashboard gateway's log line do not count.kirocrew telemetry enablename the Kiro export on its own line beside a collector of the user's own that still waits for the opt-in. The chapter's first paragraph uses newwhereBodyRecorded*strings (recorded here) when only Kiro telemetry gets metrics.dashboard.kiro_telemetry_ack_version(config field, theme-config PUT,config-baseline.json), written by Keep on only when the chapter showed the Kiro line, and bykirocrew telemetry enableonly after its Kiro line printed to a terminal; read-only in the all-settings page.notice_shownreads it, notprivacy_ack_version. The headless stamp is written only when the line went to a terminal. Turn off's scope line names the Kiro export; the Telemetry panel footer drops "local-only, no egress".live_recorder_statealso reports a kept Kiro export (kiro_export.last_build_kept);export_permittedstays the operator opt-in, and the chapter's operator line no longer counts the built-in row.notice_shown()is part of_endpoints_key, so the next consent recheck rebuilds and attaches the reader.telemetry enablekeep workingcore_config_patch.py409,cli_commands.py) askkiro_export.egress_needs_written_opt_in, which ignores it. The collection status addsenable_refused;PrivacyPanel.tsxpins the switch on that (falling back tootlp_configured), and has an off-state note for this destination.export_permittedis true when it is listed.()in an LTS build (with or without a policy collector), on a CI runner, under a non-defaultKIROCREW_HOME.opentelemetry-exporter-otlp-proto-http==1.44.0moves intoinstall_requires(same pin as theotlpextra, which stays).kirocrew telemetry statusgets aCollectors:line;telemetry enablenames it.test/conftest.pyturns it off; its own tests turn it back on.metrics.md, README privacy text, the crew-log telemetry RFC's export rule, the provider contract docstring.Who sends after this change (public build, default data home, not CI):
enablednever written, notice v3 shown"enabled": truewritten under the v2 notice, v3 not shown yetfalse/ non-bool written,KIROCREW_TELEMETRY=0,DO_NOT_TRACK=1,telemetry disableotlp_hostsdoes not permit the hostKIROCREW_HOMEotlp_endpointsentry, default recordingBackwards compatibility
No key, route or function is removed or renamed; the operator's entries keep their order, descriptors and opt-in rule. Behaviour changes, by the owner's decision: public-build users see the Privacy chapter (or the CLI line) once more, and after that their product metrics go to Kiro telemetry while recording is on. The collection status gains one field (
enable_refused); a client that ignores it keeps today's behaviour. The exporter package becomes a base dependency.Tests
test/metrics/test_kiro_export_destination.py: one test per row of the table above through real config files and the realget_recorder()(notice shown / not shown, CLI stamp v3 vs old stamp, written true under v2, heartbeat on/off, off switches, CI, operator entry with and without the opt-in), the rebuild fingerprint moving when the notice is shown, the enable refusals ignoring it and still refusing for an operator entry, the collection status fields,telemetry enableturning recording on and naming it, the CLI line text and stamp, provider order and narrowing, an operator entry with the same name is not it, LTS / CI / home / suite guard hide it, an LTS policy collector is the only destination, a policy host list refuses it, identity (machine id only, stored sign-in never sent), and one wire test through the real provider, build and OTLP exporter to a loopback receiver.test/test_telemetry_consent_notice.py(current version 3),test/test_telemetry_handlers_cov80.py(enable_refusedin the payload).PrivacyChapter.test.tsx(names Kiro telemetry for the build's destination, not for an operator entry of the same name),PrivacyPanel.recordMetrics.test.tsx(switch enabled and writes when only this destination exists).-n 2/vitest run): 1,057 + 996 backend tests across the metrics, telemetry, config-patch, LTS, beacon and CLI suites; 228 frontend tests including catalog parity;tsc --noEmit, eslint, the untranslated-values, DNT and pseudolocale checks.Manual verification
Not done against the real endpoint from this machine: dev homes do not list the destination by design. The wire test drives the real provider, build and exporter against a loopback receiver. A real accepted export to the endpoint should be shown before merge.
Screenshots / video
Captured with scripted Playwright against the built SPA on a loopback static server, with
/apistubbed (no gateway).Privacy chapter re-shown for notice v3 (recording on, acknowledged v2, no collector of the user's own): metrics are recorded here, Kiro telemetry is named once on its own line, and the Turn off line says it stops that export too.
Settings > Privacy, recording off, only the build's destination: the switch stays usable and the note says turning it on also sends product usage metrics.
Settings > Privacy, recording on, only the build's destination.
Settings > Privacy, recording on, beside a collector of the user's own that still waits for the opt-in: "Your own collectors get nothing..." and the Kiro line are named apart.
Related Issues
Related #21270. Supersedes the closed draft #22221.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.