Skip to content

feat(metrics): send product metrics from non-default data homes too - #37460

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
feat/kiro-telemetry-dev-homes
Nov 15, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
feat/kiro-telemetry-dev-homes

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Installs under a non-default KIROCREW_HOME (dev homes, pods, worktree previews) also send product metrics to the built-in Kiro telemetry destination, under the same consent rules as any other install.

#37183 hid the kiro-telemetry destination under a non-default data home, copying the daily heartbeat's rule. Those installs are people using the product, so their product metrics are missing.

Why it matters

Product usage from people who run Kiro Crew under their own data home (common for developers and pods) is invisible today, even after they acknowledged the notice that names the destination.

Not a goal

  • Any client-side dedupe by IP: no IP is read or sent.
  • The daily heartbeat: it keeps its own default-home rule.
  • Any consent gate.

What changed (motivation -> approach -> change)

Owner decision (2026-11-15): counting dev machines is acceptable. The smallest change is to drop the one check.

Want Change
Non-default homes export too kiro_export.destinations() drops the beacon.is_default_home() check. LTS, CI and the test-suite gate stay.
Every consent gate unchanged No change to kiro_export.exported, notice_shown, outbound_consented, the off switches or the otlp_hosts policy.
Docs match README, docs/guides/telemetry-otlp-export.md (who-sends table), docs/system-specs/modules/metrics.md, the crew-log telemetry RFC export paragraph; two docstrings that cited the non-default home as an example.

Backwards compatibility

Behaviour change by design: an install under a non-default KIROCREW_HOME that has recording on and has seen the version 3 notice now exports PRODUCT_METRICS to Kiro telemetry. It also now lists the destination in Settings > Privacy and kirocrew telemetry status. Nothing else changes.

Tests

  • test/metrics/test_kiro_export_destination.py: new test_a_non_default_home_gets_it (the test fixture's private home is non-default, and the destination is listed); the "not listed" case drops home and keeps CI, LTS and the suite guard; the fixture no longer pretends the home is default, so every existing case now runs under a non-default home.
  • Run locally with -n 2: test_kiro_export_destination.py, test_telemetry_consent_notice.py, test_usage_notice.py, test_telemetry_handlers_cov80.py, metrics/test_otlp_endpoint_list.py, test_lts_telemetry_endpoint.py, test_api_theme_config.py: 356 passed.

Manual verification

N/A: one gate removed in a pure function; the wire test drives the real provider, build and exporter against a loopback receiver from a non-default home.

Screenshots / video

N/A: no file under website/ changes.

Related Issues

Follow-up to #37183. Related #21270.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

The built-in kiro-telemetry destination was hidden under a non-default
KIROCREW_HOME (a dev home, a pod, a worktree preview), the same hosts the daily
heartbeat does not count. Those installs are people using the product, so the
destination is now returned there by the same rules as any other home.

kiro_export.destinations() drops only the is_default_home() check. The LTS, CI
and test-suite gates stay, and so does every consent gate: recording on, the
version 3 notice shown, every off switch, the policy otlp_hosts list. No IP is
read or sent. The heartbeat keeps its own default-home rule.

README, the export guide, metrics.md and the crew-log telemetry RFC say the same.
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 7220ec4d42d99572cba340c4cca1d4521a47ec89 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7220ec4d42d99572cba340c4cca1d4521a47ec89 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The owner decided to count dev machines, every consent gate is unchanged, and the removed home pin is replaced by a test that pins the new decision.

Notes

  • Every worktree preview or pod home mints its own hashed install ID, so one developer can show up as many installs. If the metrics are read as install counts, flag that to whoever reads them. This is not a reason to change the gate.

[DESIGN-REVIEWED] 7220ec4

Description read

[DESCRIPTION-READ] 0ed306f9d5c474033f42bd1e35295bb54f5d6e49bfd8e1171fce3f880648f1dc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7220ec4d42d99572cba340c4cca1d4521a47ec89 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The default egress change rests on an owner decision of 2026-11-15 that only the description asserts. The base RFC behind the Kiro telemetry destination is draft.

Notes

  • Item 1 — inherited: its only provenance is the line "Owner decision (2026-11-15): counting dev machines is acceptable". No docs/decisions/ entry carries it. The base RFC rfc-crew-log-telemetry-export.md reads status: draft. Because this change affects only installs under a non-default home, I'm treating it as a note rather than a product-shape block. A maintainer override recorded on this head would remove the doubt.
  • Item 5 — the pin being reversed is the one feat(metrics): send product metrics to Kiro telemetry from the public build #37183 added, "The private home is not the default one, which hides the destination". The description names that reversal. The removed rule's only stated reason was symmetry, "same rule as the daily heartbeat". Dropping an inherited rule doesn't need further evidence.

What this change ships

Inventory (5 items) — 3 justified

Intent: let people running under their own data home have their product usage counted, under the existing consent rules. This is an ADDITION: a changed default.

  1. Installs under a non-default KIROCREW_HOME now send product metrics to Kiro telemetry by default. — inherited: the owner decision is asserted only in the description.
  2. Settings > Privacy and kirocrew telemetry status now list kiro-telemetry under a non-default home. — justified
  3. A non-default home that acknowledged the Privacy chapter before sees it once more, now naming the destination. — undeclared
  4. The README, the guide, the spec, the RFC paragraph and two docstrings say that non-default homes export. — justified
  5. The test fixture stops faking a default home, the home hidden-case is removed, and a non-default-home test is added. — justified

[FIRST-PRINCIPLES-REVIEWED] 7220ec4

Description read

[DESCRIPTION-READ] 08ffd81cfcc064fc84d3856a334bd185d8ea9f7ba120c165c6a61ec0622e127b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7220ec4d42d99572cba340c4cca1d4521a47ec89 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7220ec4

Verdict parsed from the review's SHA-scoped output markers for commit 7220ec4d42d99572cba340c4cca1d4521a47ec89.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7220ec4d42d99572cba340c4cca1d4521a47ec89 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7220ec4d42d99572cba340c4cca1d4521a47ec89 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7220ec4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7220ec4d42d99572cba340c4cca1d4521a47ec89 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

The PR drops the one beacon.is_default_home() check in kiro_export.destinations(), so non-default homes now send product metrics to Kiro telemetry. The LTS, CI and test-suite gates stay. I checked the code paths: the install ID is created on the recording path whatever the home is, so the hashed machine ID exists and export works. Consent is unchanged: notice_shown and exported are untouched and every off switch still applies. The dashboard Privacy chapter names the destination from the backend's own list of destinations, so it now names it and records the Kiro acknowledgement correctly. Non-default homes that acknowledged the chapter before see it once more, which is the documented behaviour of _ack_version_for_the_chapter. The other uses of is_default_home (daily heartbeat, launchd repair) are unaffected. Only the destination's own test file switches the test-suite suppression off, and test/conftest.py still sets it everywhere else. The new test and the fixture change would fail on main. The owning spec and docs are updated in the same commit.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 7220ec4d42 · 38 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 15, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 15, 2026 06:11

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On head 7220ec4, the review traced every caller of kiro_export.destinations(). These are the provider listing, the CLI notice text, kirocrew telemetry status and enable, and the Privacy chapter hold-back in dashboard/handlers/core.py. They all read the one function, so they change together. notice_shown(), outbound_consented, the off switches and the otlp_hosts policy are unchanged, so nothing is sent before a notice that names the destination has been acknowledged. The test-suite floor (_suppressed_for_tests), the LTS and CI exclusions, and the heartbeat's own default-home rule all still hold. PR Readiness is green on this head.
Done well: the home fixture no longer pretends the home is the default one. So the whole existing consent, notice and loopback wire suite now runs on the new path, and the new test checks its own precondition first.
The design is signed off under the standing maintainer sign-off for changes by team members.

Optional:

  • The Backwards compatibility section says "Nothing else changes". But an install under a non-default home that acknowledged the version 3 chapter will see the Privacy chapter once more, and a headless start prints the Kiro line once, before anything is sent (core.py:566, metrics/notice.py:109). That is the correct consent behaviour. One line in that section would tell readers about it.
  • test/metrics/test_kiro_export_destination.py:12 still says non-default homes do not list the destination. Line 105 still describes the old rule as current.
  • The RFC export paragraph still files this change under the 2026-11-14 decision. Naming the 2026-11-15 decision there, with a link to where it is recorded, would help future readers.

chenmingwei23 added a commit that referenced this pull request Nov 16, 2026
…f-form offer (#37460, #37463, #37545)

Backport to release/0.13.0 of three main commits, picked with
git cherry-pick -x in main's merge order and squashed into one commit.
They follow up #37183, which #37471 already put on 0.13:

- #37460 (2a679d6): the built-in
  kiro-telemetry destination is also returned under a non-default
  KIROCREW_HOME. kiro_export.destinations() drops only the
  is_default_home() check; the LTS, CI and test-suite gates and every
  consent gate stay. The heartbeat keeps its own default-home rule.
- #37463 (8de422d): installs that are off
  only because an older build wrote telemetry.enabled: false into their
  config are shown the Privacy chapter once, in an off form that names
  Kiro telemetry, with Turn on performance metrics / Keep off.
  kiro_export.off_offer_due decides that case; Keep off records only the
  notice version so it is not asked again.
- #37545 (09b0ddc): test only. The
  TestOffOffer tests fake the default data home, since the offer is asked
  only there and the test home is a private KIROCREW_HOME.

All three picks applied without conflicts; each backported patch is
identical to its main commit (git range-diff shows no change).

(cherry picked from commit 2a679d6)
(cherry picked from commit 8de422d)
(cherry picked from commit 09b0ddc)

Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com>
chenmingwei23 added a commit that referenced this pull request Nov 16, 2026
…f-form offer (#37460, #37463, #37545)

Backport to release/0.13.0 of three main commits, picked with
git cherry-pick -x in main's merge order and squashed into one commit.
They follow up #37183, which #37471 already put on 0.13:

- #37460 (2a679d6): the built-in
  kiro-telemetry destination is also returned under a non-default
  KIROCREW_HOME. kiro_export.destinations() drops only the
  is_default_home() check; the LTS, CI and test-suite gates and every
  consent gate stay. The heartbeat keeps its own default-home rule.
- #37463 (8de422d): installs that are off
  only because an older build wrote telemetry.enabled: false into their
  config are shown the Privacy chapter once, in an off form that names
  Kiro telemetry, with Turn on performance metrics / Keep off.
  kiro_export.off_offer_due decides that case; Keep off records only the
  notice version so it is not asked again.
- #37545 (09b0ddc): test only. The
  TestOffOffer tests fake the default data home, since the offer is asked
  only there and the test home is a private KIROCREW_HOME.

All three picks applied without conflicts; each backported patch is
identical to its main commit (git range-diff shows no change).

(cherry picked from commit 2a679d6)
(cherry picked from commit 8de422d)
(cherry picked from commit 09b0ddc)

Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants