Skip to content

fix(metrics): put the hashed install id on Kiro telemetry data points so they are not dropped - #37969

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/kiro-telemetry-machineid-point-attr
Nov 16, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/kiro-telemetry-machineid-point-attr

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Product usage metrics that the public build sends to Kiro telemetry reach product analytics instead of being dropped.

The built-in kiro-telemetry destination identifies an install only with the x-kiro-machineid request header, and sends no sign-in token. The endpoint accepts such a request (200 Accepted), but it does not copy that header onto the points it stores. So each stored point has no identity, and the analytics forwarder drops every point with no identity.

Evidence from 2026-11-16: 66 kiro.crew records from public-build installs reached the analytics stage that day, and 0 of them carried any identity. A probe export from a fresh home got 200 Accepted; its stored record had no identity either. Anonymous records from the Kiro IDE do get through, because the IDE puts machineId on each data point itself.

Why it matters

Every public-build install that exports product metrics is invisible in product analytics today. The data leaves the machine, under the privacy notice, and is then thrown away, so the export costs the user bandwidth and gives the product team nothing.

Not a goal

  • Changing what is sent: the same reviewed metrics, the same hash, the same endpoint.
  • Changing the receiving side so it copies the header onto points.
  • Sending a sign-in token or any other identity.

What changed (motivation → approach → change)

Points are dropped because the identity sits only in a header the endpoint does not store. So each data point now carries the same hash as an attribute, under the key the receiving side already reads.

Part Change
metrics/kiro_export.py destinations() point_attributes adds machineId = machine_id_hash(), the exact value of the header, when an install id exists. With no install id nothing is sent anyway, so the key is left out.
metrics/destination_filter.py ALLOWED_POINT_ATTRIBUTE_KEYS admits machineId. The allowlist is shared, so an operator's otlp_endpoints entry may name that key too.
metrics/kiro_export.py is_builtin() Compares descriptors without machineId. A descriptor built before the install id existed and one built after are the same destination; the shared session still tells an operator entry of the same name apart.
metrics/provider.py _build_recorder Mints the install id before it resolves destinations, not after the readers are built. Without this, a fresh install's first export carried the header but no machineId on its points. The mint still runs only with consent.

The raw install id still never leaves the machine; only its SHA-256 under a fixed prefix does, which the privacy notice already describes as "a hashed install ID". The notice text and the Privacy screen still match, so they are unchanged. The two docs that said ClientType was the only point attribute name machineId now.

Backwards compatibility

Compatible: one key is added to an allowlist and to this destination's points; every config and descriptor that works today still works.

Removes nothing: no key, path, check or record is taken away.

Tests

  • test/metrics/test_destination_filter.py: the allowlist is {ClientType, machineId} and a wrapper accepts machineId.
  • test/metrics/test_kiro_export_destination.py::TestIdentity::test_the_points_carry_the_header_hash: the point attribute equals the header value.
  • test/metrics/test_kiro_export_destination.py::TestIdentity::test_it_is_still_builtin_when_the_install_id_appears_later: descriptors from before and after the install id exist are both built in; an operator entry named kiro-telemetry and None are not.
  • test/metrics/test_kiro_export_destination.py::test_the_real_provider_sends_product_metrics_to_the_endpoint: the real provider and exporter, from a home with no install id, post to a loopback receiver; every data point carries machineId equal to the header.

Proof: PROVEN test/metrics/test_destination_filter.py::TestRefusedPointAttributes::test_the_reviewed_key_is_accepted, test/metrics/test_kiro_export_destination.py::TestIdentity::test_the_points_carry_the_header_hash, test/metrics/test_kiro_export_destination.py::TestIdentity::test_it_is_still_builtin_when_the_install_id_appears_later, test/metrics/test_kiro_export_destination.py::test_the_real_provider_sends_product_metrics_to_the_endpoint (head 357a0d7)

Manual verification

N/A for the receiving side: whether stored points now reach product analytics can only be seen after this ships. Locally, with the provider.py hunk reverted, the loopback test sees 40 points with no machineId; with it, all points carry the header's hash.

Related Issues

Follows #37183, which added this destination. No separate issue.

Sibling sweep

Swept: git grep -nE "x-kiro-machineid|HEADER_MACHINE_ID|identity_headers" -- src/
Siblings: none -- kiro_export.py is the only destination that sends identity in a header; operator entries carry no built-in identity.

Pattern harvest

Not generalizable: the receiving side's choice not to store a request header is specific to this endpoint.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: it reuses the existing hash on the existing endpoint. The only new piece is a stored per-point copy of an identity the privacy notice already says is sent.

Notes

  • Opening ALLOWED_POINT_ATTRIBUTE_KEYS to every destination means an operator's otlp_endpoints entry can now set a constant machineId. That does no harm today. If only this destination should ever carry machineId, a narrower option is to keep the shared allowlist as it is and let only kiro_export add the key.

[DESIGN-REVIEWED] a90d0ce

Description read

[DESCRIPTION-READ] 73583784090e012e7856922f7b26b5a34277a1807d51fd049d366280b6e82149

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix is at mechanism level, and the receiving side, which is the real cause, is declared out of scope. The one rider is that operators can now set machineId too, because the allowlist is shared.

Not checked: whether the receiving side really reads machineId from each point. The diff can't show this.

Notes

  • Item 3 — rides along: "The allowlist is shared, so an operator's otlp_endpoints entry may name that key too." Operators can now put any short constant under machineId on their own endpoints. Subtraction: keep ALLOWED_POINT_ATTRIBUTE_KEYS as {"ClientType"} and let only kiro_export.destinations() attach machineId. This holds only if the built-in destination's attributes can skip the operator allowlist without adding a new path.

What this change ships

Inventory (4 items) — 3 justified

Intent: get the product metrics from public-build installs into product analytics instead of having them dropped. FIX.

  1. Every Kiro telemetry data point now carries machineId, the same hash the header already sends — justified
  2. The built-in destination counts as built in both before and after the install id exists (4 is_builtin call sites: kiro_export.py ×3, dashboard/handlers/telemetry.py ×1) — justified
  3. Operator otlp_endpoints may now set a machineId point attribute — rides along; widens operator allowlist
  4. On a fresh install, the install id is minted before destinations are resolved, so the first export already carries it — justified

[FIRST-PRINCIPLES-REVIEWED] a90d0ce

Description read

[DESCRIPTION-READ] 3653f6eb863e61b657e1ed115c3b5a015ab6e694e5aa33fbc50e96d8e37e07fe

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a90d0ce

Verdict parsed from the review's SHA-scoped output markers for commit a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a90d0ce

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a90d0ce1e3f4cd6cebd10f4f8f7deccd28326cdf false-positive: <reason, 20 to 500 characters>

… so they are not dropped

The built-in kiro-telemetry destination sent its identity only in the
x-kiro-machineid request header. The endpoint accepts such a request but
does not copy the header onto the points it stores, so every stored point
had no identity and was dropped before product analytics.

Each data point now carries machineId, the same hash as the header, when
an install id exists. ALLOWED_POINT_ATTRIBUTE_KEYS admits machineId.
is_builtin() compares descriptors without machineId, so one built before
the install id existed still matches one built after. The recorder build
mints the install id before it resolves destinations, so a fresh install's
first export already carries it.
@chenmingwei23
chenmingwei23 force-pushed the fix/kiro-telemetry-machineid-point-attr branch from 357a0d7 to a90d0ce Compare November 16, 2026 14:04
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Approve. The PR adds machineId (the same SHA-256 install-id hash the x-kiro-machineid header already sends) to the built-in Kiro telemetry destination's points, adds the key to the shared allowlist, makes is_builtin compare without machineId, and creates the install id before destinations are resolved so the first export already carries it. Checks: the value comes from the same function as the header and is left out when there is no id (then nothing is sent). Install-id creation still runs only after consent. An operator entry named kiro-telemetry is still told apart by the shared session. label_sources behaves the same. The descriptor is never logged or shown in the UI. The privacy notice already says metrics are "labelled with a hashed install ID". Tests cover the allowlist, the point value matching the header, built-in detection before and after the id exists, and an end-to-end loopback export from a fresh home. CI and all review lanes passed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit a90d0ce1e3 · 152 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 16, 2026 15:38

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. On head a90d0ce we checked four things. The built-in destination adds machineId only when an install id already exists, using the same machine_id_hash() as the x-kiro-machineid header, and it never mints an id. is_builtin strips machineId on both sides, so every caller still agrees before and after the id appears. The moved mint in _build_recorder still runs after the consent and OTel returns. Each code hunk has a test that fails when that hunk is reverted. The Proof head 357a0d7 is a patch-identical rebase of this head, and PR Readiness is green here.

Well done: the new precondition line assert beacon.install_id(create=False) == "" turns the existing loopback test into a guard for the ordering fix. With that line, every received point must carry the header hash from the very first export.

Optional:

  • The point attribute is fixed when the recorder is built, but the header is read on every request. If the install id is minted again while the process runs, the two values can differ until the next rebuild. Adding machine_id_hash() to _endpoints_key would close that gap if it ever matters.
  • The Proof line still names head 357a0d7. You can refresh it the next time you edit the body.

@KiroCrewCaptain
KiroCrewCaptain merged commit cdc86b1 into main Nov 16, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/kiro-telemetry-machineid-point-attr branch November 16, 2026 15:39
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
chenmingwei23 added a commit that referenced this pull request Nov 16, 2026
…try data points so they are not dropped (#37969) (#37988)

The built-in kiro-telemetry destination sent its identity only in the
x-kiro-machineid request header. The endpoint accepts such a request but
does not copy the header onto the points it stores, so every stored point
had no identity and was dropped before product analytics.

Each data point now carries machineId, the same hash as the header, when
an install id exists. ALLOWED_POINT_ATTRIBUTE_KEYS admits machineId.
is_builtin() compares descriptors without machineId, so one built before
the install id existed still matches one built after. The recorder build
mints the install id before it resolves destinations, so a fresh install's
first export already carries it.

(cherry picked from commit a90d0ce)

Co-authored-by: Raymond Chen <chenmingwei23@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants