Repository navigation
feat(metrics): count each installed app once per UTC day (kirocrew.app.present) - #39783
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this adds one row, takes nothing away, and the stamp on disk handles both restarts and a gateway that runs for days. Notes
[DESIGN-REVIEWED] 157ee0d Description read[DESCRIPTION-READ] f437b0f244969b77083f2ab0ecb91e92bb017a17e0b8eeb1859e1657f93d8f17 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The one note is the Notes
What this change shipsInventory (8 items) — 6 justifiedIntent: let an app author see how many machines still have their app installed, and on which version, day by day. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 157ee0d Description read[DESCRIPTION-READ] 7347679bfe72a28c36242754529ac2cdc3c2a483ba5fe2724f3684c38787bdc0 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
9751049 to
f7e5c67
Compare
|
Fixed in f7e5c67. |
f7e5c67 to
efe1056
Compare
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 404 lines, more than the 400 it may approve on its own. Adds a daily Checked against the repo:
Tier: 2, held for a person: security-sensitive path. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
efe1056 to
165e689
Compare
…p.present) Implements RFC section 11 PR 6 (app usage metrics, closed #39107). A gateway background task records one kirocrew.app.present row per installed app per UTC day, labelled entity_version, status (enabled/disabled/other) and backend (none/running/stopped/other). The first pass waits 10 minutes; the task then re-checks hourly and just after each UTC midnight, so a gateway that runs for days records every day. The last recorded UTC day is stamped next to the beacon's daily stamp, so restarts the same day record once. Nothing is recorded and no stamp is written while the recorder is off. Refs #39772
165e689 to
157ee0d
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. On head 157ee0d the review read the whole diff against linked issue #39772. It checked that record_today runs off the event loop through asyncio.to_thread, that the boot path gains only one deferred create_task, and that every app label goes through the existing app_public_id / app_public_version helpers and closed value sets. The review also ran 8 mutations against the new tests, and each one turned the test named for it red. The head merges cleanly onto current main.
Security: the only flagged path is src/kiro_crew/metrics/schema_manifest.json, which matched by name. It is the append-only metrics schema, not a scope or permission manifest. The diff touches no auth, sandbox or secret code. A private app's name and version never leave the machine. The new app_present_last_sent stamp only throttles telemetry, and it is read through the existing bounded, regular-file-only _read_state.
Done well: the pass reads the UTC day once and stamps that same day. So a pass that runs across midnight cannot mark the new day as done, and a test pins this.
- Optional: in
test/metrics/test_app_present.py:97, wrap the call asasyncio.run(asyncio.wait_for(app_present.run_daily(), timeout=10)). Then a loop that stops callingseconds_until_next_checkfails by name, instead of hanging until the 120 s suite timeout. - Optional: in the
_seamsfixture intest/test_gateway_process_entry.py, stubapp_present.startand assert it is called once outside test mode and never withtest_mode=True. Right now no test fails if you delete the start block inprocess_entry.py:745-748. - Optional: add a Before line on main and an After line from a real gateway run (one row per installed app, stamp file written) to
## Real-machine verification.
Problem / Motivation
Goal: Count how many machines have each app installed, per version, once per UTC day.
Today Crew counts installs, updates and uninstalls, but nothing says how many machines still have an app, which version they run, or whether it is switched off. This PR implements RFC section 11 PR 6 of the app usage metrics RFC (closed PR #39107; its text is the spec, the rest ships as normal PRs): a daily
kirocrew.app.presentrow per installed app.Why it matters
"Installed machines per version" and "version adoption" (RFC section 6.3) are read from this row. Without it an app author sees install events but not how many machines still have the app or how fast they move to a new version.
Not a goal
session_started(PR 4),run_time(PR 5),opened/foreground_time(PR 7) and Space ids (PR 8).entity_version,statusandbackendbefore the analytics store keeps these keys; that is a privacy review by the collector owner, outside this repo. Until then the events arrive without those keys; the metrics store keeps every key.telemetry.beacon_enabled(it does not carry app rows).What changed (motivation -> approach -> change)
The row must be sent once per app per UTC day, survive restarts, and keep firing on a gateway that runs for days (the daily re-arm review point on #39107). So the task wakes on a timer and checks a stamp on disk, instead of firing once after start.
metrics/app_present.py(new)list_apps()and the tracked backend processes off the event loop, records one row per app, and writes the stamp. A failed listing writes no stamp and retries on the next wake.beacon.pyapp_present_last_sentbeside the heartbeat'sbeacon_last_sent, written with the same atomic write (_write_stamp, shared with_mark_sent).metrics/feature_usage.pyrecord_app_present(app, status, backend, ids):Entityandentity_versionas oninstalled;statusenabled / disabled / other;backendnone (nobackend.entryPoint) / running / stopped / other.slack/gateway_runtime/process_entry.pytest_mode.metrics/behavioral_metrics.py,metrics/product_metrics.py,metrics/schema_manifest.jsonkiro.crew, shape("app", None, "app"); manifest regenerated withpython test/metrics_schema_helpers.py --write(+1 row).docs/guides/telemetry-otlp-export.md,docs/system-specs/modules/metrics.mdstatusandbackendare closed.Entityandentity_versionuse the exemptionkirocrew.app.installedalready has (RFC section 7). No request path runs the pass. Nothing is recorded, and no stamp is written, while telemetry is off.Backwards compatibility
Compatible: one new metric row and one new state file; the schema gate allows a new row.
Removes nothing:
_mark_sentkeeps its stamp and its never-raise contract through the shared_write_stamp; no key, row or function is removed.Tests
test/metrics/test_app_present.py: one row per app with the right version,status=disabledfor a disabled app, and all threebackendstates (running,stopped,none); two restarts on one UTC day record one row per app (the stamp is on disk); the realrun_dailyloop crossing UTC midnight records one more row per app and nothing on a third wake the same day; a pass whose clock crosses UTC midnight stamps the day it recorded, so the new day is still due; the wait ends 1 s past midnight and never exceeds an hour; telemetry off records nothing and writes no stamp; a raising recorder leaves the pass complete and the stamp written; a failed listing writes no stamp and the next pass records.test/metrics/test_behavioral_metrics.py: the new row gets theFeature/Activity/Entityshape.scripts/check_metrics_schema_history.py --test(10 probes ok) and the gate against origin/main (+1 rows);test/metrics/test_metrics_schema.py,test/metrics/test_product_metrics.py,test/test_metrics_schema_history_gate.pypass.Manual verification
N/A -- unit coverage sufficient: the tests drive the real
run_dailyloop and the real stamp file in a temp dir; the row is an in-memory add with no UI.Real-machine verification
Machine: Linux cloud desktop, Amazon Linux 2023 (kernel 6.12), x86_64, 64 vCPU / 123 GB
After (head 157ee0d, rebased on main bdc6526):
pytest -n 2 test/metrics/test_app_present.py test/metrics/test_behavioral_metrics.py test/metrics/test_feature_usage.py test/metrics/test_metrics_schema.py test/test_metrics_schema_history_gate.py test/test_beacon.py test/test_gateway_process_entry.py test/metrics/test_product_metrics.py test/test_flake_pattern_ratchet.py test/test_sweep_check.py test/test_review_repair_routing_skill.py test/test_packaged_docs_runtime_facts.py-> 586 passedCleanup: own computer
Related Issues
Closes #39772
Implements RFC section 11 PR 6 from closed #39107, and answers its review point that the task must re-arm every UTC day on a long-running gateway and send one row per app across restarts. Built on #39715 (RFC PR 3, merged), whose
app_public_version()labels the row.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsContribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.