Repository navigation
feat(metrics): count an app's turn seconds as kirocrew.app.run_time - #39784
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the new counter is added at the one turn boundary both callers already share, so the existing Notes
[DESIGN-REVIEWED] b4b4670 Description read[DESCRIPTION-READ] 58d10458db657c65033159bf163eebbe0bf05bce4a19e05a0ceaa51f984b5974 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The run-time counter is justified. The only weak spot is its Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: let an app author see how much agent time runs under their app, per version. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] b4b4670 Description read[DESCRIPTION-READ] 84a6fe899a389bd10e2d1413cc1e506c98c7afab4809dd3964c0559032da659a That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
fb5da8a to
d48625a
Compare
|
GPT 6.1 finding on fb5da8a (usage.py:626, shared-session app subagents contribute no run time): fixed in d48625a. persist_token_record_async now passes |
d48625a to
7e4b3cc
Compare
Implements RFC section 11 PR 5 (app usage metrics, closed #39107): emit_turn_duration gains owning_app, and both turn callers pass the live provider's owning_app. Each finished turn of an app's session adds its wall seconds, rounded up, with the bucket of that turn. A turn with no duration adds nothing; a raising recorder leaves the turn metric unchanged. Closes #39773
7e4b3cc to
b4b4670
Compare
Code change approver deep review: ApproveApproved on commit Tier: 2, cleared by a deep review: a security-sensitive path, the size and category tally. A deep review by the code change approver, not by a person · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved by the code change approver's deep review, not by a person.
The review read the full diff on head b4b4670 against main. record_app_run_time has one call site, in emit_turn_duration, and both turn callers pass the provider's owning_app. The fallback to the usage row's app stamp in persist_token_record_async has its own test. The metric is not on main, and no open pull request duplicates it. PR Readiness is green on this head, and the Metric Schema Only Grows and backend test checks passed.
Done well: the tests drive the real emit_turn_duration, _emit_turn_metric and persist_token_record_async with a patched recorder, and one test runs a full dashboard turn.
Security: the flagged file is metrics/schema_manifest.json, a metric schema list, not a permission or app manifest. The change adds one row to it. Every label on that row is closed: Entity and entity_version go through app_public_id and app_public_version, so a private app sends only private, and duration has 8 fixed values. The diff changes no auth, sandbox, approval or secret code.
Optional:
- Turns an app runs through
ctx.turns.run(apps/agent_turn_sdk.py, used by the built-in meetings app) do not reachemit_turn_duration, so they add no run time. Either callrecord_app_run_timethere, or say indocs/system-specs/modules/metrics.mdthat these turns are not counted. - The schema history check refuses a removed key or value, so the
durationlabel is permanent after merge. Itsothervalue cannot occur on this row, because the seconds are always 1 or more. Confirm that the label is wanted on this row now. - No test covers the 1 m, 5 m, 15 m and 1 h edges of
app_duration_bucket. Add those rows to the parametrize intest/metrics/test_app_run_time.py. Open #39793 adds the same helper with a different rule for 0, so the second to merge must keep one.
Tier: 2, cleared by a deep review: a security-sensitive path, the size and category tally.
Criteria: no merge conflict, no owner block, no change request, the default branch, no security-sensitive path, no design document to agree on, the readiness status, the review lanes, the P0 history, the linked issues' tier, the review lessons, no concern label, the file count, a safe change category, not its own change, the decider.
Design documents ride along with the code (docs/system-specs/modules/metrics.md): they were not reviewed as a design, and a person who changes what they decide should ask for that review.
To overrule it, request changes or dismiss this review.
Problem / Motivation
Goal: Count how long each public app's agents run, per app version, as
kirocrew.app.run_time.Today an app author cannot see how much agent time runs under their app. Crew already measures every turn's wall time (
kirocrew.turn.duration), but nothing ties it to the app that owns the session. This PR implements RFC section 11 PR 5 of the app usage metrics RFC (closed PR #39107; its text is the spec, the rest ships as normal PRs).Why it matters
Total run time per app and version is one of the wave 1 numbers the RFC promises app authors (RFC section 6.3). Without it, an app's use can only be read from installs and opens, not from the agent work it causes.
Not a goal
session_started,present,opened/foreground_timeand Space ids (RFC PRs 4, 6, 7, 8).entity_versionanddurationbefore the analytics store keeps these keys; that is a privacy review by the collector owner, outside this repo. Until then the events arrive without those keys; the metrics store keeps every key.kirocrew.turn.durationorkirocrew.turn.completed.What changed (motivation -> approach -> change)
Every finished turn already passes through
emit_turn_duration(metrics/turns.py), the one boundary both turn callers share. So the app count is added there, fed by the session's trustedowning_app, which Crew sets on the provider from the slot's app at allocation.metrics/turns.pyemit_turn_durationtakesowning_appand, after the turn histogram, callsrecord_app_run_timewith the same duration it used.metrics/feature_usage.pyrecord_app_run_time(owning_app, duration_ms)adds the turn's wall seconds, rounded up, to the counterkirocrew.app.run_time(units) withapp(app_public_id),entity_version(app_public_version) andduration(app_duration_bucketof the rounded seconds:le_10s,le_1m,le_5m,le_15m,le_1h,le_4h,gt_4h,other; each edge is in the lower bucket). An empty or non-stringowning_app, or no positive duration, records nothing. A raising recorder is caught and logged at debug.dashboard/chat_turn/turn_stats.py,dashboard/handlers/usage.pyowning_app:_emit_turn_metricreads it off the turn'sclient,_emit_turn_histogramoffpersist_token_record_async'smodel_source, else that call's ownappownership stamp (a shared-session app subagent's provider carries noowning_app;subagent_manager/usage_row.pypassesapp=info.app).metrics/behavioral_metrics.py,metrics/product_metrics.py,metrics/schema_manifest.jsonkiro.crew, shape("app", None, "app"); the manifest is regenerated withpython test/metrics_schema_helpers.py --write(+1 row,durationclosed to its 8 values).docs/guides/telemetry-otlp-export.md,docs/system-specs/modules/metrics.mdEntityandentity_versionuse the exemptionkirocrew.app.installedalready has: only shipped or store apps and their published versions, elseprivateorother(RFC section 7).Backwards compatibility
Compatible: one new row and one new keyword argument with a default; the schema gate allows a new row, and every existing caller of
emit_turn_duration,_emit_turn_metricand_emit_turn_histogramkeeps working unchanged.Removes nothing: no metric row, key, value or function is removed or renamed.
Tests
test/metrics/test_app_run_time.py: seconds are rounded up (1 ms is 1 s, 10.001 s is 11 s); bucket edges 10 s isle_10sand 11 s isle_1m, 4 h isle_4hand 4 h + 1 s isgt_4h; the wall clock is used when the provider reports 0; a private app isprivate/private; a turn with no duration adds nothing; a turn with noowning_app(empty,None, a mock) adds nothing; a recorder that raises on the app row leaves the turn histogram and counters exactly as a turn outside an app; a non-number bucket isother; the dashboard caller passesowning_appthrough_emit_turn_metric, and a real dashboard turn (turn_harness.run_turn) on a provider withowning_appadds its seconds once; the shared boundarypersist_token_record_asyncreads it offmodel_source, and falls back to itsappargument when the provider has none (red without the fallback, green with it).test/metrics/test_behavioral_metrics.py: the new row gets theFeature/Activity/Entityshape.test/test_chat_runner_base_surface.py: the_emit_turn_metricsignature snapshot gainsowning_app: 'str' = ''.test/metrics/test_app_run_time.pypassed 5 of 5 repeats (20 passed each) and once in a shuffled order (shufof the collected ids, 20 passed).durationvalues are pinned by the manifest row, which the schema test compares with the code.Metric schema only grows against 3a651796a0 (+1 rows)),scripts/check_metrics_schema_history.py --test(10 probes ok);test/metrics/test_metrics_schema.py,test/metrics/test_product_metrics.pyandtest/test_metrics_schema_history_gate.pypass.pytest -n 2:test_app_run_time.py,test_behavioral_metrics.py,test_feature_usage.py,test_turn_metric.py,test_turn_metric_all_surfaces.py,test_harness_label.py,test_metrics_schema.py,test_product_metrics.py,test_turn_trigger_source.py,test_chat_runner_composition_contract.py,test_usage.py,test_usage_harness_sessions.py,test_chat_runner_base_surface.py,test_source_pin_budget.py,test_turn_profile.py,test_usage_dashboard_turn.py,test_metrics_schema_history_gate.py,test_session_started_attrs.py-> 934 passed.Manual verification
N/A -- unit coverage sufficient: the tests drive the real
emit_turn_duration,_emit_turn_metricandpersist_token_record_asyncwith a patched recorder, and the counter is an in-memory add with no UI.Real-machine verification
Machine: Linux cloud desktop, Amazon Linux 2023, x86_64
After (head b4b4670, on main 3a65179):
pytest -n 2on the touched files above -> 934 passedCleanup: own computer
Related Issues
Closes #39773
Implements RFC section 11 PR 5 from closed #39107. Built on #39715 (RFC PR 3, merged), which added
app_public_version(), and rebased on #39777 (RFC PR 4, merged); the manifest is regenerated on top of it.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsContribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.