Skip to content

feat(metrics): count an app's turn seconds as kirocrew.app.run_time - #39784

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
feat/app-run-time
Nov 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
feat/app-run-time

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Count how long each public app's agents run, per app version, as kirocrew.app.run_time.

Today an app author cannot see how much agent time runs under their app. Crew already measures every turn's wall time (kirocrew.turn.duration), but nothing ties it to the app that owns the session. This PR implements RFC section 11 PR 5 of the app usage metrics RFC (closed PR #39107; its text is the spec, the rest ships as normal PRs).

Why it matters

Total run time per app and version is one of the wave 1 numbers the RFC promises app authors (RFC section 6.3). Without it, an app's use can only be read from installs and opens, not from the agent work it causes.

Not a goal

  • RFC rows session_started, present, opened / foreground_time and Space ids (RFC PRs 4, 6, 7, 8).
  • The downstream collector key list. It must keep entity_version and duration before the analytics store keeps these keys; that is a privacy review by the collector owner, outside this repo. Until then the events arrive without those keys; the metrics store keeps every key.
  • Any change to kirocrew.turn.duration or kirocrew.turn.completed.

What changed (motivation -> approach -> change)

Every finished turn already passes through emit_turn_duration (metrics/turns.py), the one boundary both turn callers share. So the app count is added there, fed by the session's trusted owning_app, which Crew sets on the provider from the slot's app at allocation.

Part Change
metrics/turns.py emit_turn_duration takes owning_app and, after the turn histogram, calls record_app_run_time with the same duration it used.
metrics/feature_usage.py record_app_run_time(owning_app, duration_ms) adds the turn's wall seconds, rounded up, to the counter kirocrew.app.run_time (unit s) with app (app_public_id), entity_version (app_public_version) and duration (app_duration_bucket of the rounded seconds: le_10s, le_1m, le_5m, le_15m, le_1h, le_4h, gt_4h, other; each edge is in the lower bucket). An empty or non-string owning_app, or no positive duration, records nothing. A raising recorder is caught and logged at debug.
dashboard/chat_turn/turn_stats.py, dashboard/handlers/usage.py Both turn callers pass the live provider's owning_app: _emit_turn_metric reads it off the turn's client, _emit_turn_histogram off persist_token_record_async's model_source, else that call's own app ownership stamp (a shared-session app subagent's provider carries no owning_app; subagent_manager/usage_row.py passes app=info.app).
metrics/behavioral_metrics.py, metrics/product_metrics.py, metrics/schema_manifest.json One new row on scope kiro.crew, shape ("app", None, "app"); the manifest is regenerated with python test/metrics_schema_helpers.py --write (+1 row, duration closed to its 8 values).
docs/guides/telemetry-otlp-export.md, docs/system-specs/modules/metrics.md Export table and metrics spec list the new row and its keys.

Entity and entity_version use the exemption kirocrew.app.installed already has: only shipped or store apps and their published versions, else private or other (RFC section 7).

Backwards compatibility

Compatible: one new row and one new keyword argument with a default; the schema gate allows a new row, and every existing caller of emit_turn_duration, _emit_turn_metric and _emit_turn_histogram keeps working unchanged.

Removes nothing: no metric row, key, value or function is removed or renamed.

Tests

  • test/metrics/test_app_run_time.py: seconds are rounded up (1 ms is 1 s, 10.001 s is 11 s); bucket edges 10 s is le_10s and 11 s is le_1m, 4 h is le_4h and 4 h + 1 s is gt_4h; the wall clock is used when the provider reports 0; a private app is private / private; a turn with no duration adds nothing; a turn with no owning_app (empty, None, a mock) adds nothing; a recorder that raises on the app row leaves the turn histogram and counters exactly as a turn outside an app; a non-number bucket is other; the dashboard caller passes owning_app through _emit_turn_metric, and a real dashboard turn (turn_harness.run_turn) on a provider with owning_app adds its seconds once; the shared boundary persist_token_record_async reads it off model_source, and falls back to its app argument when the provider has none (red without the fallback, green with it).
  • test/metrics/test_behavioral_metrics.py: the new row gets the Feature / Activity / Entity shape.
  • test/test_chat_runner_base_surface.py: the _emit_turn_metric signature snapshot gains owning_app: 'str' = ''.
  • Determinism: test/metrics/test_app_run_time.py passed 5 of 5 repeats (20 passed each) and once in a shuffled order (shuf of the collected ids, 20 passed).
  • The 8 duration values are pinned by the manifest row, which the schema test compares with the code.
  • Flake-pattern ratchet against 3a65179: passed.
  • Schema gates: the gate against the merge base (Metric schema only grows against 3a651796a0 (+1 rows)), scripts/check_metrics_schema_history.py --test (10 probes ok); test/metrics/test_metrics_schema.py, test/metrics/test_product_metrics.py and test/test_metrics_schema_history_gate.py pass.
  • Touched-file run, pytest -n 2: test_app_run_time.py, test_behavioral_metrics.py, test_feature_usage.py, test_turn_metric.py, test_turn_metric_all_surfaces.py, test_harness_label.py, test_metrics_schema.py, test_product_metrics.py, test_turn_trigger_source.py, test_chat_runner_composition_contract.py, test_usage.py, test_usage_harness_sessions.py, test_chat_runner_base_surface.py, test_source_pin_budget.py, test_turn_profile.py, test_usage_dashboard_turn.py, test_metrics_schema_history_gate.py, test_session_started_attrs.py -> 934 passed.

Manual verification

N/A -- unit coverage sufficient: the tests drive the real emit_turn_duration, _emit_turn_metric and persist_token_record_async with a patched recorder, and the counter is an in-memory add with no UI.

Real-machine verification

Machine: Linux cloud desktop, Amazon Linux 2023, x86_64
After (head b4b4670, on main 3a65179): pytest -n 2 on the touched files above -> 934 passed
Cleanup: own computer

Related Issues

Closes #39773

Implements RFC section 11 PR 5 from closed #39107. Built on #39715 (RFC PR 3, merged), which added app_public_version(), and rebased on #39777 (RFC PR 4, merged); the manifest is regenerated on top of it.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 20, 2026
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b4b4670b2c8d5d0b5707b6d6deb852cff08de36a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b4b4670b2c8d5d0b5707b6d6deb852cff08de36a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the new counter is added at the one turn boundary both callers already share, so the existing emit_metric gate keeps it from double counting.

Notes

  • Rounding every turn up to the next second makes run time look longer than it was when an app's turns are short (each sub-second turn counts as a full second). If that matters, the RFC row should say whether it wants per-turn ceilings or the summed milliseconds ceiled once.
  • The duration bucket labels each turn's length on a counter that sums seconds. That puts the turn-length distribution on the same row as the total, so readers must sum across buckets to get an app's run time.

[DESIGN-REVIEWED] b4b4670

Description read

[DESCRIPTION-READ] 58d10458db657c65033159bf163eebbe0bf05bce4a19e05a0ceaa51f984b5974

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b4b4670b2c8d5d0b5707b6d6deb852cff08de36a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The run-time counter is justified. The only weak spot is its duration label, whose sole support is RFC text from a closed PR (#39107) that is not on base.

Notes

  • Item 3 — inherited: the 8-value duration label rests on "RFC section 11 PR 5 … (closed PR docs(rfc): propose app usage metrics (installs, versions, sessions, run time) #39107; its text is the spec)". No RFC on base covers app usage metrics, and the Goal ("count how long each public app's agents run, per app version") is met by the summed seconds alone. Per-turn spread is already measured by the kirocrew.turn.duration histogram. Subtraction: drop APP_DURATION_VALUES, app_duration_bucket and the duration attribute and manifest key, and keep app and entity_version only.

What this change ships

Inventory (5 items) — 4 justified

Intent: let an app author see how much agent time runs under their app, per version. ADDITION.

  1. A new exported counter kirocrew.app.run_time adds each app-owned turn's seconds, rounded up, tagged with app and version — justified
  2. Both turn callers now pass the session's trusted owning_app into the shared turn-duration boundary — justified
  3. Each run-time point carries an 8-value duration bucket label (le_10s … gt_4h, other) — oversized
  4. A shared-session app subagent with no provider owning_app is counted under its usage row's app stamp — justified
  5. The schema manifest, behavioral-metric shape, export guide and metrics spec list the new row — justified

[FIRST-PRINCIPLES-REVIEWED] b4b4670

Description read

[DESCRIPTION-READ] 84a6fe899a389bd10e2d1413cc1e506c98c7afab4809dd3964c0559032da659a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b4b4670b2c8d5d0b5707b6d6deb852cff08de36a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b4b4670

Verdict parsed from the review's SHA-scoped output markers for commit b4b4670b2c8d5d0b5707b6d6deb852cff08de36a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b4b4670b2c8d5d0b5707b6d6deb852cff08de36a false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b4b4670b2c8d5d0b5707b6d6deb852cff08de36a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b4b4670

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b4b4670b2c8d5d0b5707b6d6deb852cff08de36a false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 force-pushed the feat/app-run-time branch 2 times, most recently from fb5da8a to d48625a Compare November 20, 2026 05:15
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

GPT 6.1 finding on fb5da8a (usage.py:626, shared-session app subagents contribute no run time): fixed in d48625a.

persist_token_record_async now passes getattr(model_source, "owning_app", "") or app, so a provider without owning_app (the shared-session AcpSessionProvider built in subagent_manager/run_start.py) falls back to the row's own trusted ownership stamp, which subagent_manager/usage_row.py passes as app=info.app. New test test_a_provider_without_owning_app_falls_back_to_the_rows_app in test/metrics/test_app_run_time.py is red without the fallback and green with it.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 20, 2026
Implements RFC section 11 PR 5 (app usage metrics, closed #39107):
emit_turn_duration gains owning_app, and both turn callers pass the live
provider's owning_app. Each finished turn of an app's session adds its wall
seconds, rounded up, with the bucket of that turn. A turn with no duration
adds nothing; a raising recorder leaves the turn metric unchanged.

Closes #39773
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 20, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver deep review: Approve

Approved on commit b4b4670b2c: the review says why.

Tier: 2, cleared by a deep review: a security-sensitive path, the size and category tally.

A deep review by the code change approver, not by a person · commit b4b4670b2c · 373 changed lines. If it is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 20, 2026 07:58

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by the code change approver's deep review, not by a person.

The review read the full diff on head b4b4670 against main. record_app_run_time has one call site, in emit_turn_duration, and both turn callers pass the provider's owning_app. The fallback to the usage row's app stamp in persist_token_record_async has its own test. The metric is not on main, and no open pull request duplicates it. PR Readiness is green on this head, and the Metric Schema Only Grows and backend test checks passed.

Done well: the tests drive the real emit_turn_duration, _emit_turn_metric and persist_token_record_async with a patched recorder, and one test runs a full dashboard turn.

Security: the flagged file is metrics/schema_manifest.json, a metric schema list, not a permission or app manifest. The change adds one row to it. Every label on that row is closed: Entity and entity_version go through app_public_id and app_public_version, so a private app sends only private, and duration has 8 fixed values. The diff changes no auth, sandbox, approval or secret code.

Optional:

  • Turns an app runs through ctx.turns.run (apps/agent_turn_sdk.py, used by the built-in meetings app) do not reach emit_turn_duration, so they add no run time. Either call record_app_run_time there, or say in docs/system-specs/modules/metrics.md that these turns are not counted.
  • The schema history check refuses a removed key or value, so the duration label is permanent after merge. Its other value cannot occur on this row, because the seconds are always 1 or more. Confirm that the label is wanted on this row now.
  • No test covers the 1 m, 5 m, 15 m and 1 h edges of app_duration_bucket. Add those rows to the parametrize in test/metrics/test_app_run_time.py. Open #39793 adds the same helper with a different rule for 0, so the second to merge must keep one.

Tier: 2, cleared by a deep review: a security-sensitive path, the size and category tally.
Criteria: no merge conflict, no owner block, no change request, the default branch, no security-sensitive path, no design document to agree on, the readiness status, the review lanes, the P0 history, the linked issues' tier, the review lessons, no concern label, the file count, a safe change category, not its own change, the decider.
Design documents ride along with the code (docs/system-specs/modules/metrics.md): they were not reviewed as a design, and a person who changes what they decide should ask for that review.

To overrule it, request changes or dismiss this review.

@KiroCrewCaptain
KiroCrewCaptain merged commit f8a805a into main Nov 20, 2026
99 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the feat/app-run-time branch November 20, 2026 07:58
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

metrics: add kirocrew.app.run_time counter fed from emit_turn_duration

2 participants