Repository navigation
feat(metrics): measure chat runtime sharing adoption, savings, turn lag and stalls - #37189
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Additive telemetry on closed labels at seams that already exist; the backwards-compatibility claim holds against the code, so nothing blocks. Notes
[DESIGN-REVIEWED] 568a60d Description read[DESCRIPTION-READ] f397bbc1d10f45cac0af38ea0d64fdf499aa115472775ffc4e284f4fe212f0ab That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every series answers a question in the Goal, except two riders: the KAS engine install counter and the preview-flip counter. Both carry permanent allowlisted names with no Goal question behind them. Notes
What this change shipsInventory (10 items) — 8 justifiedIntent: give maintainers a week of data to decide whether the chat runtime sharing preview goes on by default. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 568a60d Description read[DESCRIPTION-READ] 95cc4a4b1174a3feb43474cbe5f1337fcaf422f937b110cbb8d430d42c964d2e That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
9dc5839 to
6f4ab9c
Compare
|
Fixed in 6f4ab9c (findings on 9dc5839):
|
6f4ab9c to
e1b5439
Compare
|
Fixed in e1b5439 (First Principles on 6f4ab9c):
The rate-limited lanes on 6f4ab9c (Dependency License Gate, Publish readiness signal, Analyze, GPT) re-run with this push. |
70c8e08 to
4ac3173
Compare
|
Fixed in 4ac3173 (findings on 70c8e08):
|
4ac3173 to
568a60d
Compare
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own. This PR only adds telemetry: 11 counters and gauges with closed labels for the chat runtime sharing preview, placed at seams that already exist. I checked each emit site in the code. Every emit runs after the step it measures and cannot raise. The census excludes registration leases correctly (entry.key is entry.runtime). The RSS sample and reclaim are each counted once. Turn start skips fanned and synthetic events. The death count reads held leases by identity. No new import cycle appears, and the product allowlist and docs are updated. Only stale docstring or spec wording and a measurement-scope caveat remain, none of which blocks. CI was still pending. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
The review checked this head (568a60d) and found nothing blocking. Every new emit goes through emit_counter, which swallows errors. Every label is a closed tuple and no key, pid, path or prompt reaches one. The new modules import cleanly with no new import cycle. The two large files stay under the 2,000-line cap. The 23 new tests pass from an extracted copy of the head.
Done well: each call-site test drives the real production function, including a real AcpSessionHandle turn on an injected clock. The refusal reasons, turn outcomes and harness labels are pinned against their source classifiers, so a new branch reds a test instead of exporting other.
Optional:
- The
runtime_globalandsynthetic_completionstart filters and the_yielded_terminalguard in session_handle_turn.py (around lines 922 and 989) have no mutation pin. Removing any one of them leaves all 23 tests green. One case with a co-tenant frame before the session's own chunk, plus one with two terminals, would pin them. - On main, product metrics go only to destinations the operator configures. Consider saying in the description where the week of cross-gateway data is read from, and which per-gateway key the recipe groups by there. The open #37183 drops
service.instance.id. turn_endedis recorded only at a yielded EVENT_COMPLETE, so a turn that ends by raising is never counted. A line in the metrics.md row saying so would keep its error share from being compared directly withkirocrew.turn.completed.
Problem / Motivation
Goal: A week after people turn on the "ACP runtime sharing" preview, the numbers say whether it can be on by default: how many gateways use it, what it saves, and whether shared chats start late, stall or crash together.
The
chat_runtime_sharingfeature preview (Settings > Developer) places several dashboard chats on one agent process. Nothing measures it today. The only trace is a WARNING log line per placement, which carries a session key, cannot be aggregated across gateways, and says nothing about turn latency, KAS queueing, memory or crashes.Why it matters
The default-on decision needs a fixed bar and data to hold it against. Without these counters the call rests on a few pod runs: a shared process that answers late, a KAS engine that still queues shared turns, or one crash that drops four chats would only show up as user reports after the default flips.
Not a goal
What changed (motivation -> approach -> change)
Goal -> count at the seams that already exist on main, so the numbers do not wait on #36400 -> eleven counters and gauges, all on closed labels.
kirocrew.inventory.config.togglegainskey=preview_chat_runtime_sharing, the gauge every other switch reports adoption on. A live-config applier inconfig_watch.pycounts each on/off flip from any writer (chat_runtime_share.toggled). Both read the preview through the existingchat_runtime_sharing_enabled, reached from application code via the newsrc/kiro_crew/agent_sdk/chat_runtime_sharing.py, so there is no second spelling of the preview id.providers/chat_share.pycountsplaced{outcome=spawned|joined|refused, reason, tenants}beside its existing WARNING lines.src/kiro_crew/metrics/chat_runtime_share.pyread the ownership table (newRuntimeOwnership.keyed_lease_countsinsrc/kiro_crew/runtime_ownership.py): live processes placed by sharing and the chats on them; the difference is processes saved. They register from the newsrc/kiro_crew/metrics/shared_process_gauges.py, which also takes over the MCP gateway pool registration fromsrc/kiro_crew/metrics/provider.pyso that file stays under the 2,000-line cap. The RSS watchdog tick (src/kiro_crew/session_cleanup.py,src/kiro_crew/session_cleanup_leftovers.py,src/kiro_crew/session_cleanup_spare.py) counts each shared process's tree RSS once per tick, and counts a reclaim when it closes one to joins.session_handle_turn.pycounts prompt-written-to-first-event delay and how the turn ended, both split by the chat leases on the process (0subagent/background,1alone,2+shared) andharness. The outcome and harness labels come frommetrics.turns.turn_outcomeandmetrics.harness.harness_for_backend, the classifierskirocrew.turn.completeduses.session_handle_kas.pycounts KASkind: queuednotices (AgentExecutionWaitingForActiveSlot), which must stay near zero on 0.66.104.src/kiro_crew/runtime_death.py(announce) counts every death with the chat leases it held, read by identity (newRuntimeOwnership.held_leases_on_runtime) so a process the exit watcher already reaped still reports its blast radius.kas_engine_install.pycounts download outcomes. On main a failed bundle install fails the spawn; there is no relay fallback to count.Distributions are bucket-labelled counters because the product allowlist takes counters and gauges only. All names join
PRODUCT_METRICS;metrics.mdand the OTLP export guide list every row. No session key, pid, path, model id or user text reaches a label.Backwards compatibility
Compatible: every change adds an emit after the step it measures;
emit_counterswallows its own failures and the bucketing is total.Removes nothing: no instrument, label value, config key or behaviour is taken away; the toggle gauge only gains one
keyvalue.Tests
test/metrics/test_chat_runtime_share.py(23 tests) drives each production seam and asserts its counter: a founding and a joining placement, a refused start (and none with the preview off), the census gauges on a real meter (registration leases excluded), the RSS sample and a once-only reclaim, deaths (including a shared process already reaped, whose live lease count reads 0), one real turn throughAcpSessionHandle.prompton a shared and an unshared runtime (clock moved by the fake backend, no sleep), a KAS queued frame (and a subagent on its host chat's process reading0), each install outcome, a preview flip through the registered applier, and one gauge set failing to register without costing the other. Refusal reasons are pinned by AST overchat_sharing_ineligible_reason; turn labels are checked equal tometrics.turns.turn_outcomeandharness_for_backend.Mutation check: 12 mutations, each replacing one emit call (the two
failedinstall sites together), the census key filter, or the reaped-death lease read with a no-op; every one turned at least one test red.Also run, serially:
test_inventory_gauges.py,test_product_metrics.py,test_mcp_gateway_gauges.py,test_crew_log_health.py,test_chat_runtime_sharing.py,test_chat_runtime_sharing_cleanup.py,test_acp_session_handle_abandoned_turn.py,test_acp_runtime_death.py,test_runtime_death_is_a_process_event.py,test_kas_wire.py,test_config_live_server_appliers.py,test_session_rss.py,test_kas_route.py, both flake ratchets. isort, black, flake8, mypy, semgrep, the file line cap and the agent-sdk boundary gate pass.Manual verification
N/A for UI: nothing renders. Read recipe for the week (OTLP backend, grouped by
service.instance.id):inventory.config.toggle{key=preview_chat_runtime_sharing}= 1 on any day;toggled{state=off}/toggled{state=on}is the churn.placed{outcome=joined}/ (spawned+joined);refusedbyreasonshows what keeps chats apart (kas_engine_not_shareable= engine too old or not installed).tenants - processes, time-weighted per gateway;rss_sampledbytenantsandrssfor memory per shared process;reclaimedper shared process.turn_started{harness=kas}p95 bucket fortenants=1vstenants>=2;turn_endedshare oftool_stall,stale_recover,timeoutper tenants bucket.kas_queued{tenants>=2}total.process_died{expected=false, tenants>=2}per 1,000 shared turns, and the tenants mix of those.Related Issues
Closes #37187
Part of #17545
Other suggestions
Proposed default-on bar after 7 days:
joinedplacements in total.kas_queued{tenants>=2}is 0 on engines at 0.66.104 or later.turn_startedp95 bucket for KAStenants>=2is no more than one bucket abovetenants=1, and itsgt_30sshare is within 1 point of it.tool_stall+stale_recover+timeout) fortenants>=2is within 0.5 points oftenants=1.tenantsis at most 2.5.reclaimedhit at most 2% of shared processes, andkas.engine_install{outcome=failed}is under 5% of attempts.The census gauges publish only in a process that has placed or refused a chat with the preview on, so their absence elsewhere is expected, not a broken probe.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)