Skip to content

feat(metrics): measure chat runtime sharing adoption, savings, turn lag and stalls - #37189

Merged
chenmingwei23 merged 1 commit into
mainfrom
feat/acp-sharing-metrics
Nov 14, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
feat/acp-sharing-metrics

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A week after people turn on the "ACP runtime sharing" preview, the numbers say whether it can be on by default: how many gateways use it, what it saves, and whether shared chats start late, stall or crash together.

The chat_runtime_sharing feature preview (Settings > Developer) places several dashboard chats on one agent process. Nothing measures it today. The only trace is a WARNING log line per placement, which carries a session key, cannot be aggregated across gateways, and says nothing about turn latency, KAS queueing, memory or crashes.

Why it matters

The default-on decision needs a fixed bar and data to hold it against. Without these counters the call rests on a few pod runs: a shared process that answers late, a KAS engine that still queues shared turns, or one crash that drops four chats would only show up as user reports after the default flips.

Not a goal

  • Changing what sharing does, its eligibility rules or its cap. The feature PR (fix(acp): share chats only on KAS engines that run them in parallel #36400) owns that; this PR does not touch its branch and merges before or after it.
  • A new pipeline or a new destination. Everything rides the existing recorder, local JSONL sink and product allowlist.
  • A dashboard page. The read recipe is below.

What changed (motivation -> approach -> change)

Goal -> count at the seams that already exist on main, so the numbers do not wait on #36400 -> eleven counters and gauges, all on closed labels.

  • Adoption. kirocrew.inventory.config.toggle gains key=preview_chat_runtime_sharing, the gauge every other switch reports adoption on. A live-config applier in config_watch.py counts each on/off flip from any writer (chat_runtime_share.toggled). Both read the preview through the existing chat_runtime_sharing_enabled, reached from application code via the new src/kiro_crew/agent_sdk/chat_runtime_sharing.py, so there is no second spelling of the preview id.
  • Placement. providers/chat_share.py counts placed{outcome=spawned|joined|refused, reason, tenants} beside its existing WARNING lines.
  • Effect. Two gauges in src/kiro_crew/metrics/chat_runtime_share.py read the ownership table (new RuntimeOwnership.keyed_lease_counts in src/kiro_crew/runtime_ownership.py): live processes placed by sharing and the chats on them; the difference is processes saved. They register from the new src/kiro_crew/metrics/shared_process_gauges.py, which also takes over the MCP gateway pool registration from src/kiro_crew/metrics/provider.py so that file stays under the 2,000-line cap. The RSS watchdog tick (src/kiro_crew/session_cleanup.py, src/kiro_crew/session_cleanup_leftovers.py, src/kiro_crew/session_cleanup_spare.py) counts each shared process's tree RSS once per tick, and counts a reclaim when it closes one to joins.
  • Lag and stalls. session_handle_turn.py counts prompt-written-to-first-event delay and how the turn ended, both split by the chat leases on the process (0 subagent/background, 1 alone, 2+ shared) and harness. The outcome and harness labels come from metrics.turns.turn_outcome and metrics.harness.harness_for_backend, the classifiers kirocrew.turn.completed uses. session_handle_kas.py counts KAS kind: queued notices (AgentExecutionWaitingForActiveSlot), which must stay near zero on 0.66.104. src/kiro_crew/runtime_death.py (announce) counts every death with the chat leases it held, read by identity (new RuntimeOwnership.held_leases_on_runtime) so a process the exit watcher already reaped still reports its blast radius.
  • Install. kas_engine_install.py counts download outcomes. On main a failed bundle install fails the spawn; there is no relay fallback to count.

Distributions are bucket-labelled counters because the product allowlist takes counters and gauges only. All names join PRODUCT_METRICS; metrics.md and the OTLP export guide list every row. No session key, pid, path, model id or user text reaches a label.

Backwards compatibility

Compatible: every change adds an emit after the step it measures; emit_counter swallows its own failures and the bucketing is total.
Removes nothing: no instrument, label value, config key or behaviour is taken away; the toggle gauge only gains one key value.

Tests

test/metrics/test_chat_runtime_share.py (23 tests) drives each production seam and asserts its counter: a founding and a joining placement, a refused start (and none with the preview off), the census gauges on a real meter (registration leases excluded), the RSS sample and a once-only reclaim, deaths (including a shared process already reaped, whose live lease count reads 0), one real turn through AcpSessionHandle.prompt on a shared and an unshared runtime (clock moved by the fake backend, no sleep), a KAS queued frame (and a subagent on its host chat's process reading 0), each install outcome, a preview flip through the registered applier, and one gauge set failing to register without costing the other. Refusal reasons are pinned by AST over chat_sharing_ineligible_reason; turn labels are checked equal to metrics.turns.turn_outcome and harness_for_backend.

Mutation check: 12 mutations, each replacing one emit call (the two failed install sites together), the census key filter, or the reaped-death lease read with a no-op; every one turned at least one test red.

Also run, serially: test_inventory_gauges.py, test_product_metrics.py, test_mcp_gateway_gauges.py, test_crew_log_health.py, test_chat_runtime_sharing.py, test_chat_runtime_sharing_cleanup.py, test_acp_session_handle_abandoned_turn.py, test_acp_runtime_death.py, test_runtime_death_is_a_process_event.py, test_kas_wire.py, test_config_live_server_appliers.py, test_session_rss.py, test_kas_route.py, both flake ratchets. isort, black, flake8, mypy, semgrep, the file line cap and the agent-sdk boundary gate pass.

Manual verification

N/A for UI: nothing renders. Read recipe for the week (OTLP backend, grouped by service.instance.id):

  • Adoption: gateways with inventory.config.toggle{key=preview_chat_runtime_sharing} = 1 on any day; toggled{state=off} / toggled{state=on} is the churn.
  • Placement: placed{outcome=joined} / (spawned + joined); refused by reason shows what keeps chats apart (kas_engine_not_shareable = engine too old or not installed).
  • Saved: tenants - processes, time-weighted per gateway; rss_sampled by tenants and rss for memory per shared process; reclaimed per shared process.
  • Lag: turn_started{harness=kas} p95 bucket for tenants=1 vs tenants>=2; turn_ended share of tool_stall, stale_recover, timeout per tenants bucket.
  • Queueing: kas_queued{tenants>=2} total.
  • Crashes: process_died{expected=false, tenants>=2} per 1,000 shared turns, and the tenants mix of those.

Related Issues

Closes #37187
Part of #17545

Other suggestions

Proposed default-on bar after 7 days:

  1. At least 10 gateways had the preview on for 3 or more days, with at least 500 joined placements in total.
  2. kas_queued{tenants>=2} is 0 on engines at 0.66.104 or later.
  3. The turn_started p95 bucket for KAS tenants>=2 is no more than one bucket above tenants=1, and its gt_30s share is within 1 point of it.
  4. The stall share (tool_stall + stale_recover + timeout) for tenants>=2 is within 0.5 points of tenants=1.
  5. Unexpected shared deaths are at most 1 per 1,000 shared turns, and their mean tenants is at most 2.5.
  6. reclaimed hit at most 2% of shared processes, and kas.engine_install{outcome=failed} is under 5% of attempts.

The census gauges publish only in a process that has placed or refused a chat with the preview on, so their absence elsewhere is expected, not a broken probe.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 568a60dd947c1ff1aecc86fdf91f09e715b5fc01 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 568a60dd947c1ff1aecc86fdf91f09e715b5fc01 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Additive telemetry on closed labels at seams that already exist; the backwards-compatibility claim holds against the code, so nothing blocks.

Notes

  • held_leases_on_runtime counts every lease that has the runtime as its key, registration leases included. keyed_lease_counts filters those out, so process_died{tenants} may overstate how many chats one crash took down compared with the census gauges. Use the same entry.key is not entry.runtime filter for both before criterion 5 of the default-on bar depends on it.

[DESIGN-REVIEWED] 568a60d

Description read

[DESCRIPTION-READ] f397bbc1d10f45cac0af38ea0d64fdf499aa115472775ffc4e284f4fe212f0ab

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 568a60dd947c1ff1aecc86fdf91f09e715b5fc01 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every series answers a question in the Goal, except two riders: the KAS engine install counter and the preview-flip counter. Both carry permanent allowlisted names with no Goal question behind them.

Notes

  • Item 9 — oversized: kirocrew.kas.engine_install measures bundle downloads, not sharing. The Goal asks "how many gateways use it, what it saves, and whether shared chats start late, stall or crash together". Subtraction: delete record_engine_install, its 4 call sites in kas_engine_install.py, the INSTALL_* tuples and the PRODUCT_METRICS row.
  • Item 2 — oversized: adoption is already inventory.config.toggle{key=preview_chat_runtime_sharing} (item 1). The toggled counter only adds churn, which the Goal never asks for. Subtraction: delete _count_chat_runtime_sharing_toggle, its live.subscribe, record_toggled, TOGGLED and its allowlist row.

What this change ships

Inventory (10 items) — 8 justified

Intent: give maintainers a week of data to decide whether the chat runtime sharing preview goes on by default. ADDITION.

  1. The adoption gauge reports the preview's on/off state — justified
  2. A counter for each preview on/off flip — oversized
  3. A counter for each chat placement (spawned/joined/refused, with reason) — justified
  4. Gauges for shared processes and the chats on them (processes saved) — justified
  5. Shared-process RSS sampled each watchdog tick, plus a counter when one is closed to new chats — justified
  6. Delay from prompt to first event, and how the turn ended, split by tenants — justified
  7. A counter for KAS queued notices — justified
  8. A counter for process deaths with the chats each one held — justified
  9. A counter for KAS engine install outcomes — oversized
  10. MCP gateway gauge registration moved into a new module for the line cap — justified

[FIRST-PRINCIPLES-REVIEWED] 568a60d

Description read

[DESCRIPTION-READ] 95cc4a4b1174a3feb43474cbe5f1337fcaf422f937b110cbb8d430d42c964d2e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 568a60dd947c1ff1aecc86fdf91f09e715b5fc01 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 568a60d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 568a60dd947c1ff1aecc86fdf91f09e715b5fc01 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 568a60dd947c1ff1aecc86fdf91f09e715b5fc01 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 568a60d

Verdict parsed from the review's SHA-scoped output markers for commit 568a60dd947c1ff1aecc86fdf91f09e715b5fc01.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 568a60dd947c1ff1aecc86fdf91f09e715b5fc01 false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 force-pushed the feat/acp-sharing-metrics branch from 9dc5839 to 6f4ab9c Compare November 14, 2026 11:06
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed in 6f4ab9c (findings on 9dc5839):

  • GPT, src/kiro_crew/runtime_death.py:137 (reaped shared crash counted as tenants=0): fixed. announce now counts the larger of the live lease count and the new RuntimeOwnership.held_leases_on_runtime, which reads leases by runtime identity without the liveness filter. The kill gate's live-only readers are unchanged. New test test_a_reaped_shared_crash_still_counts_the_chats_it_held drives a dead runtime holding 3 leases with leases=0 and expects tenants=3; reverting the read turns it red.
  • First Principles (second copies of the turn-outcome and harness classifiers): fixed. _STOP_TO_OUTCOME, TURN_OUTCOMES, BACKENDS and backend_label are gone; turn_ended uses metrics.turns.turn_outcome and both turn counters label harness via metrics.harness.harness_for_backend, as kirocrew.turn.completed does.
  • Backend Ratchets (provider.py over the 2,000-line cap): the MCP gateway and chat runtime sharing gauge registrations moved whole into metrics/shared_process_gauges.py; provider.py is 1,998 lines.

@chenmingwei23
chenmingwei23 force-pushed the feat/acp-sharing-metrics branch from 6f4ab9c to e1b5439 Compare November 14, 2026 11:25
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed in e1b5439 (First Principles on 6f4ab9c):

  • Blocker, second spelling of the preview read: fixed. preview_on, PREVIEW_ID and record_preview_change are gone. config_watch.py and the inventory gauge both call the existing acp.chat_runtime_sharing.chat_runtime_sharing_enabled. They reach it through a 20-line agent_sdk/chat_runtime_sharing.py, because application code may not import kiro_crew.acp directly (agent-sdk boundary gate).
  • Note, one-entry PREVIEW_TOGGLES: taken. It is now one read under PREVIEW_CHAT_RUNTIME_SHARING_KEY.
  • Note, engine install counter: kept. The default-on decision was asked to cover "bundle install fails, falls back". On main a failed install fails the spawn, so this counter is the only signal for it. It is one counter on closed labels.

The rate-limited lanes on 6f4ab9c (Dependency License Gate, Publish readiness signal, Analyze, GPT) re-run with this push.

@chenmingwei23
chenmingwei23 force-pushed the feat/acp-sharing-metrics branch 2 times, most recently from 70c8e08 to 4ac3173 Compare November 14, 2026 12:28
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Fixed in 4ac3173 (findings on 70c8e08):

  • Opus, src/kiro_crew/acp/session_handle_turn.py:878 (a subagent on its host chat's process was counted under the host's lease bucket): fixed. chat_leases_on now returns the lease count only when the turn's own session key holds a lease on that process (RuntimeOwnership.lease_keys_on_pid), otherwise 0. The KAS queued counter uses the same rule. A new parametrized case runs a subagent turn on a 2-lease process and expects tenants=0; dropping the key check turns it red.
  • Opus note, a watchdog-synthesized terminal counted as the turn's first event: fixed. turn_started skips synthetic_completion events, so a turn the backend never answered records no start.
  • GPT, src/kiro_crew/runtime_death.py:123 (downgraded on adjudication, non-blocking): left as is. It needs an acquire landing between a shared process's exit and its exit watcher's announce, and the only effect is one telemetry bucket. Fixing it would add snapshot state and an ordering contract to the ownership registry; the adjudication judged that disproportionate, and I agree.

@chenmingwei23
chenmingwei23 force-pushed the feat/acp-sharing-metrics branch from 4ac3173 to 568a60d Compare November 14, 2026 13:09
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 14% chance that this change introduces a defect, above the 12% limit for approving on its own.

This PR only adds telemetry: 11 counters and gauges with closed labels for the chat runtime sharing preview, placed at seams that already exist. I checked each emit site in the code. Every emit runs after the step it measures and cannot raise. The census excludes registration leases correctly (entry.key is entry.runtime). The RSS sample and reclaim are each counted once. Turn start skips fanned and synthetic events. The death count reads held leases by identity. No new import cycle appears, and the product allowlist and docs are updated. Only stale docstring or spec wording and a measurement-scope caveat remain, none of which blocks. CI was still pending.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 568a60dd94 · 1269 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 14, 2026
@chenmingwei23
chenmingwei23 enabled auto-merge (squash) November 14, 2026 14:23

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The review checked this head (568a60d) and found nothing blocking. Every new emit goes through emit_counter, which swallows errors. Every label is a closed tuple and no key, pid, path or prompt reaches one. The new modules import cleanly with no new import cycle. The two large files stay under the 2,000-line cap. The 23 new tests pass from an extracted copy of the head.
Done well: each call-site test drives the real production function, including a real AcpSessionHandle turn on an injected clock. The refusal reasons, turn outcomes and harness labels are pinned against their source classifiers, so a new branch reds a test instead of exporting other.
Optional:

  • The runtime_global and synthetic_completion start filters and the _yielded_terminal guard in session_handle_turn.py (around lines 922 and 989) have no mutation pin. Removing any one of them leaves all 23 tests green. One case with a co-tenant frame before the session's own chunk, plus one with two terminals, would pin them.
  • On main, product metrics go only to destinations the operator configures. Consider saying in the description where the week of cross-gateway data is read from, and which per-gateway key the recipe groups by there. The open #37183 drops service.instance.id.
  • turn_ended is recorded only at a yielded EVENT_COMPLETE, so a turn that ends by raising is never counted. A line in the metrics.md row saying so would keep its error share from being compared directly with kirocrew.turn.completed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure chat runtime sharing for a week before turning it on by default

2 participants