Skip to content

fix(metrics): split the sharing tenants bucket past 4 and count unshared turns' start - #37980

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/sharing-metrics-gaps
Nov 16, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/sharing-metrics-gaps

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the chat runtime sharing metrics able to answer two launch questions: how many chats a shared process really carries, and whether a turn starts later with sharing on than with it off.

Two gaps in kirocrew.chat_runtime_share.* (src/kiro_crew/metrics/chat_runtime_share.py, #37189), found in a pod run with a KAS lead and 8 goose / 8 deepseek child chats (#37979):

  1. tenants topped out at 4+. The default per-process cap (agent.chat_runtime_sharing_max_sessions) is 10, so a process with 8 chats read the same as one with 4.
  2. With sharing off (preview off, or that harness's own switch off), claude, goose and deepseek turns recorded no turn_started at all. Only KAS turns did.

Why it matters

These numbers decide whether sharing goes on by default. Without gap 1 nobody can see how full a process gets near the cap, or how many chats one crash takes down (process_died and rss_sampled use the same bucket). Without gap 2 there is no "off" arm for first-event delay, so "does a turn start later when it shares a process" cannot be read from one chart.

Not a goal

  • turn_ended on the AcpClient arm. It stays handle-arm only; see Other suggestions below.
  • Any change to placement, the per-process cap, or which harnesses share.
  • New metric names. Both changes are new closed values on existing counters.

What changed (motivation -> approach -> change)

Gap 1: tenants_bucket now returns 0 / 1 / 2 / 3 / 4-7 / 8-10 / 11+. 8-10 holds the default cap; 11+ is only reachable with a raised cap. The old 4+ is split into 4-7, 8-10 and 11+; from the release carrying this change, 4+ no longer appears.

Gap 2, symptom -> root cause: the counter was emitted only in acp/session_handle_turn.py::_run_turn, the AcpRuntime + AcpSessionHandle arm. claude, goose and deepseek (ACP_BACKENDS_SHARED_RUNTIME_ONLY_FOR_CHATS) take that arm only when a chat got a shared placement; otherwise providers/acp_lifecycle.py::start keeps them on AcpClient, whose stream_events emitted nothing. Now:

  • acp/client_turn.py::stream_events counts the same span (prompt written -> first non-synthetic event), shared=false. tenants is 1 for a session the gateway registered (a chat on any channel, a cron or task session) and 0 for an unregistered one (a subagent, a worker-pool client), the same split the handle arm gets from its registration lease. AcpClient holds no lease, so runtime_ownership.acquire_session_lease (the registry's own call in get_or_create) now marks its _registered_session slot where a runtime-backed session would take the lease (src/kiro_crew/acp/client_class.py).
  • turn_started gains a closed shared attribute (true / false). On the handle arm it is the new RuntimeOwnership.placed_by_sharing(runtime) (src/kiro_crew/runtime_ownership.py): a live runtime placed under a ChatRuntimeKey, the same rule the processes gauge uses. A KAS chat with sharing off holds only a registration lease keyed by the runtime itself, so it reads false.

turn_started{harness=goose} split by shared is then the on-vs-off comparison. Export allowlist (metrics/product_metrics.py): still a counter, every value closed, no key, pid or path added. docs/system-specs/modules/metrics.md and docs/guides/telemetry-otlp-export.md are updated.

Backwards compatibility

Breaking: the tenants value 4+ is no longer emitted on placed, rss_sampled, process_died, turn_started, turn_ended and kas_queued; counts from 4 up now read 4-7, 8-10 or 11+. Accepted: the preview has been out two days and no decision reads that series yet. turn_started gains a shared attribute, which a query that ignores it still sums the same. Writer sweep on origin/main @ 529da1e: git grep -n '"4+"' -- src test finds only chat_runtime_share.py (changed here) and an unrelated test/test_cli_doctor_install.py string.

Reader: src/kiro_crew/metrics/chat_runtime_share.py:tenants_bucket -- release -- the only producer of the value; every caller passes a count, none compares to 4+

Tests

  • test/metrics/test_chat_runtime_share.py:
    • test_bucket_edges_are_inclusive, new test_the_tenant_buckets_split_around_the_default_cap (reads the shipped cap from AgentConfig).
    • test_a_turn_counts_its_start_delay_and_how_it_ended asserts shared and has an 8-lease case reading 8-10.
    • new test_only_a_process_sharing_placed_reads_shared: a registration lease reads false, a keyed placement true.
    • new test_a_turn_on_a_process_of_its_own_is_counted_unshared: drives client_turn.stream_events for registered goose, deepseek and claude sessions (tenants=1) and an unregistered one (tenants=0); every case is red with the client_turn.py hunk reverted, and the registered cases are red with the acquire_session_lease hunk reverted.
    • new test_registration_marks_a_client_that_owns_its_process: a real AcpClient keyed on dashboard, Slack, Telegram and cron is marked by acquire_session_lease; a MagicMock provider is not. Red with that hunk reverted.
  • Determinism: the changed tests passed 20 of 20 runs at -n0, and the whole file in a shuffled order (seed 17589). The -n 4 repeat was not run (serial-only on this host).
  • Also run serially: test/metrics/test_product_metrics.py, test/test_acp_client_prompt_turn.py, test/test_acp_client_agent_started_turns.py, test/test_runtime_ownership.py, test/test_chat_runtime_sharing.py, test/test_acp_bindings.py, test/test_client_bindings.py, test/test_flake_pattern_ratchet.py; black, isort, flake8, mypy on the changed files; semgrep scan --config semgrep/ against the merge base; scripts/check_file_line_cap.py.

Proof: PROVEN test/metrics/test_chat_runtime_share.py::test_bucket_edges_are_inclusive, test/metrics/test_chat_runtime_share.py::test_the_tenant_buckets_split_around_the_default_cap, test/metrics/test_chat_runtime_share.py::test_a_turn_counts_its_start_delay_and_how_it_ended[2-dashboard:chat-0-1-12.0-end_turn-kas-want0], test/metrics/test_chat_runtime_share.py::test_a_turn_counts_its_start_delay_and_how_it_ended[8-dashboard:chat-0-1-0.5-end_turn-kas-want1], test/metrics/test_chat_runtime_share.py::test_a_turn_counts_its_start_delay_and_how_it_ended[2-subagent:sub-1-0.0-end_turn-kas-want2] (head ac9d556)

Manual verification

Pod run on head ac9d556 (systemd pod, loopback fake model, KAS lead + 8 goose children, goose switch on and then off, 2 rounds of 8 concurrent turns): switch on recorded turn_started{harness=goose, shared=true, tenants=8-10} = 16; switch off recorded turn_started{harness=goose, shared=false, tenants=1} = 16 (0 before this PR). Details in the PR comment with the pod totals.

Related Issues

Closes #37979

Sibling sweep

Swept: git grep -n "record_turn_started|_sent_at = time.monotonic" -- src/
Sibling: src/kiro_crew/acp/session_handle_turn.py -- changed here
Sibling: src/kiro_crew/acp/client_turn.py -- changed here

Pattern harvest

Not generalizable: a per-arm metric emit missing on one of two turn paths; the doc row that claimed "every ACP turn" is corrected, and the new test pins the second arm.

Other suggestions

  • turn_ended is still emitted only on the handle arm, so on the AcpClient arm turn_started has no matching turn_ended. Worth a follow-up if the outcome mix should be compared on vs off too.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5d008ea4f77a51da87bc7bd4edc1278680f0ada1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5d008ea4f77a51da87bc7bd4edc1278680f0ada1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Both gaps are fixed with new closed values on existing counters, and the only value removed (4+) is listed and accepted.

Notes

  • runtime_ownership.acquire_session_lease now also flips _registered_session on an AcpClient, a flag that only telemetry reads. That puts a metrics concern inside the lease owner. If more readers start depending on "registered", move the flag to the registry's get_or_create call site, or document it in runtime-ownership.md.
  • On the AcpClient arm, turn_started{shared=false} has no matching turn_ended. Until that follow-up lands, any on-vs-off ratio that divides ends by starts will be skewed toward the shared arm.

[DESIGN-REVIEWED] 5d008ea

Description read

[DESCRIPTION-READ] e5e416df30fbe9c8b12a2bb5c62421e89c78ef92bd685190da3f78110127146c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 5d008ea4f77a51da87bc7bd4edc1278680f0ada1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Each item answers one of the two launch questions in #37979. The per-client turn clock is a test-only seam that ships in production code.

Notes

  • Item 5 — oversized: self._turn_clock: Callable[[], float] = time.monotonic sits on AcpClient, but no production code ever sets it to anything else. I grepped _turn_clock under src/kiro_crew/acp: the default at client_class.py:391 and the read at client_turn.py:968, both the defining site. The handle arm (session_handle_turn.py) times the same span with plain time.monotonic(). Subtraction: delete _turn_clock and the getattr(..., None) or time.monotonic fallback. Call time.monotonic() as _run_turn does, and freeze it in the test at client_turn's own time binding.

What this change ships

Inventory (6 items) — 5 justified

Intent: let whoever decides whether sharing goes on by default see how full a shared process gets, and compare turn-start delay with sharing on and off. This is a FIX.

  1. The tenants count 4+ now reads 4-7 / 8-10 / 11+ on every counter that carries it — justified
  2. Chats that do not share (claude, goose, deepseek and other per-process harnesses) now count turn_started — justified
  3. turn_started gains a closed shared true/false attribute — justified
  4. Registering a session now marks a lone AcpClient as registered, so it reads tenants 1 and an unregistered client reads 0 — justified
  5. A replaceable turn clock on every AcpClient — oversized: no production writer
  6. New RuntimeOwnership.placed_by_sharing and on_shared_process reads — justified

[FIRST-PRINCIPLES-REVIEWED] 5d008ea

Description read

[DESCRIPTION-READ] 9fdfefed6d7579a5f24df0eba723554c589609dfad7f1966f914966f70b7d089

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5d008ea4f77a51da87bc7bd4edc1278680f0ada1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5d008ea

Verdict parsed from the review's SHA-scoped output markers for commit 5d008ea4f77a51da87bc7bd4edc1278680f0ada1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5d008ea4f77a51da87bc7bd4edc1278680f0ada1 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5d008ea4f77a51da87bc7bd4edc1278680f0ada1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5d008ea

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5d008ea4f77a51da87bc7bd4edc1278680f0ada1 false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 force-pushed the fix/sharing-metrics-gaps branch from 3813626 to 2c06427 Compare November 16, 2026 14:35
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for the Opus 5.5 BLOCKING finding on 3813626 (src/kiro_crew/acp/client_turn.py:958, tenants=1 for every AcpClient turn): fixed in 2c06427.

fixed: the AcpClient arm now reads tenants=1 only for a dashboard chat (telemetry_channel_of(session_key) == "dashboard") and 0 for any other session (subagent, cron, task), matching what the handle arm reads for a session without a chat lease. Pinned by test_a_turn_on_a_process_of_its_own_is_counted_unshared (subagent and cron cases read "0").
rebutted (the /compact part): providers/acp_turns.py sends /compact through self._client.stream_events on BOTH arms, so the handle arm (AcpSessionProvider.stream_events -> _run_turn) already counts it the same way. Dropping it only on the AcpClient arm would make the two arms disagree, which is what this PR is fixing.

@chenmingwei23
chenmingwei23 force-pushed the fix/sharing-metrics-gaps branch from 2c06427 to ac9d556 Compare November 16, 2026 14:59
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Disposition for the Opus 5.5 BLOCKING finding on 2c06427 (src/kiro_crew/acp/client_turn.py:80, a Slack/Discord/Telegram chat read tenants=0 on the AcpClient arm but 1 on the handle arm): fixed in ac9d556.

fixed: tenants on the AcpClient arm no longer reads the session key's channel. runtime_ownership.acquire_session_lease, the registry's own call in get_or_create, now marks an AcpClient's _registered_session slot where a runtime-backed session takes its lease, so every registered session (any channel, cron, task) reads 1 and an unregistered one (subagent, worker-pool client) reads 0 on both arms.
Pinned by test_registration_marks_a_client_that_owns_its_process (dashboard, Slack, Telegram, cron keys) and test_a_turn_on_a_process_of_its_own_is_counted_unshared; both red with the acquire_session_lease hunk reverted.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Pod run on head ac9d556 (systemd pod, loopback fake model for goose; a KAS lead opens 8 goose children with session_create; ACP runtime sharing and the session harness preview on; the goose switch on for one tree and off for another, pod restarted between; 2 rounds of all 8 children asking at once). Totals from the pod's local metrics JSONL, summed over DELTA exports:

switch on  (gateway A): turn_started {"delay": "le_1s", "harness": "goose", "shared": "true",  "tenants": "8-10"} = 16
switch on  (gateway A): placed goose spawned tenants=1 x1, joined tenants=2 x1, 3 x1, 4-7 x4, 8-10 x1
switch off (gateway B): turn_started {"delay": "le_1s", "harness": "goose", "shared": "false", "tenants": "1"}    = 16
switch off (gateway B): placed goose refused reason=harness_switched_off x8
  • 16 = 8 children x 2 rounds on both arms. Before this PR the off arm recorded 0 goose turn_started.
  • Off-arm goose chats read tenants=1, so the registration mark from acquire_session_lease reaches a real AcpClient through get_or_create.
  • 32 replies, no errors, no cross-chat tags; no Traceback or ERROR in the pod log.
  • An earlier run on 2c06427 gave the same counts.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 16, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/sharing-metrics-gaps branch 2 times, most recently from ffa4108 to 1deb8a7 Compare November 16, 2026 15:53
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 16, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/sharing-metrics-gaps branch from 1deb8a7 to ba77aa2 Compare November 16, 2026 16:53
…red turns' start

tenants_bucket now reads 0/1/2/3/4-7/8-10/11+ so a process at the default cap of 10 reads apart from one with 4. turn_started gains a closed shared=true/false attribute and is now also counted on AcpClient.stream_events, the arm a claude, goose or deepseek chat takes when it does not share, so the preview's on and off arms compare on one series.

Closes #37979
@chenmingwei23
chenmingwei23 force-pushed the fix/sharing-metrics-gaps branch from ba77aa2 to 5d008ea Compare November 16, 2026 17:07
@chenmingwei23
chenmingwei23 enabled auto-merge (squash) November 16, 2026 18:11
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Telemetry-only change: splits the tenants bucket past 4 into 4-7/8-10/11+ and adds turn_started on the AcpClient arm with a closed shared attribute. Both record_turn_started callers pass the new shared kwarg. The AcpSessionProvider path doesn't route through client_turn.stream_events, so no turn is counted twice. The new _registered_session slot has a single writer (acquire_session_lease, which only runs when no lease holder exists and the slot is exactly False) and a single reader. Nothing in the repo reads 4+, the export allowlist works per metric rather than per attribute, and emitting can't break a turn. The tests pin the new behaviour and CI is green.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 5d008ea4f7 · 272 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1: every tier gate came back clean and the automated review approved this head. Approved at the hand-over of PR review to the Code Change Approver (2026-11-16), for a head decided just before it took over.

@chenmingwei23
chenmingwei23 merged commit e1d7ff2 into main Nov 16, 2026
99 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/sharing-metrics-gaps branch November 16, 2026 19:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chat runtime sharing metrics: tenants tops out at 4+, and unshared claude/goose/deepseek turns record no turn_started

2 participants