Repository navigation
docs(rfc): crew usage metrics, wave 2 - #22180
chenmingwei23 wants to merge 2 commits into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this PR only adds a design document and its index row. It changes no code, config or rendered surface, and takes nothing away. [DESIGN-REVIEWED] d57b8eb Description read[DESCRIPTION-READ] c79fd774f77815919be505912d8173898896b6e38735e3224e1af47c67a04286 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
8a6ae75 to
dd9e49c
Compare
buluoray
left a comment
There was a problem hiding this comment.
Requesting changes on dd9e49c.
- Labels that are not listed. Section 9 says each enum is listed in this document, but three are not: autonudge.loops.stopped reason (L143, the fixed stop reasons), monitor.decisions kind (L144, the closed monitor kinds) and crew_log.fold.apply fold (L186, FOLD_NAMES). On main, autonudge stop reasons are store-sourced strings clipped to 200 characters (autonudge_stop_log.py safe_text) and monitor stop reasons can be up to 500 characters (monitoring/models.py:51), so the raw-text to closed-value mapping needs to be written down here.
- Cardinality math. agent_kind has 16 values in #21342 (13 agent-file stems plus default, crewmate, other), not 6. turn.started is at most 8 x 16 = 128 series (160 after the actor change), not 48 (L102), and session_control.calls is 18 x 16 x 2 = 576, not 216 (L132). Please restate the bounds and add one for fold.apply and loops.stopped.
- Failure isolation at commit points. Wave 2 records from request handlers and service code (session control, WebhookRunStore.record, the ledger handler, autonudge firing), not only through the bus, which already isolates subscriber errors. Please add a rule that a failure while recording is caught and never changes the outcome of the call it measures.
- Relation to the base RFC. L25 and L71 hand egress to rfc-telemetry-export-kiro-endpoint.md on a branch with no pull request. That document adds a default destination, depends on telemetry.enabled turning on by default and sends the sign-in credential, and L73 counts people at the endpoint. #21288 sections 8 and 9 and its question 3 say the opposite. Please link a pull request or remove the dependency, and say which RFC governs. Also note that wave 2 spends #21288 question 1 (category vs named list) and pulls in its S4 work ledger phase.
Follow-ups, not blocking: section 2 says depth is a histogram although #21288 and #21308 already moved it to a counter; L162 justifies commit-point counting by the existing work fold, while a new counting fold would not have the 32-item limit.
For maintainers: this RFC and the export-endpoint RFC need a ruling before the wave-2 code PRs can rely on it. Checks are green (84 pass, 16 skipped) and readiness passed.
|
Addressed the review in 8caf72a.
Follow-ups: section 2, section 5 and section 8 now say depth is already a counter. Section 6.3 now says a new counting fold would not hit the 32-item limit; the commit point is used because the |
|
Opus 5.5 finding (rfc-crew-usage-metrics.md:321, stop reason mapping): fixed in 9b05c72. Section 6.6 now says a reason with no table entry falls back to the row's terminal MonitorOutcome value when the monitor row records one (so a pull_request_merged stop reports success), and only a reason with neither reports other. The success / blocked / budget / target_unavailable row says the same. This matches the stop emitter in src/kiro_crew/metrics/monitor_loops.py. |
|
Rebased onto main (b700d47) again; the branch was conflicting. The only conflict was docs/request-for-change/README.md: kept main's text and re-applied this PR's one history sentence and its index row. docs/request-for-change/rfc-crew-usage-metrics.md is unchanged from 9b05c72, so both review-finding fixes carry over to head b8d7dd6. Still 2 commits. (An intermediate push, c37f903, left conflict markers in the README; b8d7dd6 removes them.) |
|
Rebased onto main (2dd19ba); the branch was conflicting. The only conflict was docs/request-for-change/README.md: kept main's text and re-applied this PR's one history sentence and its index row. docs/request-for-change/rfc-crew-usage-metrics.md is unchanged from b8d7dd6. New head b7dc496, still 2 commits. The Real Adapter Contract Tests job is expected to stay red until the main fix for issue #22958 lands. |
|
HOLD -- reviewed 8a1528f. The factual problem behind our r114 change request is fixed: the RFC is What still blocks:
What clears it: a maintainer ruling on both questions, plus a reply to each Design note. — crew-pr-reviewer (r119) |
|
Rebased onto current main; the branch was conflicting. The only conflict was docs/request-for-change/README.md: kept main's text and re-applied this PR's one history sentence and its index row. docs/request-for-change/rfc-crew-usage-metrics.md is byte-identical to 8a1528f. New head a1520ed, still 2 commits. Replies to the two Design Review notes on 8a1528f:
Also open for a maintainer: #21288 question 1 (rfc-crew-usage-metrics.md L363), whether approval is for a data category or for a named list of metrics. Wave 2 reads it as a category; if the answer is a named list, each phase returns for review with section 6.6 as its list. No design text changed in this push. |
|
Rebased onto current main (8edd26d); new head 5c99523, still 2 commits. Conflict was only in docs/request-for-change/README.md: main had added new index rows and edited the history paragraph. Main's text is kept as is; this PR re-adds only its one history sentence and its index row (now after the rfc-adaptive-warm-pool row at the end of the table). docs/request-for-change/rfc-crew-usage-metrics.md is byte-identical to the previous head a1520ed. docs_lint passes. |
|
HOLD at 5c99523. The rebase left the RFC byte-identical to a1520ed, and the r114 factual problem stays fixed: the RFC is What still blocks:
What clears it: a maintainer ruling on both questions, the count fix or a reply, and a rebase. — crew-pr-reviewer (r126) |
|
Rebased onto main (c7189a8). New head: cf5bb22, still 2 commits. Conflicts were only in docs/request-for-change/README.md: the history sentence on line 28 and the index table, where main had added the rfc-idle-compaction-before-cache-expiry and rfc-fold-delta-push rows at the same spot. Main's text is kept as is; this PR's one history sentence and its rfc-crew-usage-metrics row are re-applied after main's new rows. docs/request-for-change/rfc-crew-usage-metrics.md is byte-identical to the previous head 5c99523. |
|
Re-audited the RFC value lists and series bounds against main at c7189a8. New head: 9333b45, still 2 commits. This answers the Opus 5.5 finding (#22180 (comment)) and the count point in r126 (#22180 (comment)). What changed on main after the old audit point ba5a6fc:
New bounds in 6.1, 5 and 6.7:
The other value lists (actor, monitor stop reasons, monitor kinds, MonitorDecision) are unchanged at c7189a8. The two maintainer questions stay open as before: the type of |
|
HOLD at 9333b45. The r126 count point is answered: the RFC is re-audited at c7189a8, and What still blocks: this RFC asks for maintainer acceptance and defers two rulings to that maintainer:
Minor: the description's Manual verification still says claims were checked against What clears it: a maintainer ruling on both questions, recorded in the RFC or on this PR. — crew-pr-reviewer (r129) |
|
Rebased onto main f2ce16f (still 2 commits). The README conflict was resolved by keeping main's version and re-adding only this PR's index row and history line. One RFC change, from a re-audit at f2ce16f: main added a |
|
Hold at 6c9f19c. The RFC content stays sound: the f2ce16f re-audit adds the What blocks:
Non-blocking: the Opus notes on the 6.2 source wording and the missing What clears it: a rebase onto main, plus a maintainer ruling on both questions recorded in the RFC or on this PR. — crew-pr-reviewer (r133) |
|
Rebased onto main (76e470d); new head 445d860, still 2 commits. The only conflict was docs/request-for-change/README.md: kept main's text and re-added only this RFC's history sentence and index row. RFC changes, folded into the second commit:
The two maintainer questions (the |
|
Rebased onto current main; new head b59a567, still 2 commits. The only conflict was docs/request-for-change/README.md: kept main's text and re-added only this RFC's history sentence and index row. docs/request-for-change/rfc-crew-usage-metrics.md is byte-identical to 445d860. docs_lint passes. This RFC still waits on maintainer acceptance and the two deferred rulings (write.lost type, #21288 question 1). |
|
Opus 5.5 review finding (value lists and series bounds, rfc-crew-usage-metrics.md): fixed in c0164ff, checked against current main.
|
|
@iamwhatever rebased onto current main (f4369b0); new head 1fb3e92, still 2 commits, docs only. The only conflict was the long row-notes paragraph in docs/request-for-change/README.md; I kept main's text and our sentence. Re your 2026-10-20 change request, re-verified against current main:
The Design punchline (#21288 question 1) is left as written; it needs a maintainer ruling together with #21288. Could you re-review when you have a moment? |
Code change approver recommendation: A person should decideWhy: the decider estimates a 7% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 439 lines, more than the 400 it may approve on its own. This PR only changes docs. It adds an RFC with status An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
@iamwhatever rebased onto current main; new head 11929c0, still 2 commits, docs only. The only conflict was the row-notes paragraph in docs/request-for-change/README.md (kept main text and our sentence). No content change since the last reply. |
Propose counters and gauges for conductor use, session-tree depth and width, session control, monitor loops, the work ledger, event sources and crew log health. Counters and gauges only, closed-enum attributes, no ids. Refs #21270.
…lation, egress - List the agent_kind, actor, stop reason (raw text to closed value), monitor kind and fold enums, copied from the wave-2 code. - Restate cardinality with the 16-value agent_kind and add a bounds table. - Add rule 8: a failure while recording at a commit point never changes the outcome of the measured call. - Drop the export-endpoint dependency; #21288 sections 8 and 9 govern egress and this RFC adds no outbound path. Note wave 2 relies on #21288 question 1 and pulls in its S4 phase. - Depth is already a counter; restate why ledger counters use the commit point rather than a new counting fold.
|
GPT 6.1 finding (rfc-crew-usage-metrics.md:35 and :408, telemetry.enabled "off by default"): fixed in d57b8eb. Checked against main (TelemetryConfig in config/sections_telemetry.py, outbound_consented in metrics/consent_pins.py): local recording is on by default (a local JSONL sink, no network egress), and nothing leaves the machine unless a collector is configured (otlp_endpoint or otlp_endpoints) and the user has opted in to recording themselves (telemetry.enabled: true written in a config file, or KIROCREW_TELEMETRY truthy). Both sites now say that, and so does the same claim in the Non-goals egress bullet (it said "opt-in through telemetry.enabled"). No other off-by-default or opt-in claim about recording is left in the RFC. Docs only; head is d57b8eb, still 2 commits. |
|
Closing: this design is outdated and no longer needed. |
Problem / Motivation
Goal: Give maintainers a design to review for the next set of crew usage metrics, before any implementation PR lands.
We cannot answer basic questions about how the crew is used: how many workers a conductor runs, how wide and deep session trees get, how agents drive each other through session control, whether monitor loops do real work or tick quietly, how work ledger items end, which event sources wake sessions, and whether the crew log drops data or folds slowly. The first export round (#21342, #21308) counts sessions and turns only.
Why it matters
Without these numbers, decisions about the conductor, monitors and the work ledger rest on guesses. Regressions that people do not report (failed sub-agents, quiet monitor ticks, slow folds) stay invisible.
Not a goal
What changed (motivation -> approach -> change)
Adds
docs/request-for-change/rfc-crew-usage-metrics.mdand its index row indocs/request-for-change/README.md.Approach: counters and gauges only; every attribute is a closed enum listed in the document; a duration or size is a counter with a small
bucketattribute so the distribution survives backends that do not keep explicit histogram buckets; "how many right now" is a gauge read from live state, never an up/down counter. Sources prefer small counting folds on the crew log (the model of #21288), otherwise the single commit point of the feature. The first crew log health batch adds no work to the append thread.It also records two changes to the open first-round PRs:
crew.turn.startedgainsagent_kind, and a newcrew.session.tree.childcounter gives tree width.crew.session.tree.depthis unchanged: it is already a counter with a closeddepthlabel.Backwards compatibility
Compatible: documentation only; no code, config or behaviour changes.
Removes nothing: the diff adds one document and one index row.
Tests
N/A -- documentation only.
python3 scripts/docs_lint.pypasses locally.Manual verification
Every "today" claim in section 2 and the README row was checked against
76e470d39ewithgit grep.Related Issues
Refs #21270. Builds on #21288.
Checklist
Contribution License Agreement
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.