Repository navigation
feat(metrics): count session-control calls by op, caller kind and outcome - #22220
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check before merging: the Goal line never mentions the [DESIGN-REVIEWED] 03ebea1 Description read[DESCRIPTION-READ] 35befa57cc66ec2b40d37bd326c951a67b55ad5cd7483dc083bb7e6b9e62ef95 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The counter's op × outcome half repeats Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: let maintainers see which session-control verbs agents use and how often each fails. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 03ebea1 Description read[DESCRIPTION-READ] aa9e441c63bdfb045f3521219abde2e55a524b7f9143ffca4e86e03f499b6045 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Thanks for this. The branch has a merge conflict with Review of head
The linked issue is a tier:T3 issue, so this needs a design sign-off from a maintainer before merge. |
4f1914f to
ed94601
Compare
ed94601 to
a2ecae3
Compare
a2ecae3 to
a0ff631
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the rebase. Review of head a0ff6319ee57f7706041d5dea2f5da87dd7fc79b.
The merge conflict in docs/system-specs/modules/metrics.md is gone, and the branch is one commit on main.
Both should-fix items from the earlier review are addressed:
- Overlap with
kirocrew.gateway.request.duration: the description and the spec row now name that histogram, say that per-route volume and failures are readable there, and give the reasoncalleris a separate counter (adding it to the gateway histogram multiplies every route by sixteen). This answers the point. - "The RFC" in code comments: the comments now say this module defines the op list and no document on
mainspecifies it. One leftover is the test nametest_the_closed_list_is_the_rfcs_eighteenintest/metrics/test_session_control_calls.py. A rename such astest_the_closed_list_has_eighteen_opskeeps the name true. This is a nit and does not block.
No must-fix findings on this head. The _counted wrapper, the outcome rule and the caller classifier are unchanged in behavior from the earlier review.
One point is still open from the First Principles lane on this head: revive and history fold to op=other, so the caller axis, which the PR names as the reason for this counter, is lost for those two routes. Please either add them to the op list (the route test then counts each route by its own verb), or state in the description why counting them is not wanted.
CI note: the red Fast Gate comes from the harness parity x-capture check, which reports a stale allowlist entry. The same Fast Gate fails on main today, so this PR does not cause it. Coverage Gate and the cancelled lane follow from it.
The linked issue is a tier:T3 issue, so this needs a design sign-off from a maintainer before merge. I am leaving a comment, not an approval.
|
Answering the two advisory suggestions on The RFC is #22180, First Principles (CONCERNS, advisory) -- Item 3, fold Design Review (PASS, advisory) -- split Neither finding is blocking, neither is an over-count, a privacy leak or a performance problem, and no defect in the code this PR adds is left unfixed. Flagging both on #22180 rather than pushing a shape here, so this counter and the RFC that specifies it stay in agreement. |
a0ff631 to
d0ef91d
Compare
…come Add kirocrew.crew.session_control.calls, a counter with three closed attributes: op (the eighteen session-control tool verbs, anything else other), caller (the agent_kind enum, derived from OWNED_KIRO_AGENT_FILES) and outcome (ok / error). tool.call.duration cannot answer which session-control verbs agents use: it labels an MCP-served call mcp whatever kind it claimed, folding all eighteen verbs and every other server's tools into one series. The emit site is one wrapper per route, placed outside each handler's own internal-auth gate so a call refused before the handler body runs is still counted once, with outcome=error. It is the only emit site, so the deferred route binder cannot count a call twice. Classification reads live in-memory state only: no disk read lands on the request path.
d0ef91d to
03ebea1
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Incremental review of head 03ebea15fa822fa4c4d7da42ca99640515a02f52 against the previous head a0ff6319ee57f7706041d5dea2f5da87dd7fc79b.
The branch is rebased onto the current main and stays one commit. The only code change since the last review is how _caller_kind reaches caller_slot_key:
- The session-control handler module (
dashboard/handlers/session_control.py) now callssc.caller_slot_keythrough thesession_controlfacade instead of a local import fromcaller_access. The facade re-exports that name, and the module already imports the facade assc, so this matches the other calls in the file. test/metrics/test_session_control_calls.pypatchescaller_slot_keyon the same facade. The patch target and the call site now agree, so the history-key fallback test exercises the real lookup path.
No new findings on this change. The wrapper, the outcome rule and the caller classifier are unchanged in behavior.
Still open from the earlier review:
reviveandhistoryfold toop=other, so thecalleraxis is lost for those two routes. Please add them to the op list, or say in the description why counting them is not wanted.- Nit: the test name
test_the_closed_list_is_the_rfcs_eighteenstill names an RFC. A name such astest_the_closed_list_has_eighteen_opskeeps it true.
CI and the review lanes are still running on this head, so I make no claim about their results.
The linked issue is a tier:T3 issue, so this needs a design sign-off from a maintainer before merge. I am leaving a comment, not an approval.
buluoray
left a comment
There was a problem hiding this comment.
Reviewed head 03ebea1. Approve on the code; merging waits on the design decision for #22180.
All three attributes are closed: op is the 18 tool verbs plus other, caller is 16 values derived from OWNED_KIRO_AGENT_FILES plus default, crewmate and other, and outcome is ok or error. record_session_control_call folds each attribute at the emit site, so no agent name, session key or error string can reach the recorder. Mutating that fold or the classifier fallback turns test_an_out_of_set_attribute_is_folded_at_the_emit_site and test_an_unowned_agent_name_cannot_mint_a_series red. The _counted wrapper covers all 20 registered routes and re-raises handler exceptions untouched. The caller lookup reads only the in-memory slot registry and swallows its own failures, so a session-control call cannot fail because of the counter. The related tests pass on this head (382).
Follow-ups, not blocking:
- RFC 22180 section 6.1 still states 18 x 6 x 2 = 216 series, but the caller enum here has 16 values (608 series, as the spec row says). Please amend the RFC row so the two agree.
- The test name test_the_closed_list_is_the_rfcs_eighteen could become test_the_closed_list_has_eighteen_ops.
- When a caller closes or renames its own session, the slot can be gone by the time the wrapper classifies it, so caller reads other. Minor, but worth a sentence in the spec row.
Problem / Motivation
Goal: make it answerable which session-control operations agents actually reach for, and how often each one fails.
Session control is the set of MCP tools an agent uses to act on ANOTHER live conversation: open one, send into it, stop its turn, read its tail, adopt it, close it. People use the dashboard for all of that, so every caller of those routes is an agent -- and today nothing counts them.
kirocrew.tool.call.durationlooks like it should answer this and cannot. That instrument labels a call by its ACPtool_kind, and an MCP-served call is labelledmcpwhatever kind it reported. That is the right trade there (an MCP server's kind vocabulary is its own, and "this left the process over MCP" is the more useful fact for a latency histogram) and exactly the wrong grain here: all eighteen session-control verbs, every other installed server's tools, and their outcomes collapse into one series. So "do agents usefork?", "isbroadcastfailing?" and "how much of the crew's work is driven by a conductor versus a crewmate?" have no answer at all.Why it matters
Session control is the mechanism the whole multi-session product rests on -- a conductor dispatching workers, a crewmate patrolling what it created, a team lead stopping a wedged turn. Decisions about which of those verbs to improve, simplify or retire are currently made with no usage signal: an operation nobody calls and an operation that fails half the time look identical from outside. A failure rate per verb is also the cheapest early warning that one of these paths has regressed for agents, who cannot file a bug about it.
Not a goal
kirocrew.tool.call.duration's question and stays there.What changed (motivation → approach → change)
Not a product-shape change: no default moves, no user-facing capability is added, changed or removed, and no loop, monitor, agent or command behaves differently. The diff adds one counter and its documentation row.
What the existing instruments already answer, and the one thing they do not
The First Principles lane is right, and the description above understated it:
kirocrew.gateway.request.durationalready counts these routes, so "today nothing counts them" is wrong as written. Naming that instrument is the only honest way to say what is left.kirocrew.gateway.request.duration(metrics/http_metrics.py::REQUEST_METRIC) is recorded bymake_route_latency_middleware, mounted app-wide (server_runtime/middleware_chain.py:211,331), and labelledmethodxroute_templatexstatus_class.route_templateis the matched resource's canonical template, so each of these twenty paths gets its own label value, andstatus_classbuckets the answer to2xx..5xx. Per-route volume and per-route failures for/api/session-control/*are readable there today, and this counter does not replace that: for "how manyforkcalls, how many of them 4xx", the request histogram is the instrument.What neither instrument carries is who called. The request histogram has no caller-identity label, and that is not an oversight to correct in place: its label domain is deliberately bounded at
(len(known_templates) + 1) x 8 x 6, whereknown_templatesis every route the gateway registers. Adding a 16-value caller axis there would multiply the entire gateway's route table by sixteen in order to learn something about twenty paths. A separate counter scoped to those twenty paths is the smaller change, bounded at 608 series total.opandoutcomeare kept on this counter as the dimensionscalleris crossed with, not as a second way to count a request: "which verbs does a conductor drive, and which of them fail for it" is one question, and dropping either dimension leaves a caller count that cannot be attributed to an operation. That the Goal line above does not name the caller axis is a real gap between the Goal and the diff, and amending a frozen Goal is a maintainer's call, not this PR's -- it is flagged for the design sign-off the tier:T3 issue already requires rather than papered over here.Approach
Two sources were available: a counting fold over the caller's tool-call entries in the crew log, or the
/api/session-control/*handlers themselves. The handlers win because they are the operation's single commit point and they already hold both facts a fold would have to infer -- the outcome, and the calling session. A crew-log tool-call entry records that a tool was called, not what the route answered, so a fold would have to pair an entry with a later result to sayokorerror, and a pairing that can go wrong is a counter that can over-count. One wrapper per route cannot.What was built
src/kiro_crew/metrics/session_control_calls.pyholds the counter, the two classifiers and the emit. This module DEFINES the op list; no other file holds a copy of it, and no document onmainspecifies it.op-- the verb. Spelled as the TOOL's verb rather than the route's path segment, so/api/session-control/readisread_message(thesession_read_messagetool) and/set-modelisset_model: the question is which tool an agent reached for, and the tool verb is the stable name of the two.classify_opfolds an unlisted value tootherso a route added without a row cannot mint a series.reviveandhistoryare registered routes the list does not name, and both land inother-- their own per-route volume stays readable off the request histogram, which labels every route by its own template, so the fold loses nothing that is only available here.caller-- the kind of agent that called. Derived fromagent_files.OWNED_KIRO_AGENT_FILES(file stem minuskirocrew-,-rewritten to_) rather than tabulated by hand, so a new owned agent file joins the enum with no table to update and cannot fall silently intoother. A crewmate's pinned DM session is identified by its SLOT KEY's shape (members.is_member_session_key), because such a slot'sagentfield carries the crewmate's own crew name and could only ever answerother. An agent name outside the owned set -- a third-party agent, a user-authored template -- isother. A BLANK agent on a resolved slot isdefault, notother:_ChatSlottakesagent: str = ""and an ordinary dashboard chat is minted without one, so the blank is what the largest caller population carries, andcrew_log/emit_session.pyalready reads it that way (agent or _DEFAULT_AGENT). A caller whose slot could not be resolved AT ALL isother-- same blank agent, different answer, andclassify_caller'sslot_foundparameter is where that line is drawn.outcome--okbelow status 400,errorotherwise, anderrorfor a handler that raised.dashboard/handlers/session_control.py::_countedwraps each of the twenty registered handlers and is the only emit site, so the deferred route binder calling a handler cannot count the same call twice. It sits OUTSIDE each handler's own_require_internal/ member gate deliberately: a caller refused there made a session-control call that happened and failed, and a wrapper placed under the gate would not see it. A handler that raises is countederrorand its exception re-raised untouched, so this only ever ADDS the count -- the response, the status and the SEL audit line stay the handler's own.callerclassification reads LIVE STATE ONLY: the in-memory slot registry, viaget_slotfirst (the header is often already the slot key) andcaller_access.caller_slot_key's walk as the fallback for a caller authenticating as its history key. No file is read, written or stat'ed on a route an agent is waiting on, and an unresolvable caller isotherrather than a failure.telemetry.enabled: a disabled process holds a recorder with no meter and the emit returns before touching an instrument. A failing classifier or a failing recorder cannot fail a session-control call, and both are tested.On the
callerenum and the crew-log foldAn
agent_kindclassifier with the same vocabulary is being added to the crew-log usage fold with a hand-written stem-to-kind table. It is not onmain, so this PR derives the value instead of copying the table -- and the derivation produces exactly the same sixteen values.test_the_derived_set_equals_the_hand_written_agent_kind_tableasserts that against a literal set written out in the test, so the two cannot drift apart unnoticed, and one function can serve both callers once both have landed.Backwards compatibility
Compatible: the diff is additive (+933, -0). No route, signature, field, validator or status code changes; every handler keeps its name, its response and its refusals, and the wrapper is transparent to both the router and the handler-level tests that call these functions directly.
Removes nothing: no path is hidden, no record or row is deleted, no validator is tightened, no data is migrated or pruned, and no key or public function is removed or renamed.
Tests
test/metrics/test_session_control_calls.py(95 cases). The module docstring maps each guarantee to the one-line change that reddens its test, so a later reader can reproduce the proof rather than trust it.{op=<that op>, caller=..., outcome=ok}, driven end to end through the real_countedwrapper.other, including the two reachable ones (revive,history) and an invented verb._register_mcp_routes: each route is called with an unauthenticated request, its own gate refuses with 403, and the op is read off the resulting emit. So the assertion is about what the counter records, not about an annotation. A twenty-first route, or a decorator carrying the wrong verb, reds here.outcome=error; a handler that raises is oneerrorcount and the exception is re-raised unchanged; two calls are two counts.api_session_control_stopwith nointernal_authanswers 403 and emits one{op=stop, outcome=error}. This reds the moment the wrapper is moved under the auth gate.crewmateeven though itsagentis a crew name; an owned agent's slot reads its kind; an ordinary dashboard chat with a blank agent readsdefault; an unknown agent readsother; an unresolvable caller and an app with no state readotherwithout raising; a history key resolves through the slot walk.default, a blank agent with no slot isother, and a crewmate key wins over both with no slot at all.agent_kindtable literally; every owned agent file has a kind and none falls through toother; a hyphenated stem becomes an underscored kind; three unowned names cannot mint a series.{op, caller, outcome}and no attribute value contains any of them. The whole series space is asserted bounded.MetricsRecorder(None)process creates no instrument at all (_counters == {}), and a route still answers 200 with telemetry disabled.otherand the route still answers 200; a recorder that throws does not raise.Also run:
test/test_session_control_member_gate.py,test/test_session_control_parts_contract.py,test/test_session_control_doc.py-- all pass together with the new file.scripts/docs_lint.py,scripts/check_comment_history.py,scripts/check_brand_name.py,flake8,isortandblackare clean.Manual verification
N/A -- unit coverage is sufficient and more faithful here: the tests drive the real wrapper, the real classifiers and the real router, and the one thing a manual run adds (an actual exported data point) is the recorder's own contract, already covered by
test/metrics/test_recorder.py.Related Issues
Part of #21270.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
I confirm that this contribution is made under the terms of the Apache-2.0 license.