Repository navigation
feat(metrics): label sessions and turns with the harness that ran them - #22918
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every item traces back to the Goal. The one overlap, a second backend vocabulary, is already deferred to #24883. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: let operators count installs, sessions and turns per agent harness. ADDITION.
[FIRST-PRINCIPLES-REVIEWED] ab38cb5 Description read[DESCRIPTION-READ] 1fe71725782db625334d45c5be5336c26e437e3031dbcd99a7b1bad6cfc6713b That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: both labels use one vocabulary built from Notes
[DESIGN-REVIEWED] ab38cb5 Description read[DESCRIPTION-READ] 1b77e4cccaebec290193fadc8d4c43692a7686db70e4b9667a3c3b6cc49cfc0c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
bolichen97
left a comment
There was a problem hiding this comment.
Looks good to me on the code. Head reviewed: 44f2c3b. No must-fix or should-fix findings.
What I checked on this head:
- The value set in
metrics/harness.pyis derived fromACP_BACKENDS_KNOWNin config spelling plusclaude_code,unknownandother, so both attributes stay closed._closed_harnessinmetrics/turns.pyfolds any caller value outside the set toother, and an empty value tounknown. providers/acp.py::provider_backendis a pure extraction of the shape checkprovider_labelused.provider_labelreturns the same answers, and a foreign shape now returnsNoneinstead of the kiro-cli empty id.- All four
record_session_startedcall sites pass the provider that the registry entry was built around. In each site that name is in scope (open_task_session,get_or_create, the background runtime, andsession.provideron the compaction keep-alive path). - Every
persist_token_record_asynccaller that passesmodel_sourcepasses the live provider or client, soharness_of(model_source)reads the effective backend. A caller with nomodel_sourcerecordsunknown, as documented. watch_config()subscribesagent.acp_backendandagent.providerthroughlive.subscribe, which accepts several prefixes and refuses a restart-marked path at registration. Neither path is restart-marked on this head (requires_restartreturns False for both), so the subscription does not raise.- I ran
test/metrics/test_harness_label.pyand the three touched test files on this head: 398 passed. - The resource attribute is read from the same config load as the
telemetrysection. The point attribute adds a cost of one or two attribute reads on the turn path, with no disk, network or lock. - The owning spec (
docs/system-specs/modules/metrics.md) is updated in the same commit.
What remains before approval (maintainer checks, not code findings):
- PR Readiness is still running on this head. The AI review lanes have not posted yet.
- This PR is part of a parent issue that needs a design sign-off from a maintainer. The tracking design document for that issue is still a draft. A maintainer should confirm that the
harnessattribute is in the agreed scope.
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
44f2c3b to
3a83863
Compare
|
Review findings on 44f2c3b, answered on 3a83863:
|
|
Notes on 3a83863 (all non-blocking), answered without a code change:
|
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the fix round. Head reviewed: 3a83863.
The background one-liner fix holds up on this head. provider_backend now reads a bare AcpSessionHandle (through its runtime) and an AcpRuntime, _ProviderBgSession exposes a read-only provider, and harness_of unwraps it one level only. provider_label still goes through _provider_shape_backend alone, so the session map label does not move, and an AcpProvider with no client now reads unknown instead of kiro. The new tests pin all three shapes.
Two required test jobs fail on this head, on both Linux and Windows, and both failures come from this PR:
- Backend Tests shard 1:
test/test_services_config_hot_reload.py::TestTelemetryRecorderRebuild::test_watch_config_subscribes_on_telemetryasserts_one_sub("telemetry").prefixes == ("telemetry",).watch_config()now subscribes("telemetry", "agent.acp_backend", "agent.provider"), so the assertion fails. That test file is not in this PR, so I cannot anchor an inline comment there. Suggestion: update the expected tuple to the three prefixes (ideally read from_RESOURCE_CONFIG_PATHS), since the wider subscription is intended. - Backend Tests shard 3:
test/test_source_pin_budget.pyreportstest/metrics/test_harness_label.pyreading the dashboard turn source once against a budget of 0. See the inline comment.
Approval is held for a maintainer either way: the parent issue needs a design sign-off on scope (whether the harness attribute is in the agreed scope), and PR Readiness is still running on this head.
What I checked: the incremental change since 44f2c3b, all five AI review comments on this head (Design, First Principles, Opus and GPT pass, UX skipped), the author replies to their notes, and the failing job logs.
|
|
||
| from kiro_crew.dashboard import chat_runner | ||
|
|
||
| src = inspect.getsource(chat_runner) |
There was a problem hiding this comment.
🔴 This source read trips the source pin budget gate
test/test_source_pin_budget.py counts every inspect.getsource of the dashboard turn modules and gives this new module a budget of 0, so Backend Tests shard 3 fails on Linux and Windows:
src = inspect.getsource(chat_runner)
assert "harness=harness_of(client)" in srcIt also pins the exact spelling of the call, so a harmless rename breaks it while a wrong argument with the same text would still pass.
Suggestion: assert on behaviour instead. For example, run a turn through test/turn_harness.py's run_turn with a client whose backend is known, patch chat_runner._emit_turn_metric (or the recorder) to capture its kwargs, and assert harness equals the expected value. Then drop this test.
Rule: test/test_source_pin_budget.py budget for dashboard turn source reads.
There was a problem hiding this comment.
Replaced on c0d7447: the source read is removed, and test_a_dashboard_turn_labels_its_duration_with_the_turn_clients_harness now runs a real turn through turn_harness.run_turn and asserts the emitted kirocrew.turn.duration carries harness=codex for a Codex-backed client and unknown for a plain one. test_source_pin_budget.py passes.
3a83863 to
947452f
Compare
947452f to
c0d7447
Compare
|
Both fixed; now on c0d7447 (rebased on main ba53277).
Locally (-n 2): |
be3aea1 to
033f243
Compare
|
Reply to r129. New head: 033f243 (still one commit; same base). Item 8 ( Item 9: fixed.
Item 1 (scope sign-off) is unchanged and still waits on a maintainer per #21270. Related tests run locally: |
033f243 to
86cff1d
Compare
|
Head is now 86cff1d (still one commit). The Opus 5.5 lane caught that the pinned signature of |
buluoray
left a comment
There was a problem hiding this comment.
Approving 86cff1d.
The harness label is derived positively from the backend id (harness_for_backend, metrics/harness.py:71-84). The Claude Code seam takes precedence, unresolved maps to unknown, and anything outside ACP_BACKENDS_KNOWN folds to other. Both attributes therefore stay in a closed set that grows only with the backend table. On current main that set already includes copilot. All four record_session_started sites pass the provider the session was built on, and both turn-duration callers resolve the label with harness_of(...) from the live client or model_source. The change is additive: no attribute is renamed, so existing queries keep working, and the series split on a harness switch is documented.
The previous head's red base-surface pin (harness: 'str' = '' against a default of HARNESS_UNKNOWN) is fixed here, and nothing else changed in the push. Merged onto current main, the metrics suite plus the touched and related tests pass (2057 passed, 3 skipped). Removing the seam check, the other fallback, the resource attribute, the session-started label, the wider config subscription or the one-level unwrap each turns a named test red.
Non-blocking follow-ups:
docs/system-specs/modules/metrics.md:388lists the backend values by hand and leaves outcopilot, and line 459 says "11 values", which is 12 on current main. Consider pointing atHARNESS_VALUESinstead of enumerating.- With
_closed_harnessgone, the closed-set guarantee onkirocrew.turn.durationrests on callers usingharness_of. A test that asserts everyemit_turn_duration(harness=...)call site passesharness_of(...)would keep that from regressing.
This still needs the maintainer telemetry-scope confirmation that #21270 calls for before merge.
86cff1d to
a099ef8
Compare
|
Head is now a099ef8 (still one commit). It fixes the GPT 6.1 finding on 86cff1d (gatewayd keeps the previous harness after a live switch). The finding is real: the MCP gateway daemon registers no Fix, in the existing recheck path:
The reply above for items 8 and 9 is unchanged. |
|
Head is now 370b930 (still one commit). It fixes the Opus 5.5 finding on a099ef8: main added |
a099ef8 to
370b930
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed head 370b930 against our last review at be3aea1. Same base, still one commit. No blocking findings.
What changed since be3aea1, and what I checked:
- Consent recheck now relabels a process with no config watcher.
_build_recorderrecords the rawagent.acp_backend/agent.providerpair on every return path that has a config,_install_lockedkeeps it,_take_provider_lockedclears it, and_consent_workerrebuilds a live recorder when the pair on disk differs. A build that failed after consent (no OpenTelemetry, init error) still carries the key, so an unchanged config does not rebuild in a loop. A disabled recorder never rebuilds for a harness move. The new test covers unchanged config, a backend switch and a provider switch. _closed_harnessis gone andemit_turn_durationwritesharnessas given, defaulting tounknown. I grepped every caller insrc/at this head: the two production paths passharness_of(...), which maps every input intoHARNESS_VALUES, so the set stays closed.configured_harnessfolded intoset_configured, andRESOURCE_ATTR_HARNESSnow has one home inmetrics/harness.py. The pinned base surface signature matches the newunknowndefault.- The spec no longer hand-lists the value set or its size, so a new backend cannot make it stale.
Not approving yet: the existing request for changes stands on the telemetry data scope, which needs a maintainer to confirm kirocrew.harness and the harness point attribute, and several required checks and the readiness signal were still running at review time. From the code side this looks good to ship once those clear.
|
Holding on 370b930. I found no code defect; the hold is a maintainer decision plus one unanswered note.
The delta since be3aea1 reads sound: the consent recheck relabels the gateway daemon after a harness switch (provider.py#L828-L841), both production callers pass — crew-pr-reviewer (r131) |
Nothing in the metrics could say which agent harness people run. The `provider` attribute on the turn family reads `acp` for kiro-cli, Codex and most other harnesses alike, and no metric read `acp_backend`. Two readings, one closed vocabulary owned by `metrics/harness.py`: * `kirocrew.harness`, a resource attribute on every exported series: the install's configured default (global `agent.acp_backend`, or `claude_code` on that provider seam). Counts installs per harness. * `harness`, a point attribute on `kirocrew.session.started` and `kirocrew.turn.duration`: the session's effective harness, read off the live provider, so a per-session or per-agent override counts as its own harness. The value set is derived from `ACP_BACKENDS_KNOWN` in its config spelling (kiro-cli is `kiro`) plus `claude_code`, `unknown` and `other`; an id outside the set maps to `other`. The provider shape check is shared with the session-map label: `providers/acp.py::_provider_backend` (private) is factored out of `provider_label` and reached through the agent SDK surface. The recorder reads the resource value from the same config load as the `telemetry` section, and the config watcher now also rebuilds on `agent.acp_backend` and `agent.provider`, so a changed default relabels the next export. The consent recheck compares the same pair, so a process with no config watcher (the MCP gateway daemon) relabels too. Not here: `harness` on `kirocrew.session.duration` and the `turn.tokens` / `credits` / `cost_usd` family.
370b930 to
ab38cb5
Compare
|
Re r131, First Principles item 7 ( Agreed. Head ab38cb5 drops Item 1 (scope sign-off) is unchanged and still waits on a maintainer. |
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed at ab38cb5. The base is unchanged from 370b930 and the only change removes HARNESS_VALUES from __all__ in metrics/harness.py. The constant stays as a module-level name, the test reads it as h.HARNESS_VALUES, and nothing imports it through __all__ or a star import, so this closes the First Principles item 7 note with no behavior change.
No new findings. From my side this is still good to ship. Approval waits on the open maintainer scope sign-off for the telemetry data in #21270 (the remaining hold on this PR) and on the readiness check finishing on this head.
|
HOLD at The r131 note is fixed: What holds it: #21270 keeps its child PRs unmerged until a maintainer confirms the data scope. Nobody has confirmed the — crew-pr-reviewer (r133) |
Problem / Motivation
Goal: Metrics can say which agent harness an install chose, and how many sessions and turns each harness ran.
Nothing in the metrics names the harness today. The turn family carries
provider, but that value isacpfor kiro-cli, Codex, OpenCode and most other harnesses alike, so it cannot answer "how many people run Claude Code" or "how many turns ran on KAS". No metric readsagent.acp_backend.Why it matters
Harness choice drives which adapters, install paths and parity work get attention. Without a label, every harness question has to be answered from anecdote, and a regression that hits one harness is averaged into the others.
Not a goal
harnessonkirocrew.session.durationand on the usage family (kirocrew.turn.tokens,kirocrew.turn.credits,kirocrew.turn.cost_usd). Tracked in Metrics: add the harness attribute to session.duration and the turn usage metrics (follow-up to #22918) #24882; this PR adds the point attribute to the two basic counts only.backendattribute withharness_for_backend. Tracked in Metrics: unify the session-startup backend attribute with harness_for_backend (follow-up to #22918) #24883.harnesson process-level metrics (gateway boot, HTTP requests, MCP pool, crew log health, inventory gauges). They have no session, so they carry only the resource attribute.What changed (motivation -> approach -> change)
Goal: count installs per harness, and sessions and turns per harness.
Approach: one closed vocabulary, two readings.
kirocrew.harnessagent.acp_backend, orclaude_codewhenagent.providerisclaude_code)harnesskirocrew.session.startedandkirocrew.turn.durationThe resource attribute is the cheapest way to count installs: it rides every point already and adds no series inside a process. The point attribute is needed because an install whose default is KAS can still run a member on Codex.
What I built:
src/kiro_crew/metrics/harness.pyowns the value set. It is DERIVED fromagent_sdk/backends.py::ACP_BACKENDS_KNOWNin the config spelling (backend_to_stored, so kiro-cli iskiro), plusclaude_code,unknown(nothing to resolve at the emit point) andother(an id outside the known set). A harness added toACP_BACKENDS_KNOWNjoins the set with no edit here. Same idea as the fold labels in feat(metrics): count crew log losses and folds without touching the append path #22226, which are derived from their catalogue.providers/acp.py::_provider_backend(private;llm_helpers.provider_backendstays the one public reader) is factored out ofprovider_label(same behaviour) and reached throughagent_sdk/drivers/acp.py::_provider_backend, so the metrics module does not import the providers layer. A shape that carries no backend (including anAcpProviderwith no client yet) isunknown, neverkiro(kiro-cli's runtime id is the empty string, so a blind default would read every miss as kiro).get_bg_session()handle rather than a provider asmodel_source._provider_backendalso reads a bareAcpSessionHandle(its runtime's backend) and anAcpRuntime, and_ProviderBgSessiongains a read-onlyproviderproperty thatharness_ofunwraps one level.provider_labelkeeps reading only the two provider shapes, so the session map and usage-rowprovidervalues are unchanged.kirocrew.session.started:record_session_started(key, provider)now takes the provider the registry entry was built around, from all four call sites (cold start, resume, background, compaction keep-alive).kirocrew.turn.duration: both emit owners pass it.persist_token_record_asyncreads it offmodel_source(the live provider every surface already passes to name the model);_run_chatreads it off the turn'sclient. The value is always sent; an empty or out-of-set value becomesunknown/other._build_recorder()from the SAME config load as thetelemetrysection, and placed besidekirocrew.process.rolein_resource_attributes().watch_config()now subscribes the existing rebuild toagent.acp_backendandagent.provideras well astelemetry. A change to either drops the recorder and the next metric rebuilds with the new label. Both paths are live settings (not restart-marked). This is the gateway's watcher; a short-lived process keeps the value it booted with. Downstream, the switch reads as a new series set from the change on.docs/system-specs/modules/metrics.md: akirocrew.harnessrow in the resource-attribute table, a "Theharnesslabel" section (values, refresh, series bound), and updated rows for both metrics.Series bound, measured against the code rather than multiplied out:
kirocrew.session.started: at most x11, in practice the number of harnesses one install runs, usually 1 or 2 (the default plus a member override).kirocrew.turn.duration: close to x1. Each harness serves its own model ids (capabilities_for(backend).model_id_namespace), somodelalready mostly determinesharness. Growth is bounded by the (model, harness) pairs an install actually runs, not by the product of the two sets. The overlap is kiro-cli and KAS, which share theacpnamespace; there the label is what separates them, which is the point.Metrics RFC rules (#22180 section 4): no new instrument and no new histogram (the attribute rides the existing counter and the existing turn histogram); every value is from a closed enum with
other; no CUMULATIVE series added; recording stays an in-memory add (the provider read is two attribute reads, no disk, network or log on the turn path); behindtelemetry.enabledand the existing attribute sanitiser. Rule 6 (source preference): this adds no new count, so there is nothing to fold over crew_log entries; the attribute is attached at each metric's existing single commit point (the registry insert forsession.started, the turn's EVENT_COMPLETE emit forturn.duration), after the write it reports.Backwards compatibility
Compatible: every existing attribute and metric name is unchanged; a dashboard query that does not group by
harnesssums over it and reads the same totals.record_session_startedgains an optional positionalproviderand_emit_turn_metrican optionalharnesskeyword, both with defaults.Removes nothing: the change only adds a resource attribute and one point attribute on two metrics;
provider_labelkeeps its exact answers (its shape check moved into_provider_shape_backend, which it now calls).Tests
test/metrics/test_harness_label.py(new):ACP_BACKENDS_KNOWNin config spelling plusclaude_code/unknown/other; every known backend maps to its config word; an unknown backend (gemini, wrong case, the config wordkiroused as a runtime id, a non-string) becomesother;Noneisunknown; the Claude Code seam wins.AcpProvider, a started one (client swapped forAcpSessionProvider) and a bareAcpSessionProvider; a foreign shape, aMagicMockor anAcpProviderwith no client isunknown, neverkiro; an unregistered backend on a live provider isother.AcpSessionHandle, a_ProviderBgSessionand anAcpRuntimeresolve to their backend;kirocrew.turn.durationthroughpersist_token_record_asyncwith either background handle asmodel_sourcereadskas;provider_labelstill answers the default for both handles.kirocrew.session.startedfrom a Codex session readscodex;kirocrew.turn.durationthroughpersist_token_record_asyncwith a Codexmodel_sourcereadscodex(and the usage family carries noharness); the dashboard owner emitsclaude,unknown,otherfor a claude client, no harness, and an out-of-set value.kirocrew.harness=kas; after the config applier runs on a changed default, the rebuilt recorder exportskiro, and an unregistered default exportsother;watch_config()subscribes toagent.acp_backendandagent.provider.test/test_chat_runner_base_surface.py: the pinned_emit_turn_metricsignature gainsharness.test/test_compaction_failure_pause.py,test/test_compaction_stop_declined.py: theirrecord_session_startedstubs accept the provider argument.test/test_services_config_hot_reload.py: the telemetry applier's pinned prefixes now includeagent.acp_backendandagent.provider.Run locally (
-n 2):test/metrics/test_harness_label.py,test_session_duration.py,test_turn_metric.py,test_turn_profile.py,test_resource_attrs.py,test_provider_bucket_views.py,test_telemetry_handler.py,test_otlp_wire_e2e.py,test_crew_log_health.py,test_telemetry_column_taxonomy.py,test/test_acp_backend_kas.py,test_acp_provider.py,test_agent_sdk_capabilities.py,test_agent_sdk_boundary.py,test_harness_parity.py,test_chat_runner_base_surface.py,test_chat_runner_composition_contract.py,test_dashboard_chat_turn_wiring.py,test_background_turn_accounting.py,test_session_allocation_bindings.py,test_compaction_*above,test_services_config_hot_reload.py,test_source_pin_budget.py. All pass excepttest_resource_attrs.py::test_a_failed_mint_costs_only_the_attribute, which fails the same way on unmodified main in my local environment (the SDK there does not substitute aservice.instance.id); CI is the reference for it.scripts/check_agent_sdk_boundary.py,scripts/check_harness_parity.py,scripts/check_file_line_cap.pyandscripts/docs_lint.pypass.Manual verification
N/A - the resource tests build a real recorder and read the exported JSONL shard, which is the same resource the OTLP reader sends; the point-attribute tests drive the real emit functions.
Related Issues
Part of #21270.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.