Skip to content

feat(metrics): label sessions and turns with the harness that ran them - #22918

Merged
iamwhatever merged 1 commit into
mainfrom
feat/metrics-harness-label
Oct 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/metrics-harness-label

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Metrics can say which agent harness an install chose, and how many sessions and turns each harness ran.

Nothing in the metrics names the harness today. The turn family carries provider, but that value is acp for kiro-cli, Codex, OpenCode and most other harnesses alike, so it cannot answer "how many people run Claude Code" or "how many turns ran on KAS". No metric reads agent.acp_backend.

Why it matters

Harness choice drives which adapters, install paths and parity work get attention. Without a label, every harness question has to be answered from anecdote, and a regression that hits one harness is averaged into the others.

Not a goal

What changed (motivation -> approach -> change)

Goal: count installs per harness, and sessions and turns per harness.

Approach: one closed vocabulary, two readings.

Reading Where Value Answers
kirocrew.harness resource attribute, every series the install's configured default (global agent.acp_backend, or claude_code when agent.provider is claude_code) installs per harness
harness point attribute on kirocrew.session.started and kirocrew.turn.duration the session's EFFECTIVE harness, per-session and per-agent overrides included sessions and turns per harness

The resource attribute is the cheapest way to count installs: it rides every point already and adds no series inside a process. The point attribute is needed because an install whose default is KAS can still run a member on Codex.

What I built:

  • src/kiro_crew/metrics/harness.py owns the value set. It is DERIVED from agent_sdk/backends.py::ACP_BACKENDS_KNOWN in the config spelling (backend_to_stored, so kiro-cli is kiro), plus claude_code, unknown (nothing to resolve at the emit point) and other (an id outside the known set). A harness added to ACP_BACKENDS_KNOWN joins the set with no edit here. Same idea as the fold labels in feat(metrics): count crew log losses and folds without touching the append path #22226, which are derived from their catalogue.
  • The effective harness is read off the live provider at the emit point, through the shape check the session-map label already uses. providers/acp.py::_provider_backend (private; llm_helpers.provider_backend stays the one public reader) is factored out of provider_label (same behaviour) and reached through agent_sdk/drivers/acp.py::_provider_backend, so the metrics module does not import the providers layer. A shape that carries no backend (including an AcpProvider with no client yet) is unknown, never kiro (kiro-cli's runtime id is the empty string, so a blind default would read every miss as kiro).
  • Background one-liners (titles, summaries, tips) pass the get_bg_session() handle rather than a provider as model_source. _provider_backend also reads a bare AcpSessionHandle (its runtime's backend) and an AcpRuntime, and _ProviderBgSession gains a read-only provider property that harness_of unwraps one level. provider_label keeps reading only the two provider shapes, so the session map and usage-row provider values are unchanged.
  • kirocrew.session.started: record_session_started(key, provider) now takes the provider the registry entry was built around, from all four call sites (cold start, resume, background, compaction keep-alive).
  • kirocrew.turn.duration: both emit owners pass it. persist_token_record_async reads it off model_source (the live provider every surface already passes to name the model); _run_chat reads it off the turn's client. The value is always sent; an empty or out-of-set value becomes unknown / other.
  • The resource value is read in _build_recorder() from the SAME config load as the telemetry section, and placed beside kirocrew.process.role in _resource_attributes().
  • Refresh on config change: a resource is fixed for the life of a MeterProvider, so watch_config() now subscribes the existing rebuild to agent.acp_backend and agent.provider as well as telemetry. A change to either drops the recorder and the next metric rebuilds with the new label. Both paths are live settings (not restart-marked). This is the gateway's watcher; a short-lived process keeps the value it booted with. Downstream, the switch reads as a new series set from the change on.
  • docs/system-specs/modules/metrics.md: a kirocrew.harness row in the resource-attribute table, a "The harness label" section (values, refresh, series bound), and updated rows for both metrics.

Series bound, measured against the code rather than multiplied out:

  • Resource attribute: one value per process, so no growth inside a process; it only partitions installs (11 values at most today).
  • kirocrew.session.started: at most x11, in practice the number of harnesses one install runs, usually 1 or 2 (the default plus a member override).
  • kirocrew.turn.duration: close to x1. Each harness serves its own model ids (capabilities_for(backend).model_id_namespace), so model already mostly determines harness. Growth is bounded by the (model, harness) pairs an install actually runs, not by the product of the two sets. The overlap is kiro-cli and KAS, which share the acp namespace; there the label is what separates them, which is the point.

Metrics RFC rules (#22180 section 4): no new instrument and no new histogram (the attribute rides the existing counter and the existing turn histogram); every value is from a closed enum with other; no CUMULATIVE series added; recording stays an in-memory add (the provider read is two attribute reads, no disk, network or log on the turn path); behind telemetry.enabled and the existing attribute sanitiser. Rule 6 (source preference): this adds no new count, so there is nothing to fold over crew_log entries; the attribute is attached at each metric's existing single commit point (the registry insert for session.started, the turn's EVENT_COMPLETE emit for turn.duration), after the write it reports.

Backwards compatibility

Compatible: every existing attribute and metric name is unchanged; a dashboard query that does not group by harness sums over it and reads the same totals. record_session_started gains an optional positional provider and _emit_turn_metric an optional harness keyword, both with defaults.

Removes nothing: the change only adds a resource attribute and one point attribute on two metrics; provider_label keeps its exact answers (its shape check moved into _provider_shape_backend, which it now calls).

Tests

  • test/metrics/test_harness_label.py (new):
    • the value set equals ACP_BACKENDS_KNOWN in config spelling plus claude_code / unknown / other; every known backend maps to its config word; an unknown backend (gemini, wrong case, the config word kiro used as a runtime id, a non-string) becomes other; None is unknown; the Claude Code seam wins.
    • each surface resolves to its harness: kas, kiro, claude (claude-agent-acp), codex, on an unstarted AcpProvider, a started one (client swapped for AcpSessionProvider) and a bare AcpSessionProvider; a foreign shape, a MagicMock or an AcpProvider with no client is unknown, never kiro; an unregistered backend on a live provider is other.
    • background shapes: a bare AcpSessionHandle, a _ProviderBgSession and an AcpRuntime resolve to their backend; kirocrew.turn.duration through persist_token_record_async with either background handle as model_source reads kas; provider_label still answers the default for both handles.
    • per-session override, on both metrics: with the install default set to KAS, kirocrew.session.started from a Codex session reads codex; kirocrew.turn.duration through persist_token_record_async with a Codex model_source reads codex (and the usage family carries no harness); the dashboard owner emits claude, unknown, other for a claude client, no harness, and an out-of-set value.
    • resource attribute: a real recorder exports a shard whose resource carries kirocrew.harness=kas; after the config applier runs on a changed default, the rebuilt recorder exports kiro, and an unregistered default exports other; watch_config() subscribes to agent.acp_backend and agent.provider.
  • test/test_chat_runner_base_surface.py: the pinned _emit_turn_metric signature gains harness.
  • test/test_compaction_failure_pause.py, test/test_compaction_stop_declined.py: their record_session_started stubs accept the provider argument.
  • test/test_services_config_hot_reload.py: the telemetry applier's pinned prefixes now include agent.acp_backend and agent.provider.

Run locally (-n 2): test/metrics/test_harness_label.py, test_session_duration.py, test_turn_metric.py, test_turn_profile.py, test_resource_attrs.py, test_provider_bucket_views.py, test_telemetry_handler.py, test_otlp_wire_e2e.py, test_crew_log_health.py, test_telemetry_column_taxonomy.py, test/test_acp_backend_kas.py, test_acp_provider.py, test_agent_sdk_capabilities.py, test_agent_sdk_boundary.py, test_harness_parity.py, test_chat_runner_base_surface.py, test_chat_runner_composition_contract.py, test_dashboard_chat_turn_wiring.py, test_background_turn_accounting.py, test_session_allocation_bindings.py, test_compaction_* above, test_services_config_hot_reload.py, test_source_pin_budget.py. All pass except test_resource_attrs.py::test_a_failed_mint_costs_only_the_attribute, which fails the same way on unmodified main in my local environment (the SDK there does not substitute a service.instance.id); CI is the reference for it. scripts/check_agent_sdk_boundary.py, scripts/check_harness_parity.py, scripts/check_file_line_cap.py and scripts/docs_lint.py pass.

Manual verification

N/A - the resource tests build a real recorder and read the exported JSONL shard, which is the same resource the OTLP reader sends; the point-attribute tests drive the real emit functions.

Related Issues

Part of #21270.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners October 21, 2026 22:02
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces back to the Goal. The one overlap, a second backend vocabulary, is already deferred to #24883.

Notes

What this change ships

Inventory (7 items) — 6 justified

Intent: let operators count installs, sessions and turns per agent harness. ADDITION.

  1. Every exported series carries a kirocrew.harness resource attribute (the install's default harness) — justified
  2. kirocrew.session.started gains a harness point attribute (the session's effective harness) — justified
  3. kirocrew.turn.duration gains a harness point attribute — duplicate of providers/acp_runtime_start.py backend label (deferred)
  4. Changing agent.acp_backend or agent.provider now rebuilds the metrics recorder (watcher plus consent recheck) — justified
  5. A closed harness vocabulary built from ACP_BACKENDS_KNOWN in metrics/harness.py — justified
  6. A private _provider_backend reader that tells a miss (None) apart from kiro-cli's "", plus a pass-through in the driver layer — justified
  7. The background session handle exposes a read-only provider property — justified

[FIRST-PRINCIPLES-REVIEWED] ab38cb5

Description read

[DESCRIPTION-READ] 1fe71725782db625334d45c5be5336c26e437e3031dbcd99a7b1bad6cfc6713b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: both labels use one vocabulary built from ACP_BACKENDS_KNOWN, a backend that can't be resolved reads as unknown rather than kiro, and the change is additive and easy to revert.

Notes

  • Every series now carries kirocrew.harness. When the harness changes, the recorder is rebuilt by two triggers, watch_config and the consent worker's _agent_key compare. That splits one policy across two owners, so check that both keep firing together.
  • The ## Backwards compatibility section is cut off in the description, so I couldn't check its full reader list. It includes one Removes nothing line, and the patch removes nothing, so this doesn't block.

[DESIGN-REVIEWED] ab38cb5

Description read

[DESCRIPTION-READ] 1b77e4cccaebec290193fadc8d4c43692a7686db70e4b9667a3c3b6cc49cfc0c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me on the code. Head reviewed: 44f2c3b. No must-fix or should-fix findings.

What I checked on this head:

  • The value set in metrics/harness.py is derived from ACP_BACKENDS_KNOWN in config spelling plus claude_code, unknown and other, so both attributes stay closed. _closed_harness in metrics/turns.py folds any caller value outside the set to other, and an empty value to unknown.
  • providers/acp.py::provider_backend is a pure extraction of the shape check provider_label used. provider_label returns the same answers, and a foreign shape now returns None instead of the kiro-cli empty id.
  • All four record_session_started call sites pass the provider that the registry entry was built around. In each site that name is in scope (open_task_session, get_or_create, the background runtime, and session.provider on the compaction keep-alive path).
  • Every persist_token_record_async caller that passes model_source passes the live provider or client, so harness_of(model_source) reads the effective backend. A caller with no model_source records unknown, as documented.
  • watch_config() subscribes agent.acp_backend and agent.provider through live.subscribe, which accepts several prefixes and refuses a restart-marked path at registration. Neither path is restart-marked on this head (requires_restart returns False for both), so the subscription does not raise.
  • I ran test/metrics/test_harness_label.py and the three touched test files on this head: 398 passed.
  • The resource attribute is read from the same config load as the telemetry section. The point attribute adds a cost of one or two attribute reads on the turn path, with no disk, network or lock.
  • The owning spec (docs/system-specs/modules/metrics.md) is updated in the same commit.

What remains before approval (maintainer checks, not code findings):

  1. PR Readiness is still running on this head. The AI review lanes have not posted yet.
  2. This PR is part of a parent issue that needs a design sign-off from a maintainer. The tracking design document for that issue is still a draft. A maintainer should confirm that the harness attribute is in the agreed scope.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ab38cb5

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ab38cb5

Verdict parsed from the review's SHA-scoped output markers for commit ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047 false-positive: <reason, 20 to 500 characters>

@chenmingwei23
chenmingwei23 force-pushed the feat/metrics-harness-label branch from 44f2c3b to 3a83863 Compare October 21, 2026 22:15
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Review findings on 44f2c3b, answered on 3a83863:

  • Opus 5.5 / GPT 6.1 (blocking, usage.py): background one-liner turns labelled harness=unknown. Real; fixed. provider_backend now reads a bare AcpSessionHandle (its runtime's backend) and an AcpRuntime, and _ProviderBgSession exposes a read-only provider that harness_of unwraps one level. provider_label is unchanged, so session-map and usage-row labels do not move. Tests: test_the_background_handle_shapes_resolve_too, test_turn_duration_from_a_background_one_liner_names_its_harness, test_the_session_map_label_ignores_the_handle_shapes.
  • Design note (an AcpProvider with no client read as kiro): real; fixed. That shape now resolves to None -> unknown; provider_label still answers the default for it. Test: test_a_provider_with_no_client_is_unknown_not_kiro.
  • First Principles note (claude_code seam is always False on this build because agent.provider admits only acp): kept on purpose. The value was asked for explicitly so the label is already closed over the Claude Code seam when an edition re-registers that provider value; it costs one bool and one extra watched config path, and on this build it never fires.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Notes on 3a83863 (all non-blocking), answered without a code change:

  • Design note (does a harness switch lose the last export window?): no. The applier is the existing _on_config_change, which runs shutdown() on a worker thread; shutdown() takes the live provider and calls _flush_detached_provider, i.e. MeterProvider.shutdown(), which runs a final collect and export on every reader before the next metric call rebuilds. Same path a telemetry.* edit already takes; test_a_config_change_relabels_the_rebuilt_recorder reads the flushed shard of each generation.
  • First Principles, _closed_harness in turns.py: kept. emit_turn_duration is a public emit function, and the closed-set guarantee belongs at the point the attribute is written, not only at today's two callers; test_turn_duration_from_the_dashboard_owner_carries_the_effective_harness pins an out-of-set value becoming other.
  • First Principles, RESOURCE_ATTR_HARNESS alias in provider.py: kept to match RESOURCE_ATTR_PROCESS_ROLE beside it, so every resource key _resource_attributes() writes is named in that module; test_the_resource_key_has_one_spelling pins the two equal.
  • First Principles, public names in metrics/harness.py: harness_for_backend and configured_harness are the documented mapping (metrics.md "The harness label") and are what the value-set tests exercise; they stay public.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix round. Head reviewed: 3a83863.

The background one-liner fix holds up on this head. provider_backend now reads a bare AcpSessionHandle (through its runtime) and an AcpRuntime, _ProviderBgSession exposes a read-only provider, and harness_of unwraps it one level only. provider_label still goes through _provider_shape_backend alone, so the session map label does not move, and an AcpProvider with no client now reads unknown instead of kiro. The new tests pin all three shapes.

Two required test jobs fail on this head, on both Linux and Windows, and both failures come from this PR:

  1. Backend Tests shard 1: test/test_services_config_hot_reload.py::TestTelemetryRecorderRebuild::test_watch_config_subscribes_on_telemetry asserts _one_sub("telemetry").prefixes == ("telemetry",). watch_config() now subscribes ("telemetry", "agent.acp_backend", "agent.provider"), so the assertion fails. That test file is not in this PR, so I cannot anchor an inline comment there. Suggestion: update the expected tuple to the three prefixes (ideally read from _RESOURCE_CONFIG_PATHS), since the wider subscription is intended.
  2. Backend Tests shard 3: test/test_source_pin_budget.py reports test/metrics/test_harness_label.py reading the dashboard turn source once against a budget of 0. See the inline comment.

Approval is held for a maintainer either way: the parent issue needs a design sign-off on scope (whether the harness attribute is in the agreed scope), and PR Readiness is still running on this head.

What I checked: the incremental change since 44f2c3b, all five AI review comments on this head (Design, First Principles, Opus and GPT pass, UX skipped), the author replies to their notes, and the failing job logs.

Comment thread test/metrics/test_harness_label.py Outdated

from kiro_crew.dashboard import chat_runner

src = inspect.getsource(chat_runner)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This source read trips the source pin budget gate

test/test_source_pin_budget.py counts every inspect.getsource of the dashboard turn modules and gives this new module a budget of 0, so Backend Tests shard 3 fails on Linux and Windows:

src = inspect.getsource(chat_runner)
assert "harness=harness_of(client)" in src

It also pins the exact spelling of the call, so a harmless rename breaks it while a wrong argument with the same text would still pass.

Suggestion: assert on behaviour instead. For example, run a turn through test/turn_harness.py's run_turn with a client whose backend is known, patch chat_runner._emit_turn_metric (or the recorder) to capture its kwargs, and assert harness equals the expected value. Then drop this test.

Rule: test/test_source_pin_budget.py budget for dashboard turn source reads.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced on c0d7447: the source read is removed, and test_a_dashboard_turn_labels_its_duration_with_the_turn_clients_harness now runs a real turn through turn_harness.run_turn and asserts the emitted kirocrew.turn.duration carries harness=codex for a Codex-backed client and unknown for a plain one. test_source_pin_budget.py passes.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/metrics-harness-label branch from 3a83863 to 947452f Compare October 21, 2026 23:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 22, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/metrics-harness-label branch from 947452f to c0d7447 Compare October 22, 2026 00:07
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Both fixed; now on c0d7447 (rebased on main ba53277).

  1. Shard 1, test_watch_config_subscribes_on_telemetry: the expected prefixes are now built from the constant, ("telemetry", *provider_mod._RESOURCE_CONFIG_PATHS), with a separate assert pinning _RESOURCE_CONFIG_PATHS == ("agent.acp_backend", "agent.provider").
  2. Shard 3, source pin budget: the inspect.getsource(chat_runner) test is gone. In its place, test_a_dashboard_turn_labels_its_duration_with_the_turn_clients_harness runs a real dashboard turn through test/turn_harness.py's run_turn and asserts on the kirocrew.turn.duration sample the turn emits: a client fronting a Codex provider gives harness=codex, and a plain scripted client gives unknown.

Locally (-n 2): test_harness_label.py, test_services_config_hot_reload.py, test_source_pin_budget.py and the other touched suites pass. Two remaining local reds are not from this PR: test_backend_cards_advertised.py::test_no_new_runtime_site_gates_a_field_backed_line_on_a_set fails the same way on unmodified main, and test_resource_attrs.py::test_a_failed_mint_costs_only_the_attribute fails on unmodified main in my local environment only.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 22, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/metrics-harness-label branch from be3aea1 to 033f243 Compare October 27, 2026 20:28
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Reply to r129. New head: 033f243 (still one commit; same base).

Item 8 (_closed_harness): fixed, removed. Both production callers pass harness_of(...), which already maps every input into HARNESS_VALUES. emit_turn_duration now writes harness as given. The only job the helper still did was turning a caller that passes nothing into unknown; that is now the parameter default (HARNESS_UNKNOWN) on emit_turn_duration and on the two wrappers that forward it (turn_stats._emit_turn_metric, usage._emit_turn_histogram). The test case that passed a raw out-of-set string to _emit_turn_metric is dropped, since no caller does that; the no-argument case still asserts unknown.

Item 9: fixed.

  • configured_harness is inlined into set_configured, its one caller, and removed from __all__. Same results: unknown for a missing or unreadable agent section, claude_code on that provider seam, otherwise the config spelling of acp_backend. The test now calls set_configured.
  • provider.RESOURCE_ATTR_HARNESS is dropped. _resource_attributes writes _harness.RESOURCE_ATTR_HARNESS, so the key has one spelling in metrics/harness.py. The spec table in docs/system-specs/modules/metrics.md now points at set_configured.

Item 1 (scope sign-off) is unchanged and still waits on a maintainer per #21270.

Related tests run locally: test/metrics/test_harness_label.py, test/metrics/test_resource_attrs.py, test/metrics/test_mcp_gateway_gauges.py, test/metrics/test_turn_profile.py, test/test_turn_metric_all_surfaces.py, test/test_chat_runner_composition_contract.py, test/test_campaign_facades_base_public_names.py: all pass except test_resource_attrs.py::test_a_failed_mint_costs_only_the_attribute, which fails the same way on the previous head be3aea1 in my local environment and passed in CI there. isort, black, flake8, mypy on the touched modules, scripts/docs_lint.py and scripts/check_agent_sdk_boundary.py are clean.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 27, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/metrics-harness-label branch from 033f243 to 86cff1d Compare October 27, 2026 20:36
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Head is now 86cff1d (still one commit). The Opus 5.5 lane caught that the pinned signature of _emit_turn_metric in test/test_chat_runner_base_surface.py still showed harness: 'str' = ''; it now pins 'unknown' to match the new default. The reply above for items 8 and 9 applies unchanged. On the lane's doc note: on this base HARNESS_VALUES has 11 values and no copilot, so the spec list and count match.

buluoray
buluoray previously approved these changes Oct 27, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 86cff1d.

The harness label is derived positively from the backend id (harness_for_backend, metrics/harness.py:71-84). The Claude Code seam takes precedence, unresolved maps to unknown, and anything outside ACP_BACKENDS_KNOWN folds to other. Both attributes therefore stay in a closed set that grows only with the backend table. On current main that set already includes copilot. All four record_session_started sites pass the provider the session was built on, and both turn-duration callers resolve the label with harness_of(...) from the live client or model_source. The change is additive: no attribute is renamed, so existing queries keep working, and the series split on a harness switch is documented.

The previous head's red base-surface pin (harness: 'str' = '' against a default of HARNESS_UNKNOWN) is fixed here, and nothing else changed in the push. Merged onto current main, the metrics suite plus the touched and related tests pass (2057 passed, 3 skipped). Removing the seam check, the other fallback, the resource attribute, the session-started label, the wider config subscription or the one-level unwrap each turns a named test red.

Non-blocking follow-ups:

  • docs/system-specs/modules/metrics.md:388 lists the backend values by hand and leaves out copilot, and line 459 says "11 values", which is 12 on current main. Consider pointing at HARNESS_VALUES instead of enumerating.
  • With _closed_harness gone, the closed-set guarantee on kirocrew.turn.duration rests on callers using harness_of. A test that asserts every emit_turn_duration(harness=...) call site passes harness_of(...) would keep that from regressing.

This still needs the maintainer telemetry-scope confirmation that #21270 calls for before merge.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Head is now a099ef8 (still one commit). It fixes the GPT 6.1 finding on 86cff1d (gatewayd keeps the previous harness after a live switch). The finding is real: the MCP gateway daemon registers no watch_config(), and its consent sweeper reaches _consent_worker, which returned early whenever consent was unchanged.

Fix, in the existing recheck path:

  • _build_recorder records the raw agent.acp_backend / agent.provider pair it read from config (_Build.agent_key, kept as _built_agent_key on install, cleared with the provider).
  • _consent_worker reads that pair from the same config load as consent. With consent on, a pair that differs from the built one takes the same rebuild a consent flip takes. With consent off there is no resource to relabel, so nothing changes.
  • New test test_the_consent_recheck_relabels_a_process_with_no_watcher (in test/metrics/test_harness_label.py): an unchanged config keeps the same provider; a switch to Codex, and then a switch of agent.provider alone, relabel the next export. It fails on the previous head.
  • The metrics spec "Refresh" paragraph and the commit message now describe the daemon path.

The reply above for items 8 and 9 is unchanged.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Head is now 370b930 (still one commit). It fixes the Opus 5.5 finding on a099ef8: main added copilot to ACP_BACKENDS_KNOWN, so on the merge ref the spec's hand-written list of kirocrew.harness values and its "11 values" bound were stale. The spec now says "any id in ACP_BACKENDS_KNOWN" with a few examples, and gives the bound as len(HARNESS_VALUES) (the known backends plus three), so neither goes stale when a backend is added. Docs only; no code change.

@chenmingwei23
chenmingwei23 force-pushed the feat/metrics-harness-label branch from a099ef8 to 370b930 Compare October 27, 2026 20:58

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 370b930 against our last review at be3aea1. Same base, still one commit. No blocking findings.

What changed since be3aea1, and what I checked:

  1. Consent recheck now relabels a process with no config watcher. _build_recorder records the raw agent.acp_backend / agent.provider pair on every return path that has a config, _install_locked keeps it, _take_provider_locked clears it, and _consent_worker rebuilds a live recorder when the pair on disk differs. A build that failed after consent (no OpenTelemetry, init error) still carries the key, so an unchanged config does not rebuild in a loop. A disabled recorder never rebuilds for a harness move. The new test covers unchanged config, a backend switch and a provider switch.
  2. _closed_harness is gone and emit_turn_duration writes harness as given, defaulting to unknown. I grepped every caller in src/ at this head: the two production paths pass harness_of(...), which maps every input into HARNESS_VALUES, so the set stays closed.
  3. configured_harness folded into set_configured, and RESOURCE_ATTR_HARNESS now has one home in metrics/harness.py. The pinned base surface signature matches the new unknown default.
  4. The spec no longer hand-lists the value set or its size, so a new backend cannot make it stale.

Not approving yet: the existing request for changes stands on the telemetry data scope, which needs a maintainer to confirm kirocrew.harness and the harness point attribute, and several required checks and the readiness signal were still running at review time. From the code side this looks good to ship once those clear.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Holding on 370b930. I found no code defect; the hold is a maintainer decision plus one unanswered note.

  1. Scope sign-off. #21270 keeps child PRs unmerged until a maintainer confirms the telemetry data scope. Nobody has confirmed kirocrew.harness and the harness point attribute yet. A maintainer comment on this PR or the issue clears this.
  2. First Principles Item 7 on this head: HARNESS_VALUES is in metrics/harness.py's __all__ with no runtime reader. Drop it from __all__, or reply with why it stays public.

The delta since be3aea1 reads sound: the consent recheck relabels the gateway daemon after a harness switch (provider.py#L828-L841), both production callers pass harness_of(...) so the value set stays closed, and #24882 and #24883 track the deferred items. Checks on this head are green.

— crew-pr-reviewer (r131)

Nothing in the metrics could say which agent harness people run. The
`provider` attribute on the turn family reads `acp` for kiro-cli, Codex and
most other harnesses alike, and no metric read `acp_backend`.

Two readings, one closed vocabulary owned by `metrics/harness.py`:

* `kirocrew.harness`, a resource attribute on every exported series: the
  install's configured default (global `agent.acp_backend`, or
  `claude_code` on that provider seam). Counts installs per harness.
* `harness`, a point attribute on `kirocrew.session.started` and
  `kirocrew.turn.duration`: the session's effective harness, read off the
  live provider, so a per-session or per-agent override counts as its own
  harness.

The value set is derived from `ACP_BACKENDS_KNOWN` in its config spelling
(kiro-cli is `kiro`) plus `claude_code`, `unknown` and `other`; an id
outside the set maps to `other`.

The provider shape check is shared with the session-map label:
`providers/acp.py::_provider_backend` (private) is factored out of `provider_label`
and reached through the agent SDK surface.

The recorder reads the resource value from the same config load as the
`telemetry` section, and the config watcher now also rebuilds on
`agent.acp_backend` and `agent.provider`, so a changed default relabels
the next export. The consent recheck compares the same pair, so a process
with no config watcher (the MCP gateway daemon) relabels too.

Not here: `harness` on `kirocrew.session.duration` and the
`turn.tokens` / `credits` / `cost_usd` family.
@chenmingwei23
chenmingwei23 force-pushed the feat/metrics-harness-label branch from 370b930 to ab38cb5 Compare October 28, 2026 11:28
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Re r131, First Principles item 7 (HARNESS_VALUES exported with no runtime reader):

Agreed. Head ab38cb5 drops HARNESS_VALUES from __all__ in src/kiro_crew/metrics/harness.py. The constant stays as a module-level name: test/metrics/test_harness_label.py reads it as h.HARNESS_VALUES to pin the closed value set, and the metrics spec cites it as the cardinality bound. Neither needs it in __all__. No runtime module imports it.

Item 1 (scope sign-off) is unchanged and still waits on a maintainer.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 28, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at ab38cb5. The base is unchanged from 370b930 and the only change removes HARNESS_VALUES from __all__ in metrics/harness.py. The constant stays as a module-level name, the test reads it as h.HARNESS_VALUES, and nothing imports it through __all__ or a star import, so this closes the First Principles item 7 note with no behavior change.

No new findings. From my side this is still good to ship. Approval waits on the open maintainer scope sign-off for the telemetry data in #21270 (the remaining hold on this PR) and on the readiness check finishing on this head.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 28, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at ab38cb5c7f956dbea25b93fb1d87cbcf5f2f0047: a maintainer still has to confirm the telemetry scope.

The r131 note is fixed: HARNESS_VALUES is out of __all__, and that is the only change since 370b9305a0. I found no code defect, every AI lane passes, and CI is green.

What holds it: #21270 keeps its child PRs unmerged until a maintainer confirms the data scope. Nobody has confirmed the kirocrew.harness resource attribute or the harness point attribute yet. A maintainer comment on this PR or the issue clears it.

— crew-pr-reviewer (r133)

@iamwhatever
iamwhatever merged commit 062fda1 into main Oct 28, 2026
110 of 111 checks passed
@iamwhatever
iamwhatever deleted the feat/metrics-harness-label branch October 28, 2026 17:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants