Skip to content

feat(metrics): count webhook requests, subagent terminals and registered hooks - #22223

Merged
iamwhatever merged 2 commits into
mainfrom
feat/22180-64-event-sources
Oct 20, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
feat/22180-64-event-sources

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An operator who has turned telemetry on can see how often inbound webhooks fire and how they end, how sub-agent runs finish, and how many webhook callbacks their install has registered.

Three event sources are currently invisible. kirocrew.subagent.spawned counts sub-agents that start and nothing counts how they end, so "half our fan-outs are being stopped by hand" and "half of them fail" look identical. Nothing counts inbound webhook requests at all, so an operator whose integration has been failing authentication, or getting turned away at the capacity gate, has only the dashboard's 50-run history ring to scroll and no series to alert on. And kirocrew.inventory.* publishes what an install has configured -- crons, skills, MCP servers, knowledge sources -- but not registered webhook callbacks, so a host that stopped answering its hooks is not visible next to one that never had any.

Why it matters

These are the questions an operator running Crew on more than one machine actually asks, and today each needs a person to open a dashboard per host. A webhook integration that silently started returning unauthorized after a token rotation produces no signal; a fan-out whose members are all dying produces a spawn count that looks healthy. Both are the kind of drift the metrics trunk exists to make visible over time, and both are cheap to answer with a counter.

Not a goal

  • Any change to what a webhook or a sub-agent DOES. These are read-only observations at points that already exist.
  • Turning collection on. Everything stays behind the existing telemetry.enabled consent gate and is a no-op without it.
  • The actor vocabulary change that accompanies these metrics in the RFC. Deferred; see the last section.
  • Latency or duration for any of the three. Those need buckets and a measurement.
  • Reformatting webhooks.py. It is in the black baseline and stays there; the diff touches only the lines this change needs.

What changed (motivation → approach → change)

Three instruments, each counted at a point that already exists, every attribute value from a closed enum, no id, name, path or free text anywhere.

kirocrew.subagent.completed{outcome} is emitted from TerminalCoordinator._record_terminal_metric, called from _report_terminal_impl past the point the finalize claim has been granted. Counting on the two crew-log closers was rejected: subagent/completed and subagent/failed are two closers for ONE run, and a stopped run reaches paths that could fire both. The exclusive terminal report is the only point reached exactly once however the race between the reaper and the run loop's finally resolves. It sits BESIDE _record_crew_log_terminal rather than inside it, because that method returns early when the crew log is off, which would narrow the population to installs that happen to keep one.

kirocrew.webhook.requests{outcome} counts every inbound request the endpoint DECIDES, which is wider than the runs the history ring stores, and that boundary is the substance of this instrument. Recorded runs are counted in WebhookRunStore.record, outside its own try, since the series counts the decision and not the history row.

That alone is not the whole population. api_hooks_agent turns a request away in 21 places before any run exists, and for two of them recording is impossible or self-defeating: the unreadable store is the thing a record would write to, and the per-source auth throttle exists to stop doing work on a flood. The chain that matters: a rotated credential produces a few unauthorized runs, the tenth failure trips the throttle, and every later request answers 429. Uncounted, the unauthorized rate falls to zero while the integration is still broken, which an alert reads as recovered -- the opposite of this metric's purpose.

So the refusals carry their own closed members, disjoint from the recorded ones by construction and pinned by a test so one request is never counted under both: throttled, unavailable and rejected_body. A revoked source keeps the recorded unauthorized member, being a credential this build judged. Both halves emit through one function, webhooks.count_request.

Each refusal gets one count_request line immediately above its own untouched return. A shared helper that both counted and answered was tried first and is wrong here: the error-code contract gate requires a literal code in a dict literal at the json_response site, and a helper taking the body as a parameter can only pass a variable, so the gate correctly read it as a new uncoded error response. Completeness therefore rests on a test rather than on the shape: test_every_refusal_branch_is_counted reads the function's syntax tree and fails if any answer reaches a caller without reaching the counter, with the accepted path as the single permitted exception. That is the real guard for this defect class -- the bug was one forgotten branch among twenty-one. The accepted path is deliberately uncounted at admission, since its run records a terminal outcome later.

kirocrew.inventory.webhooks.registered is an observable gauge read from live state on the reader's own ticker thread, never an up/down counter that drifts when a process dies. Its reader reduces through webhooks.registered_context_count, which also owns the shape test for "is this value a registration" -- hooks.json is shared with the script-hook store, whose hooks key holds a LIST, so a second spelling of that filter would let the gauge report a number the dashboard's own listing contradicts. A missing file is a genuine 0; a file present and unparseable yields no observation and increments kirocrew.inventory.probe.failures{probe=webhooks}, the same fault-versus-gap split crons.json draws.

Not a product-shape change: no default moves and no user-facing capability changes. Each instrument is an in-memory add behind the existing consent gate; nothing new runs on the crew-log append thread or the turn path.

Rows for all three go into the "Instrumented signals" table in docs/system-specs/modules/metrics.md, directly after kirocrew.subagent.spawned; no other row is moved or edited.

Backwards compatibility

Compatible: no field becomes required, no validator tightens, no type narrows, no stored shape changes. metric_outcome, count_request, is_context_registration and registered_context_count are additive, and WebhookRunStore.record keeps its signature, return value and existing refusal of an unknown outcome. The refusal members are metric-only and never written to a stored row, so list_runs' display filter and every reader of the history file are untouched.

The webhook endpoint's observable behaviour is unchanged: every refusal keeps its status code, response body, ordering and log line, and no store read was added. test_webhooks_api.py and test_webhooks_auth_surface.py pass unmodified.

Removes nothing: two names are dropped and neither has a reader. The module-level data_home import in hooks_webhook_intake.py became unused when _hook_store_path started resolving through webhooks.context_registrations_path(); grep across src/ and test/ on origin/main finds no hooks.data_home or hooks_webhook_intake.data_home reader, and the hooks facade's TYPE_CHECKING manifest entry for it is removed in the same commit, which test_hooks_handler_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers enforces. hooks_webhooks.py::_is_context_registration is deleted; its two call sites, both in that same file, now call webhooks.is_context_registration directly, and it had no tests and no reader outside the facade export removed with it.

Tests

test/test_webhooks_request_counter.py (new, 29 tests) owns the request series across its whole population: the refusal members are disjoint from the recordable ones and other is not recordable, so no request counts twice; the enum is exactly the two halves plus the remainder; every declared member survives the fold and an unknown value, empty string, None, an int and a bare object all land on other; count_request adds exactly one, writes no run, and does not raise when the recorder is wedged; the throttled 429, the unreadable-store 503 and a signed caller's rejected_body 400 are each driven through the real api_hooks_agent asserting outcome, status and response code; an invalid bearer, which DOES record a run, counts exactly once, proving the emitters do not overlap; test_every_refusal_branch_is_counted and test_every_counted_refusal_names_a_declared_member walk the endpoint's syntax tree to prove all 21 are counted and each names a module constant.

test/metrics/test_business_counters.py drives the real WebhookRunStore.record against a temp dir: each of the six recorded outcomes is one increment; the four refusal outcomes stay distinguishable; an unknown outcome folds to other AND record refuses it before counting; detail, name, the hook id, session key and token id never reach an attribute; a history write raising OSError still counts the request; a disabled recorder records nothing. TestSubagentCompleted covers each terminal outcome, a stopped run never reported as a completion, an unreadable outcome folding to unknown, a record whose outcome property RAISES still counted once, an AST check that the emit is not gated on the crew log and is called exactly once, and the enum harvested from SubagentInfo.outcome's own return literals.

test/metrics/test_inventory_gauges.py: the count is the registrations only, with the script-hook store's list entries sharing the file not counted; missing file and empty object are both a genuine 0 with no probe failure; a corrupt file is None AND a counted fault; a top-level list is 0, not a fault; through a real in-memory SDK pipeline a corrupt file produces no observation, raises nothing in the callback and publishes the failure on probe.failures{probe=webhooks}; the gauge agrees with the dashboard's own _read_hook_registrations; the new gauge publishes with no attributes at all.

Both new emit sites are in the existing TestAttributeValuesAreBounded AST sweep, which fails on an interpolated attribute value.

Run locally and green: the three files above plus test_webhooks_api.py, test_webhooks_api_read_and_tokens.py, test_webhooks_store.py, test_webhooks_auth_surface.py, test_error_code_contract.py, test_subagent_parts_composition_contract.py, test_subagent_coverage_completion.py, test_hooks_handler_facade.py, test_lazy_data_home_paths.py, test_facade_mirror_guards.py. flake8, isort, scripts/check_black_formatting.py, scripts/check_comment_history.py and scripts/docs_lint.py are clean.

Manual verification

N/A -- unit coverage is sufficient and is the stronger evidence here: each instrument is driven through its real production call site (the run store's record, the real api_hooks_agent for the refusals, the terminal coordinator's exclusive report, a real OTEL in-memory reader for the gauge), so there is no integration path a manual run would exercise that the tests do not.

Related Issues

Implements section 6.4 of the crew usage metrics RFC (wave 2), tracked under #21270.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Notes on scope

The actor vocabulary change in RFC section 6.4 is deliberately NOT in this PR. The RFC proposes adding webhook and heartbeat to the closed actor enum of turn/started, on the premise that those wakes currently fall into other. They do not: crew_log.on_turn_started has exactly one call site, in the dashboard turn runner, and neither the webhook turn path (which runs in an ephemeral hook:-prefixed session) nor the heartbeat turn path drives its provider through that runner. Both write no crew-log entry at all, so there is no turn/started for either and nothing folds to other today. Adding the two values alone would change nothing observable, and making those paths emit turn/started introduces turn entries for sessions that have no session/opened and lands on the session and turn folds -- a separate reviewable change.

One sibling still spells the registrations path itself. webhooks.context_registrations_path() is the owner for the readers this PR touches (the dashboard handler and the new gauge), but the register_hook writer in mcp_tools/control.py builds config_dir() / "hooks.json" directly. Both resolve to the same file; repointing a write path was out of scope here, so the claim is "one owner for these readers", not for every caller in the tree.

@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners October 19, 2026 16:17
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c0fa82c2013268eddaf7ebb47add4e8907bd6df1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of c0fa82c2013268eddaf7ebb47add4e8907bd6df1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

unknown and other are fallback outcome values that no code path can produce, yet each comes with a fold function, a guard and a spec entry.

Not justified as shipped

  • Item 5 — zero consumers: in SubagentInfo.outcome (run_record.py:662), every branch returns stopped, failed or completed. WebhookRunStore.record raises unknown outcome before it reaches count_request. All 21 refusal sites pass named constants. I grepped metric_outcome(|subagent_outcome(|SUBAGENT_OUTCOME_UNKNOWN|METRIC_OUTCOME_OTHER and found 0 callers outside the definitions that could yield either fallback value.
    Subtraction: delete SUBAGENT_OUTCOME_UNKNOWN, subagent_outcome and the try/except in _record_terminal_metric, and emit info.outcome directly. Delete METRIC_OUTCOME_OTHER and metric_outcome, and pass outcome straight to emit_counter.
    Clears when: the fallback values are removed, or a caller that can actually reach one is shown.

What this change ships

Inventory (7 items) — 6 justified

Intent: an operator with telemetry on can alert on how webhooks end, how sub-agents end and how many webhooks are registered. This is an ADDITION.

  1. A new count of finished sub-agent runs, split by outcome — justified
  2. A new count of webhook requests that become recorded runs, split by outcome — justified
  3. Requests turned away early (throttled, unavailable, bad body) are now counted too — justified
  4. A new gauge for registered webhook callbacks, with probe-failure reporting — justified
  5. Fallback values unknown and other, plus the functions that fold into them — zero consumers
  6. The hooks.json shape check and path now live in one owner, webhooks.py, and the private copy is deleted — justified
  7. The unused data_home re-export is dropped from the hooks facade — justified

[FIRST-PRINCIPLES-REVIEWED] c0fa82c

Description read

[DESCRIPTION-READ] a1998052c9bd14b9062bc2e60133c36b7c0089ea6c26742a343cd6c0ef6c1549

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c0fa82c2013268eddaf7ebb47add4e8907bd6df1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that every accepted webhook request reaches WebhookRunStore.record. A run that never records (for example, the gateway restarts mid-run) is never counted under any outcome.

[DESIGN-REVIEWED] c0fa82c

Description read

[DESCRIPTION-READ] 87003adf6b0d933072369d436c2281b966c4c6220a8c28994b53aba4949283ce

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c0fa82c2013268eddaf7ebb47add4e8907bd6df1 — this comment is updated in place on each push.

Review details

Advisory only: the new webhook gauge logs a WARNING on every metrics export while hooks.json is corrupt.

FINDING — src/kiro_crew/webhooks.py:280 — registered_context_count reads the file through raw = _read_json(target, {}), and that helper calls logger.warning("webhook store unreadable ...") every time a parse fails. When hooks.json is present but unparseable, the metrics reader thread therefore logs a WARNING every export interval (60s by default), and the message calls the shared script-hook file the "webhook store". This sidesteps _note_probe_failure's warn-once throttle, and the cron probe's non-raising reader (_read_job_records) stays quiet in the same situation. → Fix: in registered_context_count, parse with read_bytes_with_retry + json.loads, catching ValueError/OSError without logging and returning (0, False). That leaves the single warning to _note_probe_failure.

[OPUS-REVIEWED] c0fa82c

Verdict parsed from the review's SHA-scoped output markers for commit c0fa82c2013268eddaf7ebb47add4e8907bd6df1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c0fa82c2013268eddaf7ebb47add4e8907bd6df1: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c0fa82c2013268eddaf7ebb47add4e8907bd6df1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c0fa82c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c0fa82c2013268eddaf7ebb47add4e8907bd6df1: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/22180-64-event-sources branch from e635035 to 1b590c8 Compare October 19, 2026 16:36
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Answering both First Principles items on the new head.

Item 4 (webhook other remainder) -- cleared by the fix, not by argument. The verdict's own clearing condition was "the fold is removed, or a second caller that passes an unvalidated outcome is counted". The second caller now exists. Design Review found that record is not the whole population: api_hooks_agent answers 21 branches before any run exists, and the per-source auth throttle and the unreadable-store reads cannot record by design. Those refusals now emit through webhooks.count_request with their own members, so the fold validates values that never passed record's raise. The reviewer was right that the justification I had written for it -- a row read back off a newer build -- described a caller that did not exist; that sentence is gone.

Item 5 (sub-agent unknown remainder and its try/except) -- kept, and here is the exact limit of what the test proves. test_the_enum_covers_every_outcome_the_runtime_can_report harvests the outcome property's return literals from its syntax tree and asserts they are exactly completed / failed / stopped. So the verdict is correct that no fourth VALUE is reachable, and that half of the fold is dead weight by that measure.

What the test does not prove is that reading the property cannot RAISE, and that is the case the remainder is there for. outcome is a property over user_stopped and error on a record the manager mutates across a run's lifetime; the emit sits in the terminal report, which is also reached by the reaper and by the run loop's finally after a cancellation or a kill. I could not write a test that proves no attribute is ever missing at that point, and the failure mode if I am wrong is not a wrong label -- an exception there would propagate out of the terminal report, which is the one path that must complete.

So I have left it, with the honest scope: unknown is reachable only from the exception arm, test_a_record_whose_outcome_raises_is_counted_as_unknown pins that arm, and the enum harvest pins the other three. If the preference is to drop the guard and let the terminal report carry the risk, say so and I will remove the remainder, the fold and the try/except in one commit -- the harvest test keeps the doc row honest either way.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-64-event-sources branch from 4c29398 to ea38185 Compare October 19, 2026 17:12
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Correcting my earlier reply on item 5, and withdrawing the offer in its last paragraph -- please do not act on it.

Both remainder members are specified by the RFC this PR implements, #22180, so they stay as written rather than being argued for on their reachability:

  • Section 6.4's table gives kirocrew.subagent.completed the attribute outcome: completed / failed / stopped / unknown. unknown is a named member of the published enum, not a defensive extra I added.
  • Section 4 rule 3 is general: "Attribute values are closed enums. Unknown values map to other." That is where the webhook series' other comes from.

So the reachability argument I made for unknown was answering the wrong question. A member the RFC names is part of the signal's published shape, and a consumer reading these series is entitled to the full value list whether or not today's code can reach every member -- dropping one because it is currently unreachable would make the implementation disagree with the spec, and would have to be undone by the next change that can reach it.

Both remain pinned by tests: the enum harvest asserts SubagentInfo.outcome's three literals against the declared set, and test_a_record_whose_outcome_raises_is_counted_as_unknown covers the arm that reaches unknown. The metrics doc rows list the full value lists.

I am not pushing a change for either item. If a maintainer wants the RFC's own enum narrowed, that is an edit to #22180 and I will follow it here.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-64-event-sources branch from ea38185 to d0a2468 Compare October 19, 2026 17:23
…red hooks

Three event-source instruments from the usage-metrics RFC section 6.4, each a
counter or an observable gauge whose every attribute value comes from a closed
enum:

- kirocrew.webhook.requests{outcome} at WebhookRunStore.record, the single
  commit point every inbound request reaches (rejections and auth failures
  included), so one call is one request.
- kirocrew.subagent.completed{outcome} from the sub-agent manager's one-shot
  exclusive terminal report, past the finalize claim -- the point that makes one
  increment per run true where a counter on each crew-log closer would count a
  stopped run twice.
- kirocrew.inventory.webhooks.registered, an observable gauge reading the
  registered-hook count from live state through the same owner the dashboard's
  listing and its id-addressed delete use.

The webhook registration shape test and the registrations path move to
webhooks.py so the gauge and the dashboard surface cannot disagree on what a
registration is.
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-64-event-sources branch from d0a2468 to a0e23ee Compare October 19, 2026 17:24
kirocrew.webhook.requests claimed to count every inbound request but was
emitted only from WebhookRunStore.record, and api_hooks_agent answers 21
branches before any run exists. Two of them cannot record by design: the
unreadable store is what a record would write to, and the per-source auth
throttle exists to stop doing work on a flood.

That made the series go quiet in the case it exists for. A rotated credential
records a few unauthorized runs, the tenth failure trips the throttle, and every
later request answers 429 -- so the unauthorized rate fell to zero while the
integration was still broken, which an alert reads as recovered.

The refusals now carry their own closed members (throttled / unavailable /
rejected_body), disjoint from the recordable ones so one request is never
counted twice, both halves emitting through one webhooks.count_request. The
endpoint counts through a single _refused exit rather than an increment beside
each return, which lets a test prove from the function's syntax tree that no
branch answers without reaching the counter -- the defect was one forgotten
branch, so a guarantee resting on a human having read all of them would
reproduce it.

No behaviour change on the endpoint: same status codes, bodies, ordering and log
lines, asserted against the responses the helper replaced.
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-64-event-sources branch from a0e23ee to c0fa82c Compare October 19, 2026 17:53
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head c0fa82c. Approve on the code; merging waits on the design decision for #22180.

The webhook requests counter carries only outcome, and every emit goes through webhooks.count_request and metric_outcome, which fold onto a constant set. No hook name, session key, caller credential id, body field or run detail reaches an attribute. The registered-webhooks inventory gauge has no attributes, and the subagent completed counter folds onto four fixed values after the finalize claim, so a run is counted once. The tests catch it if the folds are loosened, if a refusal branch stops counting, or if the two outcome halves overlap. Related tests pass on this head.

Follow-ups, not blocking:

  • Please add the four new outcomes (throttled, unavailable, rejected_body, other) to RFC section 6.4 in #22180.
  • Consider logging the unparseable hooks.json case once instead of at WARNING on every collection.

@iamwhatever
iamwhatever merged commit 31880e6 into main Oct 20, 2026
111 of 112 checks passed
@iamwhatever
iamwhatever deleted the feat/22180-64-event-sources branch October 20, 2026 00:56
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 20, 2026
@buluoray buluoray mentioned this pull request Oct 20, 2026
5 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants