Repository navigation
feat(metrics): count webhook requests, subagent terminals and registered hooks - #22223
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: an operator with telemetry on can alert on how webhooks end, how sub-agents end and how many webhooks are registered. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] c0fa82c Description read[DESCRIPTION-READ] a1998052c9bd14b9062bc2e60133c36b7c0089ea6c26742a343cd6c0ef6c1549 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that every accepted webhook request reaches [DESIGN-REVIEWED] c0fa82c Description read[DESCRIPTION-READ] 87003adf6b0d933072369d436c2281b966c4c6220a8c28994b53aba4949283ce That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsAdvisory only: the new webhook gauge logs a WARNING on every metrics export while FINDING — src/kiro_crew/webhooks.py:280 — [OPUS-REVIEWED] c0fa82c Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
e635035 to
1b590c8
Compare
|
Answering both First Principles items on the new head. Item 4 (webhook Item 5 (sub-agent What the test does not prove is that reading the property cannot RAISE, and that is the case the remainder is there for. So I have left it, with the honest scope: |
4c29398 to
ea38185
Compare
|
Correcting my earlier reply on item 5, and withdrawing the offer in its last paragraph -- please do not act on it. Both remainder members are specified by the RFC this PR implements, #22180, so they stay as written rather than being argued for on their reachability:
So the reachability argument I made for Both remain pinned by tests: the enum harvest asserts I am not pushing a change for either item. If a maintainer wants the RFC's own enum narrowed, that is an edit to #22180 and I will follow it here. |
ea38185 to
d0a2468
Compare
…red hooks
Three event-source instruments from the usage-metrics RFC section 6.4, each a
counter or an observable gauge whose every attribute value comes from a closed
enum:
- kirocrew.webhook.requests{outcome} at WebhookRunStore.record, the single
commit point every inbound request reaches (rejections and auth failures
included), so one call is one request.
- kirocrew.subagent.completed{outcome} from the sub-agent manager's one-shot
exclusive terminal report, past the finalize claim -- the point that makes one
increment per run true where a counter on each crew-log closer would count a
stopped run twice.
- kirocrew.inventory.webhooks.registered, an observable gauge reading the
registered-hook count from live state through the same owner the dashboard's
listing and its id-addressed delete use.
The webhook registration shape test and the registrations path move to
webhooks.py so the gauge and the dashboard surface cannot disagree on what a
registration is.
d0a2468 to
a0e23ee
Compare
kirocrew.webhook.requests claimed to count every inbound request but was emitted only from WebhookRunStore.record, and api_hooks_agent answers 21 branches before any run exists. Two of them cannot record by design: the unreadable store is what a record would write to, and the per-source auth throttle exists to stop doing work on a flood. That made the series go quiet in the case it exists for. A rotated credential records a few unauthorized runs, the tenth failure trips the throttle, and every later request answers 429 -- so the unauthorized rate fell to zero while the integration was still broken, which an alert reads as recovered. The refusals now carry their own closed members (throttled / unavailable / rejected_body), disjoint from the recordable ones so one request is never counted twice, both halves emitting through one webhooks.count_request. The endpoint counts through a single _refused exit rather than an increment beside each return, which lets a test prove from the function's syntax tree that no branch answers without reaching the counter -- the defect was one forgotten branch, so a guarantee resting on a human having read all of them would reproduce it. No behaviour change on the endpoint: same status codes, bodies, ordering and log lines, asserted against the responses the helper replaced.
a0e23ee to
c0fa82c
Compare
buluoray
left a comment
There was a problem hiding this comment.
Reviewed head c0fa82c. Approve on the code; merging waits on the design decision for #22180.
The webhook requests counter carries only outcome, and every emit goes through webhooks.count_request and metric_outcome, which fold onto a constant set. No hook name, session key, caller credential id, body field or run detail reaches an attribute. The registered-webhooks inventory gauge has no attributes, and the subagent completed counter folds onto four fixed values after the finalize claim, so a run is counted once. The tests catch it if the folds are loosened, if a refusal branch stops counting, or if the two outcome halves overlap. Related tests pass on this head.
Follow-ups, not blocking:
- Please add the four new outcomes (throttled, unavailable, rejected_body, other) to RFC section 6.4 in #22180.
- Consider logging the unparseable hooks.json case once instead of at WARNING on every collection.
Problem / Motivation
Goal: An operator who has turned telemetry on can see how often inbound webhooks fire and how they end, how sub-agent runs finish, and how many webhook callbacks their install has registered.
Three event sources are currently invisible.
kirocrew.subagent.spawnedcounts sub-agents that start and nothing counts how they end, so "half our fan-outs are being stopped by hand" and "half of them fail" look identical. Nothing counts inbound webhook requests at all, so an operator whose integration has been failing authentication, or getting turned away at the capacity gate, has only the dashboard's 50-run history ring to scroll and no series to alert on. Andkirocrew.inventory.*publishes what an install has configured -- crons, skills, MCP servers, knowledge sources -- but not registered webhook callbacks, so a host that stopped answering its hooks is not visible next to one that never had any.Why it matters
These are the questions an operator running Crew on more than one machine actually asks, and today each needs a person to open a dashboard per host. A webhook integration that silently started returning
unauthorizedafter a token rotation produces no signal; a fan-out whose members are all dying produces a spawn count that looks healthy. Both are the kind of drift the metrics trunk exists to make visible over time, and both are cheap to answer with a counter.Not a goal
telemetry.enabledconsent gate and is a no-op without it.actorvocabulary change that accompanies these metrics in the RFC. Deferred; see the last section.webhooks.py. It is in the black baseline and stays there; the diff touches only the lines this change needs.What changed (motivation → approach → change)
Three instruments, each counted at a point that already exists, every attribute value from a closed enum, no id, name, path or free text anywhere.
kirocrew.subagent.completed{outcome}is emitted fromTerminalCoordinator._record_terminal_metric, called from_report_terminal_implpast the point the finalize claim has been granted. Counting on the two crew-log closers was rejected:subagent/completedandsubagent/failedare two closers for ONE run, and a stopped run reaches paths that could fire both. The exclusive terminal report is the only point reached exactly once however the race between the reaper and the run loop'sfinallyresolves. It sits BESIDE_record_crew_log_terminalrather than inside it, because that method returns early when the crew log is off, which would narrow the population to installs that happen to keep one.kirocrew.webhook.requests{outcome}counts every inbound request the endpoint DECIDES, which is wider than the runs the history ring stores, and that boundary is the substance of this instrument. Recorded runs are counted inWebhookRunStore.record, outside its owntry, since the series counts the decision and not the history row.That alone is not the whole population.
api_hooks_agentturns a request away in 21 places before any run exists, and for two of them recording is impossible or self-defeating: the unreadable store is the thing a record would write to, and the per-source auth throttle exists to stop doing work on a flood. The chain that matters: a rotated credential produces a fewunauthorizedruns, the tenth failure trips the throttle, and every later request answers 429. Uncounted, theunauthorizedrate falls to zero while the integration is still broken, which an alert reads as recovered -- the opposite of this metric's purpose.So the refusals carry their own closed members, disjoint from the recorded ones by construction and pinned by a test so one request is never counted under both:
throttled,unavailableandrejected_body. A revoked source keeps the recordedunauthorizedmember, being a credential this build judged. Both halves emit through one function,webhooks.count_request.Each refusal gets one
count_requestline immediately above its own untouchedreturn. A shared helper that both counted and answered was tried first and is wrong here: the error-code contract gate requires a literalcodein a dict literal at thejson_responsesite, and a helper taking the body as a parameter can only pass a variable, so the gate correctly read it as a new uncoded error response. Completeness therefore rests on a test rather than on the shape:test_every_refusal_branch_is_countedreads the function's syntax tree and fails if any answer reaches a caller without reaching the counter, with the accepted path as the single permitted exception. That is the real guard for this defect class -- the bug was one forgotten branch among twenty-one. The accepted path is deliberately uncounted at admission, since its run records a terminal outcome later.kirocrew.inventory.webhooks.registeredis an observable gauge read from live state on the reader's own ticker thread, never an up/down counter that drifts when a process dies. Its reader reduces throughwebhooks.registered_context_count, which also owns the shape test for "is this value a registration" --hooks.jsonis shared with the script-hook store, whosehookskey holds a LIST, so a second spelling of that filter would let the gauge report a number the dashboard's own listing contradicts. A missing file is a genuine 0; a file present and unparseable yields no observation and incrementskirocrew.inventory.probe.failures{probe=webhooks}, the same fault-versus-gap splitcrons.jsondraws.Not a product-shape change: no default moves and no user-facing capability changes. Each instrument is an in-memory add behind the existing consent gate; nothing new runs on the crew-log append thread or the turn path.
Rows for all three go into the "Instrumented signals" table in
docs/system-specs/modules/metrics.md, directly afterkirocrew.subagent.spawned; no other row is moved or edited.Backwards compatibility
Compatible: no field becomes required, no validator tightens, no type narrows, no stored shape changes.
metric_outcome,count_request,is_context_registrationandregistered_context_countare additive, andWebhookRunStore.recordkeeps its signature, return value and existing refusal of an unknown outcome. The refusal members are metric-only and never written to a stored row, solist_runs' display filter and every reader of the history file are untouched.The webhook endpoint's observable behaviour is unchanged: every refusal keeps its status code, response body, ordering and log line, and no store read was added.
test_webhooks_api.pyandtest_webhooks_auth_surface.pypass unmodified.Removes nothing: two names are dropped and neither has a reader. The module-level
data_homeimport inhooks_webhook_intake.pybecame unused when_hook_store_pathstarted resolving throughwebhooks.context_registrations_path();grepacrosssrc/andtest/on origin/main finds nohooks.data_homeorhooks_webhook_intake.data_homereader, and the hooks facade'sTYPE_CHECKINGmanifest entry for it is removed in the same commit, whichtest_hooks_handler_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkersenforces.hooks_webhooks.py::_is_context_registrationis deleted; its two call sites, both in that same file, now callwebhooks.is_context_registrationdirectly, and it had no tests and no reader outside the facade export removed with it.Tests
test/test_webhooks_request_counter.py(new, 29 tests) owns the request series across its whole population: the refusal members are disjoint from the recordable ones andotheris not recordable, so no request counts twice; the enum is exactly the two halves plus the remainder; every declared member survives the fold and an unknown value, empty string,None, an int and a bare object all land onother;count_requestadds exactly one, writes no run, and does not raise when the recorder is wedged; the throttled 429, the unreadable-store 503 and a signed caller'srejected_body400 are each driven through the realapi_hooks_agentasserting outcome, status and responsecode; an invalid bearer, which DOES record a run, counts exactly once, proving the emitters do not overlap;test_every_refusal_branch_is_countedandtest_every_counted_refusal_names_a_declared_memberwalk the endpoint's syntax tree to prove all 21 are counted and each names a module constant.test/metrics/test_business_counters.pydrives the realWebhookRunStore.recordagainst a temp dir: each of the six recorded outcomes is one increment; the four refusal outcomes stay distinguishable; an unknown outcome folds tootherANDrecordrefuses it before counting;detail,name, the hook id, session key and token id never reach an attribute; a history write raisingOSErrorstill counts the request; a disabled recorder records nothing.TestSubagentCompletedcovers each terminal outcome, a stopped run never reported as a completion, an unreadable outcome folding tounknown, a record whoseoutcomeproperty RAISES still counted once, an AST check that the emit is not gated on the crew log and is called exactly once, and the enum harvested fromSubagentInfo.outcome's own return literals.test/metrics/test_inventory_gauges.py: the count is the registrations only, with the script-hook store's list entries sharing the file not counted; missing file and empty object are both a genuine 0 with no probe failure; a corrupt file isNoneAND a counted fault; a top-level list is 0, not a fault; through a real in-memory SDK pipeline a corrupt file produces no observation, raises nothing in the callback and publishes the failure onprobe.failures{probe=webhooks}; the gauge agrees with the dashboard's own_read_hook_registrations; the new gauge publishes with no attributes at all.Both new emit sites are in the existing
TestAttributeValuesAreBoundedAST sweep, which fails on an interpolated attribute value.Run locally and green: the three files above plus
test_webhooks_api.py,test_webhooks_api_read_and_tokens.py,test_webhooks_store.py,test_webhooks_auth_surface.py,test_error_code_contract.py,test_subagent_parts_composition_contract.py,test_subagent_coverage_completion.py,test_hooks_handler_facade.py,test_lazy_data_home_paths.py,test_facade_mirror_guards.py.flake8,isort,scripts/check_black_formatting.py,scripts/check_comment_history.pyandscripts/docs_lint.pyare clean.Manual verification
N/A -- unit coverage is sufficient and is the stronger evidence here: each instrument is driven through its real production call site (the run store's
record, the realapi_hooks_agentfor the refusals, the terminal coordinator's exclusive report, a real OTEL in-memory reader for the gauge), so there is no integration path a manual run would exercise that the tests do not.Related Issues
Implements section 6.4 of the crew usage metrics RFC (wave 2), tracked under #21270.
Checklist
Notes on scope
The
actorvocabulary change in RFC section 6.4 is deliberately NOT in this PR. The RFC proposes addingwebhookandheartbeatto the closedactorenum ofturn/started, on the premise that those wakes currently fall intoother. They do not:crew_log.on_turn_startedhas exactly one call site, in the dashboard turn runner, and neither the webhook turn path (which runs in an ephemeralhook:-prefixed session) nor the heartbeat turn path drives its provider through that runner. Both write no crew-log entry at all, so there is noturn/startedfor either and nothing folds toothertoday. Adding the two values alone would change nothing observable, and making those paths emitturn/startedintroduces turn entries for sessions that have nosession/openedand lands on the session and turn folds -- a separate reviewable change.One sibling still spells the registrations path itself.
webhooks.context_registrations_path()is the owner for the readers this PR touches (the dashboard handler and the new gauge), but theregister_hookwriter inmcp_tools/control.pybuildsconfig_dir() / "hooks.json"directly. Both resolve to the same file; repointing a write path was out of scope here, so the claim is "one owner for these readers", not for every caller in the tree.