Repository navigation
feat(metrics): count monitor loop starts, ticks, stops and decisions - #22217
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that Suggestions
[DESIGN-REVIEWED] e503d11 Description read[DESCRIPTION-READ] e4b8931b081d269b72c2047d7ee15b4fdf1cfc2b5e52f0b9ebd02a2b5936090a That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Before merge, check one thing: within What this change shipsInventory (8 items) — 8 justifiedIntent: let an operator see whether their monitor loops do useful work or just tick quietly, and why each one stopped. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] e503d11 Description read[DESCRIPTION-READ] 52176937bfaacfcdbe12752ab7d2d3cbbcce1a0f0bec652590fe50bbad325074 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
66729ac to
5b17801
Compare
5b17801 to
f38a57d
Compare
|
Answering the First Principles advisory on the The name is not chosen here for symmetry. It is what the metric table in section 6.2 of the usage-metrics RFC specifies for this counter's What the relabel does and does not touch is worth stating plainly, because the finding's concern is an operator matching two surfaces:
The other item in that verdict was a real defect and is fixed rather than answered: the earlier hand-written table had lost |
f38a57d to
5c7e99c
Compare
buluoray
left a comment
There was a problem hiding this comment.
Reviewed head 5c7e99c. Requesting changes for one counting defect. The privacy and cardinality side is sound.
Must fix: loops.stopped can count more stops than loops.started counts starts, which contradicts the exactly-once claim in metrics/monitor_loops.py lines 29 to 37, the comment at autonudge_service/store.py line 273 and the spec row. A pause commits active to inactive, so the store commit point emits a stop. A resume goes through update with active=True, the revived branch at autonudge_service/mutations.py line 1168, and that branch never calls record_loop_started (the only callers are mutations.py L486 and monitor_records.py L323). A later stop then counts a second time. restore_monitor_after_failed_session_close in monitor_records.py near line 692 has the same shape. Driving add, pause, resume and a cycle_cap stop through AutoNudgeService on this head records one start and two stops (manual, then cycle_cap). The Issue Radar and Research Lab reconcilers revive inactive loops on every pass, so in production the gap grows over time. This matches the Opus lane finding on this head. Please either count a start on a revival and on the failed-close restore, with a test for pause, resume and stop, or drop the exactly-once wording and say a stop is counted per active to inactive transition.
Verified and fine: every attribute is a closed set (kind, outcome, the 29 reason values built from source constants and MonitorOutcome, ran, decision, and the registered monitor kinds). No loop id, slot key, target or free-form reason is emitted. Mutating stop_reason or the tick outcome fold turns the matching tests red. _emit swallows recorder failures, so neither a tick nor a store commit can fail because of the counter, and the related autonudge tests pass (655).
Follow-ups, not blocking:
- Add raising-recorder tests for a tick and for a stop, next to the existing ones for a decision and an arm.
- RFC 22180 is still a draft, so the runtime_cap name is not yet fixed by an accepted decision.
- monitoring/shadow.py also calls decide_monitor. Nothing calls it in production today, but if it is wired in, its evaluations will be counted too.
|
Addressed the loops.started / loops.stopped pairing finding in c74a06d.
|
Four counters so an operator can tell a monitor loop doing work from one
ticking quietly, which the live-loop gauge cannot answer:
- kirocrew.autonudge.loops.started{kind} at both creation mutations' commit
points -- the prompt arm and the structured monitor arm -- so a restart that
restores persisted loops does not re-count, and a structured monitor's stop
is not counted without a matching arm.
- kirocrew.autonudge.ticks{kind,outcome} at the probe gate's quiet branch and
at the fire cycle, keeping a refused delivery apart from a raising one. The
quiet emit sits inside the still-live guard, so a terminal verdict that
retires the watch is counted as a stop and not also as a tick.
- kirocrew.autonudge.loops.stopped{kind,reason,ran} from the store's single
stop-record commit point, which counts a stop once however it was reached.
- kirocrew.monitor.decisions{decision,kind} once per decide_monitor call, with
its closed value set derived from MonitorDecision rather than restated.
Every attribute value is a closed enum: a stored stop reason is mapped to a
closed value and a monitored kind is checked against the registry, so no
free text, id, slot key, target or detail string becomes an attribute. A
duration is a counter with a closed bucket attribute, not a histogram.
6e2cb5e to
bb28d1b
Compare
|
runtime_cap renamed to runtime_budget, the stored spelling, so the label no longer depends on an unmerged RFC. Rebased onto main; new head bb28d1b. |
|
HOLD (AI review concerns not answered) at bb28d1b. Both r114 blockers are addressed: The Design Review and First Principles lanes both posted CONCERNS on this head after your last comment, and neither has a reply:
Taking starts at the commit point (ids active in A maintainer still needs to confirm the data scope, as #21270 requires for this telemetry wave. — crew-pr-reviewer (r115) |
|
Reply to HOLD r115, at head e503d11 (the fix commit squashed into the previous one, to stay within the two-commit limit; same tree as 315a038).
The spec row in Data scope: still waits on a maintainer, as #21270 requires for this telemetry wave. |
…comes loops.started was only counted on the two arm paths, while loops.stopped is counted at the store commit point once per active-to-inactive transition. A pause followed by a resume and a later stop recorded one start and two stops, and the reconcilers that revive inactive loops on every pass widened the gap over time. Count starts at the same commit point, from the same baseline: an id active in the rows just written and absent from the last committed store is a start, the mirror of the stop diff. No mutation names the counter, so the arms, a revival through update(), the restore after a failed session close, a replacement rollback and any activation path added later are all counted after their write commits. _load seeds the baseline from rows active on disk, so a restart counts nothing, and a row resumed from the torn-deactivation shape (inactive on disk) counts once at its first active write. Rows the load could not parse ride every write but are never served, so they are skipped. A structured monitor that stops on a provider verdict stores the observation's reason code (pull_request_merged, pull_request_closed, monitor_blocked) as its stop reason, and none of those is in STOP_REASONS, so the most common monitor stop was reported as other. When the stored reason maps to other, fall back to the row's terminal MonitorOutcome, read from the committed rows at the same commit point. A fire callback that removes its loop and then returns False was counted as a refused tick and as a stop. Count refused only while the loop is still live, the same guard the quiet branch uses. Tests drive add, pause, resume and a cycle_cap stop, a session-close retire, restore and stop, an active replacement rollback, a torn-row boot resume, an activation through a bare field write and save that names no counter, an unparsed active row, a merged and a closed pull request monitor stop through apply_monitor_probe, and a callback that removes its loop, all through AutoNudgeService. Also adds the raising-recorder tests for a tick and a stop.
315a038 to
e503d11
Compare
|
HOLD at The r115 items are fixed. Starts and stops now come from one diff at the store commit point (store.py#L259-L329), so a new activation path is counted without naming the counter, and the Blockers:
— crew-pr-reviewer (r116) |
Problem / Motivation
Goal: an operator can see whether their monitor loops are doing useful work or just ticking quietly, and why each one stopped.
Today the only monitor-loop signal is
kirocrew.inventory.monitor_loops.active, a gauge of how many loops exist right now. It cannot distinguish two very different hosts: one with four loops that wake a session every interval, and one with four loops that have probed the same quiet pull request for a week. Nothing counts how often a loop is armed, how often a tick spends a model turn, how a loop ended, or what the structured monitor controller decided. A loop that stopped hours ago on a spent runtime budget looks, from the metrics, exactly like one that merged its subject successfully.Why it matters
The probe gate exists to save a model turn when a watched subject has not changed, and nobody can currently tell whether it is earning the subprocess it costs: the quiet-to-delivered ratio is not recorded anywhere. The same blind spot hides the failure modes operators actually hit: loops standing down because cycles keep failing, loops burning their whole cycle cap on a wedged fire callback, loops stopped by an unanswered approval. Each of those writes a log line that the next boot rotates away, so after the fact there is no number to look at.
Not a goal
kirocrew.inventory.monitor_loops.activeis untouched.telemetry.enabledgate and the existing attribute sanitiser.What changed (motivation -> approach -> change)
Goal to approach. The questions above are all counting questions, so the answer is counters, and the only design choice that matters is WHERE each one is emitted: an emit in the wrong place either double-counts, or counts a different population than its name claims. Each counter is therefore placed at the point its event is observable exactly once, and the reasoning is recorded next to the call.
kirocrew.autonudge.loops.started{kind}counts every inactive-to-active transition, so starts and stops stay paired (a loop paused, resumed and stopped records two starts and two stops). It is taken at the same store commit point and from the same baseline asloops.stopped(autonudge_service/store.py::_record_starts_committed, called from_record_stops_committed): an id active in the rows just written and absent from the last committed store is a start, the mirror of the stop diff. No mutation names the counter, so the prompt arm, the structured arm, a revival throughupdate(active=True)(a user resume, and the Issue Radar and Research Lab reconcilers re-arming an inactive loop on every pass), the restore after a failed session close, a replacement rollback that writes the displaced row back active, and any activation path added later are all counted the same way, after their write has committed. The counter therefore counts active periods, not creations, anddocs/system-specs/modules/metrics.mdsays the same._loadseeds the baseline from rows active on disk, so a gateway booting with nine loops adds nothing; a row the load resumes from the torn-deactivation shape was inactive on disk, so it counts once at its first active write and its later stop is paired; a row the load could not parse rides every write but is never served, so it is skipped. A refused admission, a 409 on an existing record, and a failed persist commit nothing and leave it untouched. Loop creation writes no crew log entry, so a counting fold over the log (the preferred source) is not available for this one.kirocrew.autonudge.ticks{kind,outcome}has two emit sites inautonudge_service/firing.pythat partition the ticks which reach a verdict about spending a turn: the probe gate's quiet branch in_timerrecordsquiet, and_run_fire_cyclerecordsdelivered,refusedorfailedimmediately after the fire callback settles.refusedis recorded only while the loop is still live: a callback that removes the loop and then returns False (the session is gone, the channel is unsupported) is counted once, as the stop.refused(the callback declined, because the slot was mid-turn) is kept apart fromfailed(the callback RAISED), because a refusal is the system working as designed while an exception is a defect; folded together, a wedged fire callback is indistinguishable from a busy session. A tick that returns before deciding about a turn, such as a spent cycle cap, a stand-down or a hold for a person, records nothing here: it decided about the LOOP, whichloops.stoppedalready reports, and counting it would put events that were never candidates for a turn into the denominator the quiet-to-delivered ratio is read against. Thequietemit therefore sits INSIDE theloop.active and loop.id in self._loopsguard: a terminal verdict ("the watched pull request merged") also answers quiet while having just deactivated the loop, so emitting above the guard would report one event twice, once as a tick and once as a stop.kirocrew.autonudge.loops.stopped{kind,reason,ran}is emitted from the store's one stop-record commit point (autonudge_service/store.py::_record_stops_committed), from the same record the WARNING line and the durable stop file are written from. Exactly-once comes from that record's own definition rather than from a new guard: a record exists only for a loop that was active in the last committed store and is not active now, so a secondautonudge_stopon an already-stopped loop produces no record, a restart does not re-report a stop already counted, and the dozen paths that can stop a loop cannot double-count between them. A structured monitor that stops on a provider verdict stores the provider's own reason code (pull_request_merged,pull_request_closed, themonitor_blockeddefault) as its stop reason; those codes are per-provider and outside the closed set, so when the stored reason maps tootherthe counter reports the row's terminalMonitorOutcomeinstead (success,blocked). The durable stop file (autonudge_stops.jsonl) keeps the same records, but it is one host's local file, rotated at 1 MiB with one generation, and is not exported: this counter is what carries stop reasons through the metrics exporter, so they can be read across hosts and over time.kirocrew.monitor.decisions{decision,kind}is emitted once permonitoring/decision.py::decide_monitorcall, at the one place every path through the policy converges, AFTER the stall streak is folded, so what is counted is the verdict the caller acts on rather than an effect a later step may still replace. A pure policy module can carry this because recording is an in-memory add that reads no clock, file or socket and swallows its own failures, so the module stays deterministic for its callers and its tests. The population is POLICY EVALUATIONS, which is narrower than every verdict a caller acts on:monitor_records.py::apply_monitor_probeshort-circuits some ticks to a verdict of its own (a record whose outcome is already written, a reading it refuses) without consulting the policy, and those are deliberately uncounted, so the series answers "the policy was asked, and this is what it said".monitoring/shadow.py::run_shadow_probealso calls the policy and would be counted the same way; it has no caller in the gateway today.What
ticksdoes not observe, stated rather than left to be discovered. A structured monitor record is driven by its own typed controller (firing.pyhands it to_on_monitor_tickand returns) and reaches neither tick emit site, so it contributes nothing toticks; its per-tick verdict ismonitor.decisions, which is the controller's own decision and the only honest reading of what that path decided (firing.py cannot see whether a wake landed). The consequence to know before building a dashboard: withinticks,kind=monitormeans a probe-gated PROMPT loop, which is a narrower population thankind=monitoron the started and stopped counters, so a per-loop rate must not be taken asticksoverloops.startedfor that kind. The quiet-to-delivered ratio is read insideticksagainst its own population and is unaffected. Both halves are pinned by tests and written into the spec row rather than left implicit.Keeping the attributes closed, and every closed set DERIVED from what produces it.
STOP_REASONSis assembled from the constants and the enum that write those strings -- the*_REASONconstants inautonudge_service/model.py, theMONITOR_STOP_*constants inmonitoring/models.py, the terminalMonitorOutcomea monitor row falls back to when it carries no stop reason, andautonudge_stop_log's two defaults -- andMONITOR_DECISION_VALUESfromMonitorDecision. Neither is retyped beside its source, because a hand-written copy could only ever catch a new member by reporting it asotherwhile reading as though someone had checked: an earlier retyped stop-reason table in this PR had already lostMonitorOutcome.TARGET_UNAVAILABLE, which is exactly that failure, and two tests now pin the coverage from the source side. A stored reason still never travels as itself (it is model-authored on some paths and composed at the call site on others,campaign_<status>/orphaned_worker), and anything unrecognised becomesother. A monitored kind is checked againstmonitoring/registry.py, the catalogue every boundary that gates a kind already reads, so a kind from a row written by a newer gateway reportsotherrather than minting a series. A spent runtime budget is reported asruntime_budget, the stored spelling the WARNING line and the stop file also carry. One stored reason is reported under a different spelling, spelled out because a reported value differing from the stored string must be findable in one place: the legacyapproval_stalledfolded in with the structuredapproval_stall, which is the same event recorded by the two record shapes. No loop id, slot key, target, session key or free-text detail is an attribute; tests assert each of those absences.A runtime without a histogram.
ranis a closed attribute with edges stated in code and in the docs table (lt_1hunder 3600s,1h_24hfrom 3600s up to but not including 86400s,gt_24hat or above 86400s), so the distribution survives a backend that keeps no explicit histogram buckets. A row whose creation time was missing or unreadable reportsran=otherrather than being folded intolt_1h, which would claim a short life there is no evidence for.Everything sits behind the existing
telemetry.enabledgate (a disabled recorder is a no-op) and the existing attribute sanitiser, and every emit swallows its own failures after a debug log.Backwards compatibility
Compatible: the diff only adds a module and its counter emits; no existing metric, attribute, field, validator, schema or signature changes, and no caller is asked for anything new.
Removes nothing: no path is hidden, no record or row is deleted, no validator is tightened, no data is migrated or pruned, and no key or public function is removed or renamed. The only rename is internal to this PR's own new mapping (which closed value a stored reason is REPORTED as); the stored
runtime_budgetandapproval_stalledstrings on disk, and every reader of them, are untouched.Tests
test/metrics/test_monitor_loop_counters.py(80 cases), driven through the production call sites rather than by calling the emit helpers where a real path exists. The new module is at 100% statement and branch coverage.loops.started: add, pause, resume and a cycle_cap stop record 2 starts and 2 stops; a session-close retire, restore and stop record 2 and 2; an active replacement rollback records 3 and 3; one prompt arm counts once withkind=loop; a structuredadd_monitorarm counts once withkind=monitor; an arm and itsstop_monitorbalance one-to-one on both counters, with the user's free-text stop reason absent from the attributes; a restart that restores two persisted loops counts nothing (the property that makes this a count of arms); a refused admission counts nothing; an activation through a bare field write and save, which names no counter, is still counted and balances its stop (the test a future activation path cannot slip past); a torn-resumed row counts once and a second restart adds nothing; an unparsed active row is not a start; no loop id or slot key appears in the attributes.ticks: a quiet gated tick countsquietwithkind=monitor; a delivered tick countsdelivered; a callback returning False countsrefused; a callback that RAISES countsfailed; a callback that removes its loop and returns False counts a stop and no tick; a tick that stopped the loop on a spent cycle cap counts no tick at all; a terminal quiet verdict counts a stop and no tick, so one event is not reported twice; a structured monitor record's tick reaches no emit site, pinning the documented scope above; an unrecognised outcome becomesother.loops.stopped: a stop through the store counts once withreason=cycle_cap; a merged and a closed pull-request monitor, stopped throughapply_monitor_probewith the real PR classifier's reason code, reportsuccessandblocked; a known stored reason wins over the outcome, and an unknown reason with no valid outcome staysother; stopping an already-stopped loop and then removing it leaves the count at one; a restart does not re-report it; every stop-reason constant and everyMonitorOutcomemember is covered, asserted from the source side so the table cannot drift again; 13 parametrized spellings map to their closed value, includingruntime_budgetto itself and both approval spellings; an unknown reason becomesother(6 cases, including a composedcampaign_*,orphaned_worker, a free-text reason carrying a path, empty, None and a non-string); the record's free-textdetailnever reaches the attributes; eachranbucket edge (0, 3599, 3600, 3601, 86399, 86400, 86401, driven through the real stop-record differ); a row with no creation time reportsran=other; a monitor row is labelledmonitor; an unknown kind becomesother.monitor.decisions: every member ofMonitorDecisionis reached by a real state-and-observation pair throughdecide_monitorand counted under its own name (7 parametrized cases, with an import-time assertion that the case table covers the enum); two calls count twice; the counter's value set is asserted to equal the enum's; an unregistered kind becomesother; the target never reaches the attributes.decide_monitorcall against a no-op recorder creates no instrument at all; a recorder that raises breaks neither the decision nor the arm.Existing suites for the touched modules re-run green (321 cases with the new file):
test/test_monitor_decision.py,test/test_autonudge_firing.py,test/test_autonudge_add_replace.py,test/test_autonudge_stop_log.py,test/test_autonudge_stop_file.py,test/test_autonudge_gating.py,test/test_monitor_controller.py.Manual verification
N/A, unit coverage is sufficient: every counter is driven through its own production call site (both real arms, the real timer, the store commit and the decision policy), including the restart, double-stop and terminal-verdict paths, so there is no behaviour left that only a live gateway would exercise.
Screenshots / video
N/A, no frontend surface is touched.
Related Issues
Refs #21270
Checklist