Skip to content

feat(metrics): count monitor loop starts, ticks, stops and decisions - #22217

Merged
iamwhatever merged 2 commits into
mainfrom
feat/22180-metrics-monitor-loops
Oct 21, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
feat/22180-metrics-monitor-loops

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: an operator can see whether their monitor loops are doing useful work or just ticking quietly, and why each one stopped.

Today the only monitor-loop signal is kirocrew.inventory.monitor_loops.active, a gauge of how many loops exist right now. It cannot distinguish two very different hosts: one with four loops that wake a session every interval, and one with four loops that have probed the same quiet pull request for a week. Nothing counts how often a loop is armed, how often a tick spends a model turn, how a loop ended, or what the structured monitor controller decided. A loop that stopped hours ago on a spent runtime budget looks, from the metrics, exactly like one that merged its subject successfully.

Why it matters

The probe gate exists to save a model turn when a watched subject has not changed, and nobody can currently tell whether it is earning the subprocess it costs: the quiet-to-delivered ratio is not recorded anywhere. The same blind spot hides the failure modes operators actually hit: loops standing down because cycles keep failing, loops burning their whole cycle cap on a wedged fire callback, loops stopped by an unanswered approval. Each of those writes a log line that the next boot rotates away, so after the fact there is no number to look at.

Not a goal

  • No change to what any loop, monitor or tick DOES. Every emit site is an in-memory counter add placed after the decision it reports; no control flow, default, bound or persisted field changes.
  • No new histogram. A runtime is a counter with a closed bucket attribute instead.
  • No gauge changes: kirocrew.inventory.monitor_loops.active is untouched.
  • No export, destination or dashboard work. These counters ride the existing recorder, the existing telemetry.enabled gate and the existing attribute sanitiser.
  • No "stale loop" detection (a loop that has not ticked for hours). That needs a sampler and is not here.

What changed (motivation -> approach -> change)

Goal to approach. The questions above are all counting questions, so the answer is counters, and the only design choice that matters is WHERE each one is emitted: an emit in the wrong place either double-counts, or counts a different population than its name claims. Each counter is therefore placed at the point its event is observable exactly once, and the reasoning is recorded next to the call.

  • kirocrew.autonudge.loops.started{kind} counts every inactive-to-active transition, so starts and stops stay paired (a loop paused, resumed and stopped records two starts and two stops). It is taken at the same store commit point and from the same baseline as loops.stopped (autonudge_service/store.py::_record_starts_committed, called from _record_stops_committed): an id active in the rows just written and absent from the last committed store is a start, the mirror of the stop diff. No mutation names the counter, so the prompt arm, the structured arm, a revival through update(active=True) (a user resume, and the Issue Radar and Research Lab reconcilers re-arming an inactive loop on every pass), the restore after a failed session close, a replacement rollback that writes the displaced row back active, and any activation path added later are all counted the same way, after their write has committed. The counter therefore counts active periods, not creations, and docs/system-specs/modules/metrics.md says the same. _load seeds the baseline from rows active on disk, so a gateway booting with nine loops adds nothing; a row the load resumes from the torn-deactivation shape was inactive on disk, so it counts once at its first active write and its later stop is paired; a row the load could not parse rides every write but is never served, so it is skipped. A refused admission, a 409 on an existing record, and a failed persist commit nothing and leave it untouched. Loop creation writes no crew log entry, so a counting fold over the log (the preferred source) is not available for this one.
  • kirocrew.autonudge.ticks{kind,outcome} has two emit sites in autonudge_service/firing.py that partition the ticks which reach a verdict about spending a turn: the probe gate's quiet branch in _timer records quiet, and _run_fire_cycle records delivered, refused or failed immediately after the fire callback settles. refused is recorded only while the loop is still live: a callback that removes the loop and then returns False (the session is gone, the channel is unsupported) is counted once, as the stop. refused (the callback declined, because the slot was mid-turn) is kept apart from failed (the callback RAISED), because a refusal is the system working as designed while an exception is a defect; folded together, a wedged fire callback is indistinguishable from a busy session. A tick that returns before deciding about a turn, such as a spent cycle cap, a stand-down or a hold for a person, records nothing here: it decided about the LOOP, which loops.stopped already reports, and counting it would put events that were never candidates for a turn into the denominator the quiet-to-delivered ratio is read against. The quiet emit therefore sits INSIDE the loop.active and loop.id in self._loops guard: a terminal verdict ("the watched pull request merged") also answers quiet while having just deactivated the loop, so emitting above the guard would report one event twice, once as a tick and once as a stop.
  • kirocrew.autonudge.loops.stopped{kind,reason,ran} is emitted from the store's one stop-record commit point (autonudge_service/store.py::_record_stops_committed), from the same record the WARNING line and the durable stop file are written from. Exactly-once comes from that record's own definition rather than from a new guard: a record exists only for a loop that was active in the last committed store and is not active now, so a second autonudge_stop on an already-stopped loop produces no record, a restart does not re-report a stop already counted, and the dozen paths that can stop a loop cannot double-count between them. A structured monitor that stops on a provider verdict stores the provider's own reason code (pull_request_merged, pull_request_closed, the monitor_blocked default) as its stop reason; those codes are per-provider and outside the closed set, so when the stored reason maps to other the counter reports the row's terminal MonitorOutcome instead (success, blocked). The durable stop file (autonudge_stops.jsonl) keeps the same records, but it is one host's local file, rotated at 1 MiB with one generation, and is not exported: this counter is what carries stop reasons through the metrics exporter, so they can be read across hosts and over time.
  • kirocrew.monitor.decisions{decision,kind} is emitted once per monitoring/decision.py::decide_monitor call, at the one place every path through the policy converges, AFTER the stall streak is folded, so what is counted is the verdict the caller acts on rather than an effect a later step may still replace. A pure policy module can carry this because recording is an in-memory add that reads no clock, file or socket and swallows its own failures, so the module stays deterministic for its callers and its tests. The population is POLICY EVALUATIONS, which is narrower than every verdict a caller acts on: monitor_records.py::apply_monitor_probe short-circuits some ticks to a verdict of its own (a record whose outcome is already written, a reading it refuses) without consulting the policy, and those are deliberately uncounted, so the series answers "the policy was asked, and this is what it said". monitoring/shadow.py::run_shadow_probe also calls the policy and would be counted the same way; it has no caller in the gateway today.

What ticks does not observe, stated rather than left to be discovered. A structured monitor record is driven by its own typed controller (firing.py hands it to _on_monitor_tick and returns) and reaches neither tick emit site, so it contributes nothing to ticks; its per-tick verdict is monitor.decisions, which is the controller's own decision and the only honest reading of what that path decided (firing.py cannot see whether a wake landed). The consequence to know before building a dashboard: within ticks, kind=monitor means a probe-gated PROMPT loop, which is a narrower population than kind=monitor on the started and stopped counters, so a per-loop rate must not be taken as ticks over loops.started for that kind. The quiet-to-delivered ratio is read inside ticks against its own population and is unaffected. Both halves are pinned by tests and written into the spec row rather than left implicit.

Keeping the attributes closed, and every closed set DERIVED from what produces it. STOP_REASONS is assembled from the constants and the enum that write those strings -- the *_REASON constants in autonudge_service/model.py, the MONITOR_STOP_* constants in monitoring/models.py, the terminal MonitorOutcome a monitor row falls back to when it carries no stop reason, and autonudge_stop_log's two defaults -- and MONITOR_DECISION_VALUES from MonitorDecision. Neither is retyped beside its source, because a hand-written copy could only ever catch a new member by reporting it as other while reading as though someone had checked: an earlier retyped stop-reason table in this PR had already lost MonitorOutcome.TARGET_UNAVAILABLE, which is exactly that failure, and two tests now pin the coverage from the source side. A stored reason still never travels as itself (it is model-authored on some paths and composed at the call site on others, campaign_<status> / orphaned_worker), and anything unrecognised becomes other. A monitored kind is checked against monitoring/registry.py, the catalogue every boundary that gates a kind already reads, so a kind from a row written by a newer gateway reports other rather than minting a series. A spent runtime budget is reported as runtime_budget, the stored spelling the WARNING line and the stop file also carry. One stored reason is reported under a different spelling, spelled out because a reported value differing from the stored string must be findable in one place: the legacy approval_stalled folded in with the structured approval_stall, which is the same event recorded by the two record shapes. No loop id, slot key, target, session key or free-text detail is an attribute; tests assert each of those absences.

A runtime without a histogram. ran is a closed attribute with edges stated in code and in the docs table (lt_1h under 3600s, 1h_24h from 3600s up to but not including 86400s, gt_24h at or above 86400s), so the distribution survives a backend that keeps no explicit histogram buckets. A row whose creation time was missing or unreadable reports ran=other rather than being folded into lt_1h, which would claim a short life there is no evidence for.

Everything sits behind the existing telemetry.enabled gate (a disabled recorder is a no-op) and the existing attribute sanitiser, and every emit swallows its own failures after a debug log.

Backwards compatibility

Compatible: the diff only adds a module and its counter emits; no existing metric, attribute, field, validator, schema or signature changes, and no caller is asked for anything new.

Removes nothing: no path is hidden, no record or row is deleted, no validator is tightened, no data is migrated or pruned, and no key or public function is removed or renamed. The only rename is internal to this PR's own new mapping (which closed value a stored reason is REPORTED as); the stored runtime_budget and approval_stalled strings on disk, and every reader of them, are untouched.

Tests

test/metrics/test_monitor_loop_counters.py (80 cases), driven through the production call sites rather than by calling the emit helpers where a real path exists. The new module is at 100% statement and branch coverage.

  • loops.started: add, pause, resume and a cycle_cap stop record 2 starts and 2 stops; a session-close retire, restore and stop record 2 and 2; an active replacement rollback records 3 and 3; one prompt arm counts once with kind=loop; a structured add_monitor arm counts once with kind=monitor; an arm and its stop_monitor balance one-to-one on both counters, with the user's free-text stop reason absent from the attributes; a restart that restores two persisted loops counts nothing (the property that makes this a count of arms); a refused admission counts nothing; an activation through a bare field write and save, which names no counter, is still counted and balances its stop (the test a future activation path cannot slip past); a torn-resumed row counts once and a second restart adds nothing; an unparsed active row is not a start; no loop id or slot key appears in the attributes.
  • ticks: a quiet gated tick counts quiet with kind=monitor; a delivered tick counts delivered; a callback returning False counts refused; a callback that RAISES counts failed; a callback that removes its loop and returns False counts a stop and no tick; a tick that stopped the loop on a spent cycle cap counts no tick at all; a terminal quiet verdict counts a stop and no tick, so one event is not reported twice; a structured monitor record's tick reaches no emit site, pinning the documented scope above; an unrecognised outcome becomes other.
  • loops.stopped: a stop through the store counts once with reason=cycle_cap; a merged and a closed pull-request monitor, stopped through apply_monitor_probe with the real PR classifier's reason code, report success and blocked; a known stored reason wins over the outcome, and an unknown reason with no valid outcome stays other; stopping an already-stopped loop and then removing it leaves the count at one; a restart does not re-report it; every stop-reason constant and every MonitorOutcome member is covered, asserted from the source side so the table cannot drift again; 13 parametrized spellings map to their closed value, including runtime_budget to itself and both approval spellings; an unknown reason becomes other (6 cases, including a composed campaign_*, orphaned_worker, a free-text reason carrying a path, empty, None and a non-string); the record's free-text detail never reaches the attributes; each ran bucket edge (0, 3599, 3600, 3601, 86399, 86400, 86401, driven through the real stop-record differ); a row with no creation time reports ran=other; a monitor row is labelled monitor; an unknown kind becomes other.
  • monitor.decisions: every member of MonitorDecision is reached by a real state-and-observation pair through decide_monitor and counted under its own name (7 parametrized cases, with an import-time assertion that the case table covers the enum); two calls count twice; the counter's value set is asserted to equal the enum's; an unregistered kind becomes other; the target never reaches the attributes.
  • telemetry off: a whole loop lifetime (arm, tick, stop) plus a decide_monitor call against a no-op recorder creates no instrument at all; a recorder that raises breaks neither the decision nor the arm.

Existing suites for the touched modules re-run green (321 cases with the new file): test/test_monitor_decision.py, test/test_autonudge_firing.py, test/test_autonudge_add_replace.py, test/test_autonudge_stop_log.py, test/test_autonudge_stop_file.py, test/test_autonudge_gating.py, test/test_monitor_controller.py.

Manual verification

N/A, unit coverage is sufficient: every counter is driven through its own production call site (both real arms, the real timer, the store commit and the decision policy), including the restart, double-stop and terminal-verdict paths, so there is no behaviour left that only a live gateway would exercise.

Screenshots / video

N/A, no frontend surface is touched.

Related Issues

Refs #21270

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested review from a team and KiroCrewCaptain as code owners October 19, 2026 15:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision e503d11f0e935a975c06304fb3179999c3df9b94 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of e503d11f0e935a975c06304fb3179999c3df9b94 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that kirocrew.monitor.decisions should count before the commit, from inside the pure policy, unlike the other three counters.

Suggestions

  • Move the record_monitor_decision call out of monitoring/decision.py::decide_monitor and into monitor_records.py::apply_monitor_probe, after _write_monitor_snapshot_locked succeeds. That matches the "counted after their write has committed" rule the PR gives for starts and stops. It also stops a failed persist from counting a decision nothing acted on. Finally, it keeps a policy module the description calls pure free of a metrics import, and it closes the double-count the description already flags for run_shadow_probe.

[DESIGN-REVIEWED] e503d11

Description read

[DESCRIPTION-READ] e4b8931b081d269b72c2047d7ee15b4fdf1cfc2b5e52f0b9ebd02a2b5936090a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of e503d11f0e935a975c06304fb3179999c3df9b94 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, check one thing: within ticks, kind=monitor covers only probe-gated prompt loops, while loops.started and loops.stopped also count structured records.

What this change ships

Inventory (8 items) — 8 justified

Intent: let an operator see whether their monitor loops do useful work or just tick quietly, and why each one stopped. This is an ADDITION.

  1. A new counter for each time a loop becomes active (kirocrew.autonudge.loops.started{kind}), counted at the store's commit point — justified
  2. A new counter for each tick that decides about a turn (kirocrew.autonudge.ticks{kind,outcome}), with outcomes quiet, delivered, refused and failed — justified
  3. A new counter for each loop stop (kirocrew.autonudge.loops.stopped{kind,reason,ran}), sent through the exporter rather than only the local stop file that gets rotated away — justified
  4. A new counter for each monitor policy decision (kirocrew.monitor.decisions{decision,kind}), emitted from decide_monitor — justified
  5. A structured monitor that stops on a provider's own reason code is reported under its terminal outcome (success / blocked) instead of other — justified
  6. Loop runtime is reported as a closed bucket (lt_1h / 1h_24h / gt_24h / other), not as a histogram — justified
  7. Stop reasons and decision values are built from the existing constants and enums; any unknown value becomes other, so no free text leaves the process — justified
  8. docs/system-specs/modules/metrics.md lists the four counters — justified

[FIRST-PRINCIPLES-REVIEWED] e503d11

Description read

[DESCRIPTION-READ] 52176937bfaacfcdbe12752ab7d2d3cbbcce1a0f0bec652590fe50bbad325074

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed e503d11f0e935a975c06304fb3179999c3df9b94 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e503d11

Verdict parsed from the review's SHA-scoped output markers for commit e503d11f0e935a975c06304fb3179999c3df9b94.

False positive or not applicable? A repository writer can comment:
/ai-review override fable e503d11f0e935a975c06304fb3179999c3df9b94: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of e503d11f0e935a975c06304fb3179999c3df9b94 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e503d11

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e503d11f0e935a975c06304fb3179999c3df9b94: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the feat/22180-metrics-monitor-loops branch from 66729ac to 5b17801 Compare October 19, 2026 16:35
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-metrics-monitor-loops branch from 5b17801 to f38a57d Compare October 19, 2026 17:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 19, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Answering the First Principles advisory on the runtime_budget -> runtime_cap relabel once, so it is on the record rather than re-argued each round.

The name is not chosen here for symmetry. It is what the metric table in section 6.2 of the usage-metrics RFC specifies for this counter's reason attribute: #22180. Reporting the stored runtime_budget instead would make this counter disagree with the accepted spec, and with the four sibling counters in the same section that other PRs implement against the same table. Keeping it RFC-exact is the deliberate choice; a change to the name belongs in that RFC, not in one implementing PR.

What the relabel does and does not touch is worth stating plainly, because the finding's concern is an operator matching two surfaces:

  • Nothing persisted changes. The row on disk, the AutoNudge: WARNING line and the durable stop file all still say runtime_budget.
  • metrics/monitor_loops.py::STOP_REASONS is the single place that holds the correspondence, and the two entries whose reported value differs from the stored string are spelled out there rather than derived, precisely so a reader who sees runtime_cap on a chart can find the stored name in one lookup. The spec row in docs/system-specs/modules/metrics.md says the same.
  • Every other reason is reported under its stored name, built from the *_REASON constants, the MONITOR_STOP_* constants and MonitorOutcome rather than retyped, with tests asserting coverage from the source side.

The other item in that verdict was a real defect and is fixed rather than answered: the earlier hand-written table had lost MonitorOutcome.TARGET_UNAVAILABLE, so that stop was reported as other. It now maps to itself, and the table is derived so the same omission cannot recur.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-metrics-monitor-loops branch from f38a57d to 5c7e99c Compare October 19, 2026 17:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 19, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 5c7e99c. Requesting changes for one counting defect. The privacy and cardinality side is sound.

Must fix: loops.stopped can count more stops than loops.started counts starts, which contradicts the exactly-once claim in metrics/monitor_loops.py lines 29 to 37, the comment at autonudge_service/store.py line 273 and the spec row. A pause commits active to inactive, so the store commit point emits a stop. A resume goes through update with active=True, the revived branch at autonudge_service/mutations.py line 1168, and that branch never calls record_loop_started (the only callers are mutations.py L486 and monitor_records.py L323). A later stop then counts a second time. restore_monitor_after_failed_session_close in monitor_records.py near line 692 has the same shape. Driving add, pause, resume and a cycle_cap stop through AutoNudgeService on this head records one start and two stops (manual, then cycle_cap). The Issue Radar and Research Lab reconcilers revive inactive loops on every pass, so in production the gap grows over time. This matches the Opus lane finding on this head. Please either count a start on a revival and on the failed-close restore, with a test for pause, resume and stop, or drop the exactly-once wording and say a stop is counted per active to inactive transition.

Verified and fine: every attribute is a closed set (kind, outcome, the 29 reason values built from source constants and MonitorOutcome, ran, decision, and the registered monitor kinds). No loop id, slot key, target or free-form reason is emitted. Mutating stop_reason or the tick outcome fold turns the matching tests red. _emit swallows recorder failures, so neither a tick nor a store commit can fail because of the counter, and the related autonudge tests pass (655).

Follow-ups, not blocking:

  • Add raising-recorder tests for a tick and for a stop, next to the existing ones for a decision and an arm.
  • RFC 22180 is still a draft, so the runtime_cap name is not yet fixed by an accepted decision.
  • monitoring/shadow.py also calls decide_monitor. Nothing calls it in production today, but if it is wired in, its evaluations will be counted too.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 19, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Addressed the loops.started / loops.stopped pairing finding in c74a06d.

  • loops.started now counts every inactive-to-active transition, after the committed write: the two arm paths as before, plus a revival through update(active=True) on an inactive loop, restore_monitor_after_failed_session_close, and rollback_monitor_replacement when the prior row it writes back is active (the replacement commit removed that id, which the store counted as a stop).
  • loops.stopped is unchanged: exactly once per active-to-inactive transition. The comment in metrics/monitor_loops.py, the record_loop_started docstring and both rows in metrics.md now say starts and stops are counted per transition.
  • New tests drive add, pause, resume and a cycle_cap stop (2 starts, 2 stops), a session-close retire, restore and stop (2 and 2), and an active replacement rollback (3 and 3) through AutoNudgeService. The three pairing tests fail on the previous head. Also added the raising-recorder tests for a tick and a stop.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 19, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 20, 2026
Four counters so an operator can tell a monitor loop doing work from one
ticking quietly, which the live-loop gauge cannot answer:

- kirocrew.autonudge.loops.started{kind} at both creation mutations' commit
  points -- the prompt arm and the structured monitor arm -- so a restart that
  restores persisted loops does not re-count, and a structured monitor's stop
  is not counted without a matching arm.
- kirocrew.autonudge.ticks{kind,outcome} at the probe gate's quiet branch and
  at the fire cycle, keeping a refused delivery apart from a raising one. The
  quiet emit sits inside the still-live guard, so a terminal verdict that
  retires the watch is counted as a stop and not also as a tick.
- kirocrew.autonudge.loops.stopped{kind,reason,ran} from the store's single
  stop-record commit point, which counts a stop once however it was reached.
- kirocrew.monitor.decisions{decision,kind} once per decide_monitor call, with
  its closed value set derived from MonitorDecision rather than restated.

Every attribute value is a closed enum: a stored stop reason is mapped to a
closed value and a monitored kind is checked against the registry, so no
free text, id, slot key, target or detail string becomes an attribute. A
duration is a counter with a closed bucket attribute, not a histogram.
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-metrics-monitor-loops branch from 6e2cb5e to bb28d1b Compare October 20, 2026 08:45
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

runtime_cap renamed to runtime_budget, the stored spelling, so the label no longer depends on an unmerged RFC. Rebased onto main; new head bb28d1b.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (AI review concerns not answered) at bb28d1b.

Both r114 blockers are addressed: reason now exports the stored runtime_budget, and a torn-resumed row now counts a start, so its later stop is paired. CI is green, and Opus and GPT found nothing.

The Design Review and First Principles lanes both posted CONCERNS on this head after your last comment, and neither has a reply:

  1. Design Review: starts are counted at five hand-placed sites plus pending_starts, while stops come from one commit-point diff (store.py L272-L319). A future activation path that forgets to call record_loop_started would break the pairing with no test failing.
  2. First Principles: pending_starts, _record_pending_starts_committed and torn_resumed are oversized for one rare repair.

Taking starts at the commit point (ids active in rows but not in the prior committed_active) would clear both. It also keeps the boot pairing that r114 asked for, without the extra state. Otherwise, reply to each item with why the current shape is better.

A maintainer still needs to confirm the data scope, as #21270 requires for this telemetry wave.

— crew-pr-reviewer (r115)

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 21, 2026
@chenmingwei23

chenmingwei23 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

Reply to HOLD r115, at head e503d11 (the fix commit squashed into the previous one, to stay within the two-commit limit; same tree as 315a038).

  1. Design Review (issuecomment-6271380268), starts counted at hand-placed sites: fixed. Starts are now taken at the store commit point, from the same baseline as stops (LoopStore._record_starts_committed, called from _record_stops_committed): an id active in the rows just written and absent from the prior committed_active is a start. All five per-site record_loop_started calls are removed from mutations.py and monitor_records.py; no mutation names the counter now. record_loop_started takes the commit-point summary, so a start and its stop carry the same kind from one derivation.
    New test test_an_activation_path_that_never_names_the_counter_is_counted: a loop is paused through update, then made active again by a bare field write and _save() (no mutation, no emit), then stopped. It asserts 2 starts and 2 stops, and that a second re-save adds no start. On bb28d1b this test fails (1 start against 2 stops).
    One case the plain diff would have over-counted, now handled: a row _load could not parse is written back verbatim on every commit but is not in the loaded baseline, so it would have counted one start per boot with no stop. It is skipped, and test_an_unparsed_active_row_is_not_a_start pins it.

  2. First Principles (issuecomment-6271380997), pending_starts oversized: removed. pending_starts, _record_pending_starts_committed and the torn_resumed bookkeeping in store_loader._load are gone (the loader and both mutation modules are back to their main versions). The boot pairing r114 asked for now holds by construction: _load seeds the baseline from rows active on disk, so a restart counts nothing, and a torn-resumed row was inactive on disk, so it counts once at its first active write. The existing tests for both (test_a_boot_resume_of_a_torn_row_and_its_stop_balance, test_a_second_restart_does_not_re_count_a_boot_resume) pass unchanged, as do all other pairing tests.

The spec row in metrics.md, the module docstring and the PR description are updated to match.

Data scope: still waits on a maintainer, as #21270 requires for this telemetry wave.

…comes

loops.started was only counted on the two arm paths, while
loops.stopped is counted at the store commit point once per
active-to-inactive transition. A pause followed by a resume and a later
stop recorded one start and two stops, and the reconcilers that revive
inactive loops on every pass widened the gap over time.

Count starts at the same commit point, from the same baseline: an id
active in the rows just written and absent from the last committed
store is a start, the mirror of the stop diff. No mutation names the
counter, so the arms, a revival through update(), the restore after a
failed session close, a replacement rollback and any activation path
added later are all counted after their write commits. _load seeds the
baseline from rows active on disk, so a restart counts nothing, and a
row resumed from the torn-deactivation shape (inactive on disk) counts
once at its first active write. Rows the load could not parse ride
every write but are never served, so they are skipped.

A structured monitor that stops on a provider verdict stores the
observation's reason code (pull_request_merged, pull_request_closed,
monitor_blocked) as its stop reason, and none of those is in
STOP_REASONS, so the most common monitor stop was reported as other.
When the stored reason maps to other, fall back to the row's terminal
MonitorOutcome, read from the committed rows at the same commit point.

A fire callback that removes its loop and then returns False was counted
as a refused tick and as a stop. Count refused only while the loop is
still live, the same guard the quiet branch uses.

Tests drive add, pause, resume and a cycle_cap stop, a session-close
retire, restore and stop, an active replacement rollback, a torn-row
boot resume, an activation through a bare field write and save that
names no counter, an unparsed active row, a merged and a closed pull
request monitor stop through apply_monitor_probe, and a callback that
removes its loop, all through AutoNudgeService. Also adds the
raising-recorder tests for a tick and a stop.
@chenmingwei23
chenmingwei23 force-pushed the feat/22180-metrics-monitor-loops branch from 315a038 to e503d11 Compare October 21, 2026 02:18
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at e503d11f0e935a975c06304fb3179999c3df9b94.

The r115 items are fixed. Starts and stops now come from one diff at the store commit point (store.py#L259-L329), so a new activation path is counted without naming the counter, and the pending_starts state is gone. CI is green.

Blockers:

  1. The Design Review check on this head has no reply. kirocrew.monitor.decisions is counted inside the pure decide_monitor (decision.py#L104-L119), before the probe is persisted. The other three counters count only after their write commits. Move the call into apply_monitor_probe after the snapshot write, or reply with why counting before the commit is right.
  2. The First Principles check has no reply: in ticks, kind=monitor covers only probe-gated prompt loops, while loops.started and loops.stopped also count structured records.
  3. A maintainer still needs to confirm the data scope, as OpenTelemetry: export per-module usage counts from the crew log #21270 requires.

— crew-pr-reviewer (r116)

@iamwhatever
iamwhatever merged commit 9800507 into main Oct 21, 2026
104 checks passed
@iamwhatever
iamwhatever deleted the feat/22180-metrics-monitor-loops branch October 21, 2026 07:36
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants