Skip to content

feat(monitoring): retire a watch whose verdict stopped moving - #11455

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:feat/monitor-verdict-stall-streak
Sep 17, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:feat/monitor-verdict-stall-streak

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

A structured monitor that has nothing left to learn keeps probing anyway. It
reaches the same conclusion tick after tick -- the same failing check, already
alerted, suppressed by dedupe -- and nothing notices. It runs until a budget
retires it, up to four hours at the defaults.

The module spec has named this gap in its own present tense for a while:
"No engine state holds a streak of identical verdicts today." Streak
counters exist and count other things. quiet_streak counts quiet observations,
consecutive_provider_errors counts provider failures. None of them notices a
watch that keeps saying the same thing.

Why it matters

The user pays for the difference twice. Once in wall-clock: a watch that could
have ended in an hour holds its slot for four. Once in the record: when it
finally stops, the reason says runtime_budget, so the person reading it cannot
tell "this watch ran out of time with work still to do" from "this watch had
been finished for three hours".

What changed (motivation -> approach -> change)

MonitorState now carries the streak and decide_monitor acts on it, through
three fields: stall_digest (a hash of the last counted verdict), stall_streak
(how many ticks in a row reached exactly that verdict), and stall_started_at
(when that streak opened).

Tick the engine sees Before After
Settled, engine did nothing, under the count or the floor keeps watching [=] keeps watching [=]
Settled, engine did nothing, past both keeps watching until a budget [-] stops, reason verdict_stall [+]
Anything else, including still in flight keeps watching [=] keeps watching, streak zeroed [+]
Merged, closed, or a budget really spent stops with its own reason [=] stops with its own reason [=]

[+] added | [~] changed | [-] removed | [=] unchanged (ASCII: this host's publish gate rejects non-ASCII)

A watch now ends on the hour it stopped learning, and the record says which of
the three things happened.

The notification says it too. In src/kiro_crew/slack/gateway.py a stall fell
into the terminal-blocker map, titled "stopped on a terminal blocker" and telling
the reader to "resolve the reported problem". It now branches beside
pull_request_closed: same conclusion N checks running over at least M minutes,
naming the verdict it kept reaching from last_observation_reason_code. It deliberately does not say the subject is fine:
the likeliest stall is an already-alerted red inside its re-alert interval, where
that would be false exactly when it matters. N and M come from the constants, so
the sentence cannot outlive the thresholds.

Four choices carry it. The digest is derived from the verdict every tick,
never stored beside a second copy, so layer 3's one-source-of-truth rule holds one
layer up; the decision is inside it, which makes "no progress" the same question
as identity. A tick counts only when it settled the subject and the engine then
did nothing about it
, and every other tick zeroes the streak -- a wake acted, a
RECORD_ONLY held a change, a RETRY_PROVIDER waited, a stop already ended it,
an unsettled tick concluded nothing. The trip needs a measured wall-clock
floor
as well as the count, because cadence is user-set from 15s to 86400s:
twelve ticks is eleven intervals, 3300s at the 300s default but 165s at the
minimum. Comparing stall_started_at against the tick's clock translates nothing,
so no derived value goes stale when the cadence changes. Zeroing on an unsettled
tick is what makes reading that clock safe: a streak left standing through a long
pending stretch would let time alone satisfy the floor, and the next
non-retryable provider error would be filed as a stall. Both thresholds are
bounded by constants already in the module
-- 6 < 12 < 48 ticks and
240 << 1800 << 21600 seconds -- so a reader can check them. And the stop records
verdict_stall, distinct from approval_stall and every *_budget reason,
because a stop reading like a convergence or a cost is the cycle-cap anti-pattern
renamed.

monitor_stall_reason in monitoring/decision.py names that reason as
monitor_budget_reason already does, and _stall_tripped is the single place the
trip is evaluated. Both writers of stopped_reason -- monitoring/shadow.py and
autonudge.py -- ask it first, or they file a stall as checks_failed.
autonudge.py also clears the three fields on a retarget.

The layer-4 paragraph, the rules row and the layer-5 required-contents row in
docs/system-specs/modules/monitor-architecture.md now describe what exists
instead of what is missing. That rules row was the last one carrying the target
marker, so the sentence defining the marker goes with it.

Tests

test/test_monitor_decision.py adds twenty-one cases on the pure engine, built at the
real constants rather than patched-down ones, so every trigger is one production
reaches:

  • The count and the floor stop the watch together; one tick short of either does
    not. At the minimum cadence the count alone never retires it -- 121 ticks, not
    12 -- and a cadence change mid-streak needs no invalidation, because nothing is
    derived from the cadence.
  • A coalescing hold never retires it, and the wake it was folding still arrives
    once the floor passes. The case asserts
    MIN_MONITOR_CADENCE_SECS * DEFAULT_MONITOR_STALL_TICKS < DEFAULT_MONITOR_COALESCE_SECS
    so the arithmetic that made a tick-only ceiling wrong stays on the record.
  • Both thresholds are bounded by constants already in the module, asserted as
    relations and pinned exactly -- outer bounds included, because a relation alone
    lets a constant move while the spec keeps quoting the old number. That case also
    READS the spec paragraph and rebuilds both inequality chains from the constants,
    so a change fails here naming the file that has to move with it.
  • A pending tick, a provider-error tick, a wake and a stop each zero the streak.
    A provider stop after a long quiet stretch is not filed as a stall, which is the
    case that makes zeroing on an unsettled tick load-bearing; a merge after one is
    still a success; a clock that went backwards retires nothing.
  • A differing counted verdict restarts at one; a head change alone restarts it,
    which pins the decision being inside the digest.
  • The three stops stay distinguishable: convergence, spent budget, stall. A merge,
    a close, and a spent budget at the streak's edge each keep their own reason.
  • A record written before the fields starts fresh; a present streak round-trips;
    a malformed one is refused rather than coerced.

test/test_monitor_controller.py drives the composed production path through the
real wake, completion and reason writers, and asserts the retarget reset clears
all three fields after a counted tick has filled them. A second test pins the
condition invariant clause by clause: the stop's outcome and the streak land
together, a merge arriving afterwards is refused before the engine and carries no
entries, and the close rollback declines a stall-stopped watch.
test/test_github_pull_request_monitor.py does the same for the persistence-only
writer, so both branches of the reason precedence are covered.

test/test_slack_gateway_more_coverage.py pins the new notification, and replaces
the hand-written reason enumeration beside it with one DERIVED from the
MONITOR_STOP_* constants: every reason that can reach the blocked branch must
have its own copy, with two exclusions carrying stated structural reasons rather
than convenience. That is what would have caught this before review -- a map keyed
by a constant goes stale exactly as a comment does, but fails silently.

One case drives a real stall through the engine and then reads the notice, so the
thresholds the sentence renders are provably the ones that retired the watch rather
than two numbers that happen to match today. Writing it found that the engine does
not record the observation's code -- the probe writers do -- which is worth knowing
before trusting the field anywhere else.

Ten mutations were run and each reddened the case that names it: unfencing the
close rollback, removing the terminal refusal, counting deferrals again, dropping
the wall-clock floor, dropping the cadence guard the earlier tick-ceiling shape
needed, letting an unsettled tick leave the streak standing, making the new
notification branch unreachable, restoring the "nothing is wrong" claim the
suppressed-red case forbids, truncating the verdict the notice reports, and editing the spec to quote a bound
the constants do not support.

Manual verification

Two stop-reason namespaces here look like one, and it is worth stating for the
next reader. APPROVAL_STALL_REASON is approval_stalled on loop.stopped_reason,
the timer loop's field; MONITOR_STOP_APPROVAL_STALL is approval_stall on
loop.monitor.stopped_reason. One letter apart, two subsystems, two records. A
review finding on this PR reasoned across them and asked for a dashboard label
keyed on the wrong field. The notification half of that finding was right and is
fixed here; the label half has no path that can deliver the value, so the answer
is evidence in a comment rather than code. Separately, monitor_terminal IS
missing from that label map and does render raw -- reachable, pre-existing, and not
from this diff.

N/A -- the change is pure decision policy plus two persistence writers, and the
composed tests drive both through their real callers rather than mocks.

Related Issues

no linked issue: closing a target the module spec named for itself.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested review from a team and cixuuz September 17, 2026 04:24
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Intent: close the stall-streak gap the monitor-architecture spec named for itself, so a structured monitor that keeps reaching the same conclusion retires itself on its own reason instead of running out a budget under a reason that reads like something else.

Not a goal: any new MonitorDecision or MonitorOutcome member, any change to the dashboard contract or the persistence funnel, any change to irq.py or the layer-3 spec section, and any reshaping of the state document into a per-key model.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of de436cd12ce725b2d125629bf9c4e1be831ddb47 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

A stall now arrives everywhere as STOP_BLOCKED/BLOCKED; verify the Slack notice is the only consumer that branches on that outcome without reading stopped_reason.

What this change ships

Inventory (9 items) — 9 justified

Intent: stop a monitor watch on the hour it stops learning, and record that as its own reason — an ADDITION, licensed by the base spec's own target rule (monitor-architecture.md:499: "The stall streak is engine state, so a stuck watch stops itself").

  1. A watch repeating the same settled verdict retires after 12 ticks and ≥30 min instead of its 4-hour budget — justified
  2. The stop record says verdict_stall, distinct from budget and convergence stops — justified
  3. The stalled-watch Slack notice names the repeated verdict instead of "resolve the reported problem" — justified
  4. Three new persisted fields (stall_digest, stall_streak, stall_started_at), absent-means-fresh — justified
  5. Two new constants, both bounded by existing module constants (6<12<48, 240≪1800≪21600) — justified
  6. monitor_stall_reason, asked by both stopped_reason writers (counted: autonudge.py, shadow.py = 2 consumers) — justified
  7. A retarget also clears the stall fields — justified
  8. Spec paragraph, rules row and state-doc row updated in the same commit; orphaned target marker definition removed — justified
  9. Hand-written stop-reason enumeration in the Slack test replaced by one derived from MONITOR_STOP_* — justified

Existing streak counters (quiet_streak, consecutive_provider_errors; 9 occurrences, 3 files) count other things per the base spec's own text — not a duplicate.

[FIRST-PRINCIPLES-REVIEWED] de436cd

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of de436cd12ce725b2d125629bf9c4e1be831ddb47 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] de436cd

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed de436cd12ce725b2d125629bf9c4e1be831ddb47 via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/slack/gateway.py:7463 -- At 15s cadence, 121 checks trigger the stall, but "12 checks in a row, over at least 30 minutes" misstates the measured streak -> Fix: say “at least 12 checks in a row over at least 30 minutes.” (origin: validation)
[GPT-REVIEWED] de436cd

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed de436cd12ce725b2d125629bf9c4e1be831ddb47 via the fork AI-review pipeline; updated in place on each push.

Review details

I've independently verified the core: the stall-fold logic (the _verdict_is_settled gate correctly excludes PENDING-that-decides-NO_CHANGE via _decide_effect line 100), the zero-on-any-non-counted-tick invariant that makes monitor_stall_reason safe to read the streak alone in both writers' stop branches, that both apply_monitor_probe (autonudge) and run_shadow_probe (shadow) route STOP_BLOCKED→BLOCKED with provider_error=None, the retarget reset zeroing the three fields, and that the new fields auto-(de)serialize via fields()/asdict with absent-means-default. The spec is updated in-commit. No candidate to falsify and nothing new clears the bar.

No findings.

[OPUS-REVIEWED] de436cd

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/monitor-verdict-stall-streak branch from d4fc150 to cc04365 Compare September 17, 2026 04:52
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23

chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: stall streak fields on MonitorState plus a fold in decide_monitor

  • Coalescing can be mistaken for a verdict stall (span=571ae6d39894)

fixed in cc04365. Reproduced exactly as described:
_coalesce_actionable returns RECORD_ONLY on an ACTIONABLE observation while holding a
follow-up fingerprint and does not update coalesce_fingerprint, so the digest is
identical every tick and 12 of them land 180s into the 240s floor.
The fix is wider than the one suggested, because RECORD_ONLY is not the only deferral:
RETRY_PROVIDER on an ACTIONABLE observation carrying a supplemental provider error
repeats identically too, reachable at max_provider_errors >= 13. The streak now counts
ONLY a settled NO_CHANGE tick -- concluded something, did nothing -- and every other
settled decision zeroes it, which also removes the mirror case where a merge could
carry a stall reason. Pinned by test_a_coalescing_hold_never_retires_the_watch;
mutation-verified, restoring the old branch reddens it at "assert 1 == 0".

@chenmingwei23

chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: stall ceiling snapshotted per streak, carrying a wall-clock floor

  • Stall window is 12 ticks x cadence, so a short cadence retires a live watch in minutes (span=ef858d2fbb85)

fixed in cc04365, via the first branch of Clears-when:
the trip now requires a minimum wall-clock as well as the count. DEFAULT_MONITOR_STALL_MIN_SECS
is 1800, and stall_ceiling is the larger of the tick count and enough ticks at this watch own
cadence to cover it -- 12 at the 300s default, 120 at the 15s minimum, so the fast-end window
is 30 minutes rather than 3. It is snapshotted when the streak starts rather than recomputed:
a recomputed ceiling reads a mutable cadence_secs and could become satisfiable between two
folds, which would let a stop that happened for some other reason be filed as verdict_stall.
The reachability test now pins BOTH ends, which is the gap you named -- the previous one
defended only the slow end. Mutation-verified: dropping the floor term reddens it at
"assert 12 == 120".

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Cross-reference, not a review comment: #11460 is open and also edits docs/system-specs/modules/monitor-architecture.md.

The two do not overlap. This PR's hunks in that file are at lines 327, 491 and 526 plus a layer-5 table cell; #11460 owns 235-268 (layer 3's Observation section) plus name-only fixes at 126 and 307. Whichever merges second takes a trivial rebase.

Neither depends on the other. #11460 renames irq.Observation's epoch_scoped field to resets_on and irq.Severity.NMI to IMMEDIATE, and touches nothing under src/kiro_crew/monitoring/, which is where this PR works.

Flagging it only so a reviewer who opens one can see the other exists.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/monitor-verdict-stall-streak branch from cc04365 to cee19d0 Compare September 17, 2026 05:06
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: stall_ceiling snapshotted at streak start, now re-snapshotted when the cadence it was derived from changes

  • Cadence changes bypass the wall-clock stall floor (span=571ae6d39894)

fixed in cee19d0. Confirmed reachable: update_monitor sets reset_baseline only for a
target or objective change, so its cadence branch rewrites cadence_secs and leaves the stall
fields intact -- a streak opened at the 300s default keeps a ceiling of 12 while the watch
speeds to 15s, and trips about 465s in against an intended 1800s floor.
Fixed at the CHOKEPOINT rather than the suggested call site: _fold_stall_streak now
re-snapshots when the stored ceiling no longer matches the cadence, so any route to a new
cadence is covered -- update_monitor, a config reload, or a record loaded with a different
one -- instead of only the writer I know about. This is the second round on this span, and a
per-call-site reset would have invited a third.
The recomputation only DETECTS the change; the trip still reads the stored value, so it stays
a pure integer comparison the fold controls and cannot become true between two folds -- the
invariant monitor_stall_reason relies on to ignore the decision.
Pinned by test_a_cadence_change_mid_streak_restarts_against_the_new_unit; mutation-verified,
dropping the guard reddens it at "assert 2 == 1".

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/monitor-verdict-stall-streak branch from cee19d0 to 87457ce Compare September 17, 2026 05:26
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: stall_started_at replaces the derived tick ceiling; the fold zeroes on every non-counted tick

  • Cadence changes bypass the wall-clock stall floor (span=571ae6d39894)

fixed in 87457ce, and NOT by the prescribed remedy. Resetting the stall fields on a
cadence change works but keeps the translation and adds an invalidation rule to a derived
cache; any translation from ticks to seconds breaks on a cadence change in one direction or
the other. So the translation is gone: stall_ceiling and _stall_ceiling_for are deleted, and
the trip is stall_streak >= DEFAULT_MONITOR_STALL_TICKS and now - stall_started_at >= DEFAULT_MONITOR_STALL_MIN_SECS -- elapsed time measured, not derived. Nothing reads
cadence_secs, so there is nothing to invalidate.
This does not reopen the hazard the snapshot was avoiding. That was a predicate an operator
could make true between two folds by rewriting the cadence; elapsed time is monotonic. It
does require the fold to zero on EVERY non-counted tick, an unsettled one included: a streak
left standing through a long pending stretch would let the clock satisfy the floor, and the
next non-retryable provider error would be filed as a stall. Cost: a flapping subject never
accumulates a streak and retires on its runtime budget -- later, but honestly labelled.
Both thresholds are now anchored to existing constants rather than asserted (6 < 12 < 48
ticks; 240 << 1800 << 21600 seconds), which also answers First Principles.
Two new mutations verify it: letting an unsettled tick stand reddens the provider-stop case
at "assert verdict_stall == \x27\x27".

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the notice now names last_observation_reason_code instead of characterising the subject

  • The stall copy claims nothing is wrong, but the dominant stall is an already-alerted red (span=ef0dd52b4abe)

correct, and the evidence is in my own test file. _suppressed_red_state seeds
coalesce_alerted={"red-1": 1000.0} -- an actionable subject already alerted and inside its
re-alert interval -- and it is the helper nearly every stall case is built from. So the
likeliest stall is exactly the case where "Nothing is wrong with the pull request" is false.
Fixed in 29a8d39 by naming the verdict rather than characterising it. The notice now
reads "Last seen: " from last_observation_reason_code, which already exists on
MonitorState, and says the watch stopped because nothing was MOVING rather than because
nothing is wrong -- true for a suppressed red and for a quiet green alike, and it hands the
reader the fact instead of a judgement.
This finding also names a real cost of my own first fix: the generic copy I replaced said
"resolve the reported problem", which for a suppressed red was closer to right than what I
put in its place. I fixed the case where nothing is wrong and broke the case where something
is. Naming the verdict is the only version that is true in both.
Pinned by test_a_stalled_suppressed_red_is_not_described_as_fine, which asserts the reason
code appears and "Nothing is wrong" does not. Restoring the claim reddens it at
assert "checks_failed" in body -- the eighth mutation on this branch.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: premises conceded; the practice question is answered with merged precedent, and the authority question is left to a maintainer

  • Product shape, decision record unclear: the stall shape has no non-draft RFC (span=0f8edb626f14)

both checkable premises are TRUE and I verified them rather than argue them.
docs/request-for-change/rfc-consolidated-monitor.md on main is status: draft, and it
mentions stall ZERO times. And draft is not a vestige here: across the RFC directory the
status field carries partial (15), draft (14), in-progress (5), accepted (3), superseded (2)
and implemented (1), so it is a field that means something. The only record for this shape is
monitor-architecture.md.
What I can offer is the practice, not a counter-theory. A merged target row in that spec has
been this project decision record repeatedly and recently: #10552 implemented the coalescing
target, #10588 the kind-dispatch one, #10803 restored the rules, #10946 recorded what gates
retirement, and #10270 corrected an invariant in the same file. All merged, none carrying a
non-draft RFC for its shape. So the practice being asked about is already the practice, five
times over, in this exact spec.
That is precedent, and precedent is not authority. This lane offers two clearing paths and the
second names a MAINTAINER deliberately, because the question is whether a merged target row
counts as the recorded decision -- the lane own words are that this is the maintainers read,
not the diff. I am not posting that override and I am not treating five merged PRs as
permission to close a governance question about myself. The precedent is supplied; the read
that closes it is a maintainer.

@chenmingwei23
chenmingwei23 force-pushed the feat/monitor-verdict-stall-streak branch from 29a8d39 to cddf20f Compare September 17, 2026 07:19
@chenmingwei23

chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the monotonicity claim is replaced by the real two-direction argument, in the docstring and the spec

  • "elapsed time is monotonic" contradicts the time.time() wall clock (span=c1210cebd9f6)

correct, and the claim was doing real work in the argument, which makes it worth more than a
word fix. Production reads time.time() at both probe call sites, so the clock is NOT
monotonic and can move either way. Fixed in cddf20f by stating what is actually true,
which is a stronger argument than the false one: BACKWARDS, now - stall_started_at goes
negative and fails the floor, so a jump can only DELAY a trip -- already pinned by
test_a_clock_that_went_backwards_does_not_retire_the_watch. FORWARDS, a jump can satisfy the
floor early but cannot manufacture the twelve counted ticks, because those are counted, not
timed. So neither direction reopens the hazard a ceiling guarded against, and the reason is
precisely that BOTH halves of the condition are required -- which is the point the paragraph
exists to make. Corrected in models.py and in the parallel spec sentence.

(The other finding on this head is dispositioned in its own comment. Batching the two into one
record was my error: one rationale covers exactly one finding.)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: a new test drives a real stall through the engine, then asserts the notice renders the constants that tripped it

  • The rendered thresholds have no production-path assertion tying them to the trigger (span=f00224b0077d)

fair, and acting on it found something the copy fix alone would have hidden. Added
test_the_rendered_thresholds_are_the_ones_that_tripped_the_stall in cddf20f: it drives
decide_monitor at the real cadence until it stops, takes the reason that path recorded, and
asserts both rendered constants plus the reported verdict appear in the notice.
Writing it FAILED first, and the failure was the useful part: the ENGINE does not record the
observation code -- the probe writers do, at staged_state.last_observation_reason_code in
autonudge.py and the same line in monitoring/shadow.py. A test that called only the engine
rendered the empty fallback, so the notice read "Last seen: no change" for a suppressed red --
the exact class of defect the sibling finding on this head was about, surviving one layer
down. Production is correct because the writer runs before the decision branch, but the field
is only trustworthy at a site the writer has already passed, and the test now says so where a
reader of that field will meet it.
Truncating the rendered verdict reddens it: the ninth mutation on this branch.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the feat/monitor-verdict-stall-streak branch from cddf20f to 56a4f86 Compare September 17, 2026 07:30
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the anchoring test now pins the outer bounds exactly and reads the spec paragraph, rebuilding both chains from the constants

  • The spec inequality chains can drift because no test reads that paragraph (span=795d3ef8e10a)

right, and a narrower gap than the wording suggests, which is worth saying because the narrow
version is the one that bites. The anchoring case already pinned 12, 1800, 6 and 48 exactly.
It did NOT pin the two OUTER numbers: it asserted relations against them
(> COALESCE * 5, * 5 < REALERT), so DEFAULT_MONITOR_COALESCE_SECS could move from 240
and every assertion would still pass while the spec kept quoting 240. That is a real silent
drift and it is now closed -- both outer constants are pinned exactly.
On reading the paragraph: done, in cddf20f34763s successor 56a4f86. The case builds both
chains from the constants and asserts the spec contains them, so the failure message names
the file that has to change rather than leaving a reader to find it. Editing the spec to
quote 240 << 1800 << 9999 reddens it with "monitor-architecture.md no longer states
240 << 1800 << 21600" -- the tenth mutation on this branch.
This is the same defect class as the two prose findings you raised earlier on this PR, one
level up again: a number restated in prose has no assertion behind it. A test that reads the
prose is the only thing that makes a quoted constant load-bearing.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: none -- the arithmetic is confirmed and the intent question is left to a maintainer, with a third option costed

  • The stall calibration retires an actionable subject at ~55 minutes, well before its budget (span=afe4de61cb23)

the arithmetic is right and I am not closing this. Twelve ticks at the 300s default is eleven
intervals, 3300s -- 55 minutes -- against a 14400s runtime budget and a 21600s re-alert
interval. So a stall retires a watch at 23% of its budget, and for the case you identified
earlier on this PR as the DOMINANT stall (an actionable subject alerted once, then byte-
identical) that is a real behaviour change: at 55 minutes nothing is watching, so a push at
two hours and a green at two-and-a-half produce no wake, where the budget alone would have
kept the watch to four hours.
One correction to the counterfactual, from my own test: a head change restarts the streak but
does NOT by itself wake anyone -- dedupe still suppresses it inside the re-alert interval
(test_a_head_change_alone_restarts_the_streak documents exactly that). So what retiring
early forfeits is not the push wake; it is the watch still being alive when the subject later
reaches a state worth waking for. That is a narrower loss than "head_changed would have woken
it", and it is still a loss.
On the two clearing paths, a cost worth having before the call. Raising the count or floor
toward the budget window makes the mechanism nearly redundant with the budget: its whole value
is retiring EARLIER than the budget with a truthful reason, so a stall that fires at 3.9 hours
against a 4-hour budget buys almost nothing.
A third option neither of us has named: do not count a tick while the subject is ACTIONABLE. A
watch on a red pull request still has a job -- tell me when it goes green -- whereas a watch
that keeps concluding "ready, and I already said so" has none. That keeps the mechanism where
it adds value and removes it exactly where it costs a useful wake. Its cost is that the
dominant case disappears, so the stall fires far less often, and whether that is correct
narrowing or hollowing the feature out is the judgement.
I have not changed a constant or the counting rule, because choosing among these three IS the
question, and this lane routes it to a maintainer rather than to the diff. The arithmetic
above is verified against the constants; the intent is not mine to record.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

On design span=afe4de61cb23: the stall and the re-alert compete, and this is a choice rather than a number

Not a second disposition -- that span has one above. This is the analysis it asked a maintainer
to rule on, with the arithmetic verified against the constants and one candidate ELIMINATED on
evidence rather than taste.

Conceded first: my own docstring establishes the competition. DEFAULT_MONITOR_STALL_MIN_SECS
argues the floor must stay well under DEFAULT_MONITOR_REALERT_SECS "because a re-alert wakes the
subject and zeroes the streak: a floor at or past that interval could never be reached." The
contrapositive is this finding: a floor BELOW the re-alert makes the re-alert unreachable for an
already-alerted unchanged subject, because the stall retires the watch first. Both directions
cannot hold, so no floor value satisfies both mechanisms.

The four numbers:

constant value
stall trip at default cadence (12 ticks = 11 intervals) 3300s
stall floor 1800s
runtime budget 14400s
re-alert interval 21600s

One correction that changes the scope, and it works against the finding as stated. The re-alert
is ALREADY unreachable inside a default watch, independently of this diff: the runtime budget is
14400s and the re-alert interval is 21600s, so a default-budget watch dies 7200s before its
first re-alert could fire. Shipping at 1800s therefore disables nothing for a default watch -- the
re-alert was unreachable there already. The competition is real only for a watch whose
max_runtime_secs is set above 21600s, and there it bites exactly as described.

What the stall does cost a default watch is the rest of its life: it retires at 3300s of a 14400s
budget, 23%, so a fingerprint change at two hours -- checks going green -- finds nothing watching.
That is the real forfeit, and it is not a re-alert wake.

Candidate 1, floor at or above the re-alert. Honest and self-consistent. Note what it implies:
at 21600s the floor exceeds the default runtime budget, so the stall becomes unreachable for a
default watch and earns its place only on the long-budget watches where the competition actually
exists. That is a coherent position -- the mechanism applies precisely where it is not in conflict.

Candidate 2, excluding ACTIONABLE from the count: NOT VIABLE, and this is checkable.
_SETTLED_STATUSES is {ACTIONABLE, SUCCESS, BLOCKED}. SUCCESS returns STOP_SUCCESS. BLOCKED
falls through to return MonitorDecision.STOP_BLOCKED on the tick its fingerprint moves, which is
the tick a pull request becomes closed. PENDING is RECORD_ONLY and PROVIDER_ERROR is excluded
already. So every settled status other than ACTIONABLE terminates the watch on the tick it settles
and cannot repeat. Remove ACTIONABLE from the count and NOTHING can reach twelve identical counted
ticks: the mechanism becomes unreachable, which is the same failure my floor bound argues against.

Candidate 3, ship as-is with the forfeited late wake recorded as an accepted cost, on the
argument that a user told "last seen: checks_failed, start a new watch if you expect it to change"
can re-arm. That moves work to the human and is a real cost, not a free one.

I am not picking. The trip point is a product judgment, this lane says so, and it is the second
question on this PR that belongs to a human. No further heads until one rules -- Design has gone
CONCERNS, PASS, PASS, CONCERNS across four of them, so another push buys a re-roll rather than an
answer.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 17, 2026
A structured monitor with nothing left to learn kept probing until a
budget retired it. MonitorState now carries the stall streak the module
spec has been describing as a target -- stall_digest and stall_streak --
and decide_monitor stops the watch once DEFAULT_MONITOR_STALL_TICKS
consecutive settled ticks reach a byte-identical verdict.

The digest is derived from the verdict on every tick, never persisted
beside a second copy of it, so the one-source-of-truth rule holds one
layer up. Settled is the domain's own classification: PENDING never
counts, so a watch is not retired for being early, and PROVIDER_ERROR
never counts either. The fold sits after classification because its
input IS the classification, and it never rewrites a decision that
already stops -- a merge stays a success and a close stays a close.

The stop records verdict_stall through monitor_stall_reason, the sibling
of monitor_budget_reason, so a stall is distinguishable from a
convergence and from a spent bound in the record both writers of
stopped_reason produce.
@chenmingwei23
chenmingwei23 force-pushed the feat/monitor-verdict-stall-streak branch from 56a4f86 to de436cd Compare September 17, 2026 09:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after a full read of the diff on the current head: the change matches the body and linked issue, the destructive/decision path re-verifies independently and fails closed, tests pin the behaviour, and all review lanes are clean on this head.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 17, 2026 19:11
@iamwhatever
iamwhatever merged commit 419d9d7 into kirodotdev:main Sep 17, 2026
86 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 17, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…tdev#11455)

A structured monitor with nothing left to learn kept probing until a
budget retired it. MonitorState now carries the stall streak the module
spec has been describing as a target -- stall_digest and stall_streak --
and decide_monitor stops the watch once DEFAULT_MONITOR_STALL_TICKS
consecutive settled ticks reach a byte-identical verdict.

The digest is derived from the verdict on every tick, never persisted
beside a second copy of it, so the one-source-of-truth rule holds one
layer up. Settled is the domain's own classification: PENDING never
counts, so a watch is not retired for being early, and PROVIDER_ERROR
never counts either. The fold sits after classification because its
input IS the classification, and it never rewrites a decision that
already stops -- a merge stays a success and a close stays a close.

The stop records verdict_stall through monitor_stall_reason, the sibling
of monitor_budget_reason, so a stall is distinguishable from a
convergence and from a spent bound in the record both writers of
stopped_reason produce.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants