Repository navigation
feat(monitoring): retire a watch whose verdict stopped moving - #11455
Conversation
|
Intent: close the stall-streak gap the monitor-architecture spec named for itself, so a structured monitor that keeps reaching the same conclusion retires itself on its own reason instead of running out a budget under a reason that reads like something else. Not a goal: any new MonitorDecision or MonitorOutcome member, any change to the dashboard contract or the persistence funnel, any change to irq.py or the layer-3 spec section, and any reshaping of the state document into a per-key model. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS A stall now arrives everywhere as What this change shipsInventory (9 items) — 9 justifiedIntent: stop a monitor watch on the hour it stops learning, and record that as its own reason — an ADDITION, licensed by the base spec's own
Existing streak counters ( [FIRST-PRINCIPLES-REVIEWED] de436cd |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] de436cd |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/slack/gateway.py:7463 -- At 15s cadence, 121 checks trigger the stall, but |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsI've independently verified the core: the stall-fold logic (the No findings. [OPUS-REVIEWED] de436cd |
d4fc150 to
cc04365
Compare
|
self-added: yes
|
|
self-added: yes
|
|
Cross-reference, not a review comment: #11460 is open and also edits The two do not overlap. This PR's hunks in that file are at lines 327, 491 and 526 plus a layer-5 table cell; #11460 owns 235-268 (layer 3's Observation section) plus name-only fixes at 126 and 307. Whichever merges second takes a trivial rebase. Neither depends on the other. #11460 renames Flagging it only so a reviewer who opens one can see the other exists. |
cc04365 to
cee19d0
Compare
|
self-added: yes
|
cee19d0 to
87457ce
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
29a8d39 to
cddf20f
Compare
|
self-added: yes
(The other finding on this head is dispositioned in its own comment. Batching the two into one |
|
self-added: yes
|
cddf20f to
56a4f86
Compare
|
self-added: yes
|
|
self-added: yes
|
On design span=afe4de61cb23: the stall and the re-alert compete, and this is a choice rather than a numberNot a second disposition -- that span has one above. This is the analysis it asked a maintainer Conceded first: my own docstring establishes the competition. The four numbers:
One correction that changes the scope, and it works against the finding as stated. The re-alert What the stall does cost a default watch is the rest of its life: it retires at 3300s of a 14400s Candidate 1, floor at or above the re-alert. Honest and self-consistent. Note what it implies: Candidate 2, excluding ACTIONABLE from the count: NOT VIABLE, and this is checkable. Candidate 3, ship as-is with the forfeited late wake recorded as an accepted cost, on the I am not picking. The trip point is a product judgment, this lane says so, and it is the second |
A structured monitor with nothing left to learn kept probing until a budget retired it. MonitorState now carries the stall streak the module spec has been describing as a target -- stall_digest and stall_streak -- and decide_monitor stops the watch once DEFAULT_MONITOR_STALL_TICKS consecutive settled ticks reach a byte-identical verdict. The digest is derived from the verdict on every tick, never persisted beside a second copy of it, so the one-source-of-truth rule holds one layer up. Settled is the domain's own classification: PENDING never counts, so a watch is not retired for being early, and PROVIDER_ERROR never counts either. The fold sits after classification because its input IS the classification, and it never rewrites a decision that already stops -- a merge stays a success and a close stays a close. The stop records verdict_stall through monitor_stall_reason, the sibling of monitor_budget_reason, so a stall is distinguishable from a convergence and from a spent bound in the record both writers of stopped_reason produce.
56a4f86 to
de436cd
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Approved after a full read of the diff on the current head: the change matches the body and linked issue, the destructive/decision path re-verifies independently and fails closed, tests pin the behaviour, and all review lanes are clean on this head.
…tdev#11455) A structured monitor with nothing left to learn kept probing until a budget retired it. MonitorState now carries the stall streak the module spec has been describing as a target -- stall_digest and stall_streak -- and decide_monitor stops the watch once DEFAULT_MONITOR_STALL_TICKS consecutive settled ticks reach a byte-identical verdict. The digest is derived from the verdict on every tick, never persisted beside a second copy of it, so the one-source-of-truth rule holds one layer up. Settled is the domain's own classification: PENDING never counts, so a watch is not retired for being early, and PROVIDER_ERROR never counts either. The fold sits after classification because its input IS the classification, and it never rewrites a decision that already stops -- a merge stays a success and a close stays a close. The stop records verdict_stall through monitor_stall_reason, the sibling of monitor_budget_reason, so a stall is distinguishable from a convergence and from a spent bound in the record both writers of stopped_reason produce.
Problem / Motivation
A structured monitor that has nothing left to learn keeps probing anyway. It
reaches the same conclusion tick after tick -- the same failing check, already
alerted, suppressed by dedupe -- and nothing notices. It runs until a budget
retires it, up to four hours at the defaults.
The module spec has named this gap in its own present tense for a while:
"No engine state holds a streak of identical verdicts today." Streak
counters exist and count other things.
quiet_streakcounts quiet observations,consecutive_provider_errorscounts provider failures. None of them notices awatch that keeps saying the same thing.
Why it matters
The user pays for the difference twice. Once in wall-clock: a watch that could
have ended in an hour holds its slot for four. Once in the record: when it
finally stops, the reason says
runtime_budget, so the person reading it cannottell "this watch ran out of time with work still to do" from "this watch had
been finished for three hours".
What changed (motivation -> approach -> change)
MonitorStatenow carries the streak anddecide_monitoracts on it, throughthree fields:
stall_digest(a hash of the last counted verdict),stall_streak(how many ticks in a row reached exactly that verdict), and
stall_started_at(when that streak opened).
verdict_stall[+][+] added | [~] changed | [-] removed | [=] unchanged (ASCII: this host's publish gate rejects non-ASCII)
A watch now ends on the hour it stopped learning, and the record says which of
the three things happened.
The notification says it too. In
src/kiro_crew/slack/gateway.pya stall fellinto the terminal-blocker map, titled "stopped on a terminal blocker" and telling
the reader to "resolve the reported problem". It now branches beside
pull_request_closed: same conclusion N checks running over at least M minutes,naming the verdict it kept reaching from
last_observation_reason_code. It deliberately does not say the subject is fine:the likeliest stall is an already-alerted red inside its re-alert interval, where
that would be false exactly when it matters. N and M come from the constants, so
the sentence cannot outlive the thresholds.
Four choices carry it. The digest is derived from the verdict every tick,
never stored beside a second copy, so layer 3's one-source-of-truth rule holds one
layer up; the decision is inside it, which makes "no progress" the same question
as identity. A tick counts only when it settled the subject and the engine then
did nothing about it, and every other tick zeroes the streak -- a wake acted, a
RECORD_ONLYheld a change, aRETRY_PROVIDERwaited, a stop already ended it,an unsettled tick concluded nothing. The trip needs a measured wall-clock
floor as well as the count, because cadence is user-set from 15s to 86400s:
twelve ticks is eleven intervals, 3300s at the 300s default but 165s at the
minimum. Comparing
stall_started_atagainst the tick's clock translates nothing,so no derived value goes stale when the cadence changes. Zeroing on an unsettled
tick is what makes reading that clock safe: a streak left standing through a long
pending stretch would let time alone satisfy the floor, and the next
non-retryable provider error would be filed as a stall. Both thresholds are
bounded by constants already in the module --
6 < 12 < 48ticks and240 << 1800 << 21600seconds -- so a reader can check them. And the stop recordsverdict_stall, distinct fromapproval_stalland every*_budgetreason,because a stop reading like a convergence or a cost is the cycle-cap anti-pattern
renamed.
monitor_stall_reasoninmonitoring/decision.pynames that reason asmonitor_budget_reasonalready does, and_stall_trippedis the single place thetrip is evaluated. Both writers of
stopped_reason--monitoring/shadow.pyandautonudge.py-- ask it first, or they file a stall aschecks_failed.autonudge.pyalso clears the three fields on a retarget.The layer-4 paragraph, the rules row and the layer-5 required-contents row in
docs/system-specs/modules/monitor-architecture.mdnow describe what existsinstead of what is missing. That rules row was the last one carrying the
targetmarker, so the sentence defining the marker goes with it.
Tests
test/test_monitor_decision.pyadds twenty-one cases on the pure engine, built at thereal constants rather than patched-down ones, so every trigger is one production
reaches:
not. At the minimum cadence the count alone never retires it -- 121 ticks, not
12 -- and a cadence change mid-streak needs no invalidation, because nothing is
derived from the cadence.
once the floor passes. The case asserts
MIN_MONITOR_CADENCE_SECS * DEFAULT_MONITOR_STALL_TICKS < DEFAULT_MONITOR_COALESCE_SECSso the arithmetic that made a tick-only ceiling wrong stays on the record.
relations and pinned exactly -- outer bounds included, because a relation alone
lets a constant move while the spec keeps quoting the old number. That case also
READS the spec paragraph and rebuilds both inequality chains from the constants,
so a change fails here naming the file that has to move with it.
A provider stop after a long quiet stretch is not filed as a stall, which is the
case that makes zeroing on an unsettled tick load-bearing; a merge after one is
still a success; a clock that went backwards retires nothing.
which pins the decision being inside the digest.
a close, and a spent budget at the streak's edge each keep their own reason.
a malformed one is refused rather than coerced.
test/test_monitor_controller.pydrives the composed production path through thereal wake, completion and reason writers, and asserts the retarget reset clears
all three fields after a counted tick has filled them. A second test pins the
condition invariant clause by clause: the stop's outcome and the streak land
together, a merge arriving afterwards is refused before the engine and carries no
entries, and the close rollback declines a stall-stopped watch.
test/test_github_pull_request_monitor.pydoes the same for the persistence-onlywriter, so both branches of the reason precedence are covered.
test/test_slack_gateway_more_coverage.pypins the new notification, and replacesthe hand-written reason enumeration beside it with one DERIVED from the
MONITOR_STOP_*constants: every reason that can reach the blocked branch musthave its own copy, with two exclusions carrying stated structural reasons rather
than convenience. That is what would have caught this before review -- a map keyed
by a constant goes stale exactly as a comment does, but fails silently.
One case drives a real stall through the engine and then reads the notice, so the
thresholds the sentence renders are provably the ones that retired the watch rather
than two numbers that happen to match today. Writing it found that the engine does
not record the observation's code -- the probe writers do -- which is worth knowing
before trusting the field anywhere else.
Ten mutations were run and each reddened the case that names it: unfencing the
close rollback, removing the terminal refusal, counting deferrals again, dropping
the wall-clock floor, dropping the cadence guard the earlier tick-ceiling shape
needed, letting an unsettled tick leave the streak standing, making the new
notification branch unreachable, restoring the "nothing is wrong" claim the
suppressed-red case forbids, truncating the verdict the notice reports, and editing the spec to quote a bound
the constants do not support.
Manual verification
Two stop-reason namespaces here look like one, and it is worth stating for the
next reader.
APPROVAL_STALL_REASONisapproval_stalledonloop.stopped_reason,the timer loop's field;
MONITOR_STOP_APPROVAL_STALLisapproval_stallonloop.monitor.stopped_reason. One letter apart, two subsystems, two records. Areview finding on this PR reasoned across them and asked for a dashboard label
keyed on the wrong field. The notification half of that finding was right and is
fixed here; the label half has no path that can deliver the value, so the answer
is evidence in a comment rather than code. Separately,
monitor_terminalISmissing from that label map and does render raw -- reachable, pre-existing, and not
from this diff.
N/A -- the change is pure decision policy plus two persistence writers, and the
composed tests drive both through their real callers rather than mocks.
Related Issues
no linked issue: closing a
targetthe module spec named for itself.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)