Repository navigation
fix(monitoring): collapse superseded check attempts by workflow run - #11830
Conversation
|
Intent: Stop a structured pull-request watch from reporting a check attempt that a newer workflow run already replaced as a live failure, by asking the rollup for each row's |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the stated residual — completed replaced rounds still wake as phantoms (the PR's own wake 1) — gets a tracked follow-up alongside #11832. Suggestions
[DESIGN-REVIEWED] 608e2a0 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The sibling collapse in Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: stop a PR watch spending its capped agent turns waking on check rows a newer workflow run already replaced — a FIX.
[FIRST-PRINCIPLES-REVIEWED] 608e2a0 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/monitoring/github_pull_request.py:75 -- current Actions run IDs exceed GraphQL Int, so False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
d1f0854 to
d7e1dab
Compare
|
Disposition: fixed -- F1
self-added: yes |
d7e1dab to
e5fc1e7
Compare
|
Disposition: fixed -- F1
self-added: yes |
e5fc1e7 to
f7a9c73
Compare
|
Disposition: fixed -- F1
self-added: yes |
f7a9c73 to
5bcba53
Compare
|
Disposition: accepted-and-deferred -- target=first-principles, Item 1 (
self-added: no |
cixuuz
left a comment
There was a problem hiding this comment.
Careful, well-evidenced fix for a measured problem (96 duplicate check identities across 30 PRs in the PR's own measurement) causing phantom monitor wakes. The core rule -- displacement is proven only by the RUN's own cancellation, never inferred from a row's own CANCELLED conclusion, which also occurs inside live runs (fail-fast, failed needs, manual single-job cancel) -- is correct and consistently implemented. Field extraction in _flat_check_row populates exactly the keys the collapse functions read. Honestly documents the sibling tool's (pr_status.py) unfixed divergence via tracked issue #11832 rather than silently leaving it inconsistent.
6ec6b57 to
b5bbdb7
Compare
|
Disposition of every advisory finding on head GPT, First Principles, verify on live GitHub data that
The two genuinely displaced runs conclude Design, confirm the accepted residual is tolerable, and fold into #11890 when it lands: accepted as stated, no code change. A replaced round whose rows COMPLETED with a real verdict still reads as live and still wakes the session. That is deliberate and unchanged by this PR: dropping a row that reached a verdict is what would hide a live failure, and this change is the conservative direction of that trade. The residual is written into No |
b5bbdb7 to
2fe304e
Compare
2fe304e to
7558da6
Compare
|
Disposition of every advisory item raised on head fixed -- target=gpt, the only finding: The finding is correct and the spec was the wrong side. The winner loop consults only the row's key and its integer run id, never the run conclusion, so a row whose The claim is also pinned now, not just corrected: accepted, deferred -- target=design, the suggestion to run The reading is right. The slice at the call site truncates before confirmed -- target=design, that the accepted residual is tolerable interim. It is, and it is narrower than the general case: a completed FAILURE row of a replaced round still wakes a phantom (#11890), because only a cancelled row inside a cancelled run carries displacement proof. Budget waste is reduced, not removed, which is what the spec and the PR body both say. verified -- target=first-principles, both checks.
|
A structured watch counted a cancelled attempt a newer run had already replaced, so it woke the session on a failure that no longer existed. The rollup selection now carries each check run's startedAt and its run's databaseId, and rows a newer run replaced are dropped before the state fold. Two rows of one run stay: a workflow can publish a check run under its own job's name, and both are live.
What is the problem?
A structured
monitor_watchon a GitHub pull request reads a check row that a newerworkflow run has already replaced as a live failure, and wakes the session on it.
GitHub leaves a replaced round's completed rows in the rollup beside the round that
replaced them. In
src/kiro_crew/monitoring/github_pull_request.py,_normalize_checkshands everyCheckRunrow a synthetic per-row group key,("independent_check_run", str(row_index)), so two rows of one check survive as twoseparate
GitHubCheckvalues. Afailedone enters the failed bucket,actionable_fingerprint_factskeepsfailed_checks, and the watch reports a blockerwith no live run behind it. The asymmetry is the clearest evidence: a
StatusContextrow gets a real group key,
("status_context", context), so its duplicates fold, andonly
CheckRunrows opt out of that mechanism.docs/system-specs/modules/monitor-architecture.mdalready carried the rule under"Rules the engine enforces, not the prose", and said in its own text that the two
implementations disagreed on it.
CANCELLEDis one instance, not the mechanism. Any completed row of a replacedround reads as live, whatever its conclusion. Measured on this pull request's own
board: five workflows carried two rounds on one head, and the un-collapsed read
reported
failed: 2namingCode Review / PR HygieneandGPT 5.6 Review, bothFAILURE, neitherCANCELLED, both from a round a newer one had replaced. Across30 open pull requests and 2292 rollup rows, 96 duplicate identities spanned two runs.
Why this issue matters to the user
A phantom wake spends one of the watch's agent turns, and
max_agent_turnsis cappedat 8 by the schema, so the budget cannot be raised to absorb the waste. A watch armed
on a pull request whose CI re-ran can spend most of that budget on failures that are
not there, then fall silent before the real verdict arrives. The report is also
unactionable: the row it names has no live run to read or re-run.
Measured on this pull request's own watch, which ran against the unfixed code while
the fix was being reviewed:
CANCELLEDfrom a replaced roundThree of the six wakes named nothing that existed: replaying wakes 2 and 4 through the
collapse turns
failed: 6and7intofailed: 0, because every row in them wasCANCELLED. Wake 1's rows hadFAILUREconclusions from a replaced round, and under thecancellation rule those are KEPT, so that phantom is a stated residual rather than a fix. Wakes 3 and 5 were mixed, and
the collapse kept both live failures while dropping six replaced rows each. Wake 6 is
unmeasured rather than counted: its payload named a head the next push superseded, and
that commit's rollup has since changed, so the rows it saw cannot be reconstructed.
So a monitor woke at least three times on the defect it is being changed to fix, and the
change removes exactly those without suppressing a real signal. The abandoned commit is a
negative control for the same rule: every run on it was cancelled with no successor, and
the collapse correctly drops nothing there, reporting all 32 rows.
What changed: how the fix chains from symptom to root cause
The symptom is a failed bucket holding a replaced row. The root cause is that the
rollup selection never asked for the facts that identify one, so the normalizer had
nothing to collapse on and fell back to a per-row key.
_ROLLUP_SELECTIONnow asks eachCheckRun's run for itsdatabaseIdandevent,and its workflow's
databaseId._flat_check_rowcarries all three through, and_collapse_superseded_rowsruns before_normalize_checksgroupsanything, dropping rows a newer run replaced.
The fields do different jobs. Identity is the workflow DEFINITION's id, the RUN's
triggering event and the check name. Ordering is the monotonic RUN ID. The display name
is not an identity, since two workflow files may share one
name:. The event is in theidentity because one file may declare several triggers and each dispatches its own run.
Ordering may not use a timestamp: a job's
startedAtwaits on a runner queue, andcreatedAtresolves only to the second, which two runs of one workflow on one headroutinely share. The run id settles both and needs no extra field, since it already
groups the rows.
Recency does not license removal. The rollup has no lineage edge saying one run replaced
another, and
eventis coarser than the action that fired it, so asynchronizerun andan
editedrun sharepull_request. A row is therefore dropped only when its own run wasCANCELLED and a newer run of its identity exists: cancellation by the concurrency group
establishes displacement, and such a row carries no verdict to lose. A row that reached a
conclusion is kept however old it is. In the other direction, the NEWEST run being
cancelled is never a reason to drop it, since that revives the verdict it superseded.
Two rows of ONE run are not a retry and both survive: a workflow can publish a check
run through the Checks API under its own job's display name, so both are live at
once. On one open pull request a description check published FAILURE two seconds
after its own job reported SUCCESS, on a head whose
PR Readinesswas FAILURE.One rule covers absent evidence: a row is never collapsed on an id the response
withheld. Both
WorkflowRun.databaseIdandWorkflow.databaseIdare nullableInteven though the objects holding them are not, so either can be absent alone and
neither absence implies the other. Without the definition id or the event two rows
cannot be shown to be one check; without the run id an attempt can neither lose nor
win, since it may be the run that would replace the others. Over-report rather than
hide a live failure.
Three designs were rejected. Worst-wins across attempts folds a replaced attempt and its
successor to
failed. Rollup position cannot order anything; it is not chronological.Timestamp ordering ties, which
pr-readiness.ymlalready records. The added fields costnothing:
rateLimit.costis 1 before and after.What tests we did
test/test_github_pull_request_monitor.pygains the red-first case: a replacedattempt beside its successor must leave the failed bucket empty, in either rollup
order. It fails on
origin/mainwithfailed: ['CI / test']and passes with thechange. The display-name collision has its own red-first case: two workflow
definitions sharing the name
CI, the earlier failing, must both survive. It failswith
assert [] == ['CI / test']before the identity change.Nine further cases pin the semantics, and each passes on
origin/main, which makesthem guards rather than restatements: two rows of one run stay independent; a run
with no start time is kept; an unidentified run never loses; an unidentified run
never decides which identified run is newest; a row with no workflow definition id is
never collapsed; two runs tied on creation time are both kept; a run missing its
creation time is kept; two triggers of one workflow do not supersede each other; and
the selection asks for every field the collapse reads. That last one exists because the provider tests answer from
canned rows, so without it the selection could drop a field and every behavioural
test would stay green while the collapse silently stopped collapsing.
Two separate hazards sit behind the ordering choice, and only one of them came from a
review lane. The lane's re-run story is refuted by measurement: on re-run commit
e7b493d2b4b094b88aa59a217ca60a4202a4b481, all 16 runs atrunAttempt: 2supply exactlyone row per check name across 36 rows, with zero
(run id, check name)pairs repeated,so no older attempt's row survives beside a new one; and
CheckRunexposes no attemptfield at all, while
runAttemptis run-level and identical for both hypothetical rows,so the discriminator that story asks for does not exist. That is 36 rows on one commit,
which shows the premise is not reproducible here rather than impossible.
The second hazard is real and this repository already documented it.
pr-readiness.ymlrecords that
createdAthas one-second granularity, that two runs of one workflow on onehead routinely share a second, and that a same-second tie left
lastpicking the OLDESTrun so a lane whose newest run succeeded read as a blocking failure. A tie is observable
on this pull request's own head:
Screenshot Evidencehas two runs sharing2026-09-18T18:30:38Z, ids35380596033and35380596705. Ordering moved to the run idfor that reason, not for the refuted one.
Two properties became structural rather than conditional, so they carry no mutation: two
rows of one run share one id and are therefore always both kept, and two distinct runs
cannot tie. The tie test and the missing-timestamp test were deleted with the branches
they guarded.
On the trigger case specifically: four workflow files here declare both
pushandpull_request(build.yml,ci.yml,fast-gate.yml,internal-content-scan-gate.yml), and all four filterpushtobranches: [main], soa pull-request head never sees a push run and zero push runs appear on either measured
head.
internal-content-scan-gate.ymlalso declaresmerge_groupbesidepull_request. So the trigger in the identity closes a case the schema and the workflowsyntax permit, and that one file already pairs two events, rather than a collision
observed on this pull request.
test/test_babysit_guidance_gates.pyasserts the rule against the real normalizer, andthat the spec names the identity, the function, and BOTH halves of the divergence from
the sibling tool.
docs/system-specs/modules/monitor-architecture.mddeclares thatsibling's identity key AND its
startedAtordering, plus that it overwrites the row itdisplaces instead of retaining it, so a reader is not sent to reconcile only half.
src/kiro_crew/builtin_skills/kirocrew-dev/babysit/SKILL.mdtold agents the typedprovider did not collapse and to resolve the newest run by hand; it states the enforced
rule instead, and says plainly what the BUNDLED
pr_status.pydoes today, so an agentreading that tool's output knows it can drop a live failure where the provider keeps
one.
Fourteen mutations, each anchored on an exact source string with a marker assertion
so an unapplied mutation is a failed run rather than a silent pass: five deletions (the
collapse call and each of the four fields) and nine wrapping mutations, including
ordering by the job's start instead of the run's creation, dropping the event from the
identity, keying identity on the display name, falling back to the label when the
definition id is absent, newest-wins ignoring the run, dropping the unorderable keep,
letting an unidentified run lose, letting one win, and breaking a tie. All fourteen
redden; none survives. The surviving-mutant pass is what found the untested half of the
unidentified-run rule.
prove.pyreturns PROVEN. The scoped backend selection for this diff, 22 files,passes at 1351 tests, and both touched files pass at 174. Nineteen applicable
repository gates pass.
mypy src/kiro_crew/reports 4 errors, all insrc/kiro_crew/transcribe.pyand the Ops Mission Control CloudWatch provider, bothbyte-identical to
origin/mainand outside this diff.One red on this head is not this diff's.
Backend Tests (Windows) (2)failstest_ci_fleet_routing_expression_parity.py::test_every_ci_yml_resolver_consumer_reads_the_resolver_not_a_literal.main fails the same test on its own run
35383665208(sha0da53b3aaa), and this diffchanges no
.github/workflows/file.d0ff8107da(#11699) gavebackend-testaresolver-dependent
runs-onwithout adding it to_EXPECTED_RESOLVER_CONSUMER_JOBS.Tracked in #11883, fixed on main. That commit is already in this base, so a rebase does
not clear it and a re-run will not either.
Pattern harvest
Rule candidate: review-prompt
Pattern: a test fixture supplies a field the request under test never asked for, so the field selection is unverified and the test stays green after the selection drops it
Found by mutation, not by reading. Two selection mutations survived the first harness
run: removing a field from
_ROLLUP_SELECTIONchanged nothing, because_provideranswers from canned rows carrying those fields regardless of the query. A fixture
more complete than its request makes a whole class of adapter bug invisible while
reading as coverage. The check is: for any adapter tested against canned payloads,
one test must assert the request asks for every field the parsing path reads.
test_the_rollup_selection_asks_for_every_field_the_collapse_readsis that test here.Any other suggestions on the work
Two residuals are ACCEPTED here rather than fixed, and
#11890 tracks both.
Gating removal on cancellation does not cover the phantom this PR opened on. Head
d1f08548c4b7carriedFAILURErows from a replaced round, and a row that reached aconclusion is kept, so that wake still happens. This is the body's own point turned on
the fix: cancellation is one instance of displacement, not the mechanism, and the
rollup carries no lineage edge for the mechanism itself.
_MAX_CHECK_ROWStruncates BEFORE the collapse runs._checksslicesrows[member.raw][:_MAX_CHECK_ROWS]atsrc/kiro_crew/monitoring/github_pull_request.py:490,and
_normalize_checkscalls_collapse_superseded_rowsat:839. On a board widerthan the cap the truncation can drop a successor row and keep the row it replaced, which
is the same phantom reached another way.
#11890 also records the shape a later fix should take: declassify a superseded failure
in the wake decision instead of deleting its row. That needs
classify_pull_request_factsinsrc/kiro_crew/monitoring/pull_request.py, theclassifier four providers share, which is why it is not in this diff.
Why this pushed over green boards twice. Two heads were abandoned while passing, and
the second one had more than a green board.
5bcba53863carried Design PASS, FirstPrinciples PASS, Opus PASS, and a consumed human override on the GPT lane. Pushing voided
all four, and they have to be re-earned here.
Both pushes were still right, for the same reason. A passing lane is one sample of a
non-deterministic reviewer, not a proof -- the GPT lane returned both a fenced block and
a clean pass on the identical tree
d1f08548c4-- and in each case the code was wrong byconstruction. The first push fixed an identity keyed on a display name, inside a diff
whose own argument is that a label is not an identity. The second fixed an ordering this
repository has already documented as unsafe in the gate that decides readiness, inside the
function whose whole purpose is this collapse. A verdict obtained over either one would
have been a verdict about code we knew to be wrong. Neither push was made to refresh
verdicts; each replaced a mechanism, and the body moved with the code so the cost was one
re-roll rather than two.
Why removal is gated on cancellation rather than moved out of the provider. The
alternative considered was to keep every row and make a superseded failure non-actionable
in the wake decision instead. That is the better shape and it is deliberately not here:
classify_pull_request_factsinsrc/kiro_crew/monitoring/pull_request.pyis thecross-provider classifier that four providers share, and making a retained row
non-actionable means changing it or the canonical
checksschema. That is a secondsubsystem, and this diff's argument is the GitHub provider. Gating on cancellation keeps
the change provider-local while making the whole class of identity and ordering mistakes
unable to remove a verdict: only a row with no verdict can ever be dropped.
collapse_supersededin the prepare-pr status tool has the same label-identitydefect and is deliberately not in this diff; it is tracked in #11832. Its key
omits the run id as well, so on a same-run pair it keeps only the later row and
discards the other. Its consumer, tests and output contract are separate, so fixing
it here would double the review surface of a change whose argument is the structured
provider.
Two runs of one workflow from the SAME trigger on one commit are treated as an
attempt and its replacement, so a deliberate second dispatch of the same event
collapses to the later run. The provider exposes no replacement-lineage edge, so
definition plus trigger plus commit is the closest available approximation, and it
matches what a human reading the board does.
One hazard this does not fix, found while driving the watch: a probe can capture state
just before a push lands, so a wake's payload names a head that is already superseded.
Acting on the reported head instead of re-reading the live one means repairing something
already repaired. Re-reading the head on every wake is the only thing that catches it,
and it is not something the payload can signal.
The row cap is applied before the collapse:
_normalize_checksreceivesrows[:_MAX_CHECK_ROWS], so a truncated page can drop a successor and leave the rowit replaced. That is an over-report, the safe direction, but a very wide board can
still show a phantom.
This is independent of the entry-model migration in #11821 and shares no code with it.