Skip to content

fix(prepare-pr): collapse superseded check rows by workflow run, not label - #16510

Merged
bolichen97 merged 1 commit into
mainfrom
fix/collapse-superseded-live-runs-11832
Oct 4, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/collapse-superseded-live-runs-11832

Conversation

@NicholasRBowers

@NicholasRBowers NicholasRBowers commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: pr_status.py never hides a live check failure behind a re-run of the same check.

collapse_superseded keyed a CheckRun on ("run", workflowName, name) and kept the row with the latest startedAt. The displaced row was overwritten in winners, never added to undecidable, so the loss was unconditional for any dated pair. Three shapes put two LIVE rows under one key and dropped one of them:

  • two check runs of ONE workflow run -- a workflow publishing a check run under its own job's display name (measured on fix: cron_add inherits the caller Slack thread automatically #11783: Fork PR Description / Fork PR Description, two rows of run 35339317192, SUCCESS and FAILURE two seconds apart);
  • two workflow files sharing one name:, each publishing the same check name;
  • one workflow file dispatching on both push and pull_request for the same commit (four files in this repository declare both, ci.yml among them).

The fields the correct rule needs -- the run's id, its triggering event, its workflow definition's id, its own conclusion -- are not in the rows gh pr view --json statusCheckRollup returns. Measured on #11783 (97 rows), the complete key set is __typename completedAt conclusion context detailsUrl name startedAt state status targetUrl workflowName. There was no selection to extend.

Why it matters

pr_status.py is what the prepare-pr and babysit procedures read to decide whether a board is green. When the discarded row is the failing one, the tool reports a board greener than it is and an agent declares a pull request review-ready over a live failure. The arrival order that produces the harmful case is one nothing in the mechanism prevented.

Not a goal

  • Changing what pr_findings.py lists. It never collapsed and still does not: it fetches logs for every failing row it sees, which over-reports a displaced attempt but hides nothing. Pinned by test_the_findings_collector_lists_every_failing_row_it_fetched.
  • Declassifying rather than removing. The provider flags a displaced row so its state fold can show it; the status tool is a report of live rows and removes it, as before.
  • Any change to failing_check_identity (the progress_key label): it stays the human-readable workflow / name, and its docstring now says so.

What changed (motivation → approach → change)

Symptom: a live row of a re-run check vanishes from the report. Root cause: the collapse used a display label as identity and a job start time as order, and overwrote rather than kept the row it could not justify dropping. The correct rule already exists and is merged on the monitor side (_mark_superseded_rows, #11830 and #13394); docs/system-specs/modules/monitor-architecture.md and the babysit skill both named this tool as the divergent implementation and pointed at #11832.

Change, in both scripts (parity-pinned):

  • fetch_check_rollup(repo, number, expected_head) reads the head commit's statusCheckRollup through gh api graphql in its own call, selecting checkSuite{conclusion workflowRun{databaseId event workflow{databaseId name}}} for CheckRuns and context state targetUrl for StatusContexts. Owner, repository and cursor travel as -f raw strings (gh's typed -F turns an all-digit value such as someorg/2048 into a JSON integer the host rejects against String!); only the number is -F. The contract the comment above ROLLUP_UNAVAILABLE_NOTICE states is kept: a failed or malformed read degrades to UNAVAILABLE, the pull request's head is re-read beside the rollup -- now on every page, and the rollup's own commit oid is compared too -- and any disagreement discards the whole read as HEAD MOVED. The read pages (100 rows per page, _MAX_ROLLUP_PAGES = 10); a board past the cap reads UNAVAILABLE rather than in part, since a partial read could keep a displaced row whose successor was never fetched. flatten_rollup_row lifts the run fields out of the check suite beside the row's own fields, so every other reader of a row (classify_check, failing_check_identity, the Actions detection on workflowName, pr_findings' detailsUrl) is untouched.
  • collapse_superseded adopts the provider's rule: identity is (workflow.databaseId, run.event, name) (superseded_key); newest is the greatest run id; a row is removed only when it belongs to an older run than the newest of its identity AND that run concluded CANCELLED AND the row is COMPLETED+CANCELLED (_displaced). Rows of one run never displace each other; a row without a run identity is exempt and takes no part in choosing the newest; a replaced round's FAILURE rows, a cancelled row inside a live run, and a cancelled newest run all stay. StatusContexts are never collapsed: the host reports one status per context and the provider keeps every status row, so the base's label-and-startedAt collapse of them is removed rather than ported. Input order is preserved.
  • Call sites: both main()s detect the repository before the rollup read (the URL regex each already had) and pass repo, number, head_sha.
  • Docs: the spec's divergence paragraph now states the converged rule and the one remaining difference (UNKNOWN past the page cap, by design); the babysit skill's "does NOT yet follow this rule" sentence is replaced; prepare-pr/references/rationale.md explains why the separate call is GraphQL.

Guards enumerated. The displacement predicate must cover, and does: (1) same-run two rows -- same run id, never compared, both kept; (2) shared name: -- different definition ids, different keys; (3) push + pull_request -- different events, different keys; (4) app-created row (workflowRun null, measured on CodeQL rows of #16497-#16501) -- no key, exempt; (5) a withheld event -- no key, exempt; (6) no run id -- exempt; (7) replaced round with FAILURE rows -- run not CANCELLED, kept; (8) cancelled row in a FAILURE run -- run not CANCELLED, kept; (9) FAILURE row in a CANCELLED run -- row not CANCELLED, kept; (10) cancelled newest run -- it IS the newest, kept; (11) queue inversion of startedAt -- run id orders, not startedAt. Siblings. Three sites collapse rows of a statusCheckRollup: monitoring/github_pull_request.py:_mark_superseded_rows already follows the rule; prepare-pr/scripts/pr_status.py:collapse_superseded adopts it here; dashboard/source_providers/github.py:_github_checks is the remaining copy of the defect -- it keys on (kind, workflow label, name) and overwrites by startedAt, so the dashboard CI glyph and checks panel can still lose a live row in the same three shapes. It is left out of this PR's scope: it feeds a glyph, not the merge decision, and fixing it means moving _github_rollup_read to the same GraphQL read inside the provider's sandboxed runner, which is a larger change than this one. Tracked in #16519 (deferred-finding, needs-triage, assigned, due 2026-10-17). One site that LOOKS like a copy and is not: failing_check_identity keys on (workflowName, name) too, but it builds a progress_key label and drops nothing, so a label collision there costs at most a stall-detector miss, not a hidden failure.

Backwards compatibility

Compatible: both scripts accept the same arguments and print the same sections; the only observable change is that rows the old rule dropped by label and job start time now follow the run-keyed rule (and StatusContexts are no longer collapsed at all), which keeps strictly more live rows and removes only cancelled rows of cancelled older runs. The read needs the same Checks scope as before (a Checks-blind token degrades to the same UNAVAILABLE notice).

Removes nothing: fetch_check_rollup's signature changed from (pr, expected_head) to (repo, number, expected_head), and its only callers are the two main()s in this diff; collapse_superseded keeps its name and return shape.

Tests

Targeted, run by path with -n0 (276 passed across the three files):

  • test/test_prepare_pr_status.py -- red on base, green after: test_two_live_rows_of_one_run_both_survive_in_either_arrival_order (the issue's shape, FAILURE first and SUCCESS first), test_two_rows_of_one_run_are_both_reported_not_only_counted (identity-level, both orders), test_same_check_name_in_different_workflows_stays_distinct, test_two_triggers_of_one_workflow_are_concurrent_dispatches_not_attempts, test_rows_without_a_run_identity_are_exempt_not_grouped_together, test_rows_missing_only_the_event_are_exempt_not_merged, test_unordered_duplicates_are_all_kept_fail_closed, test_a_replaced_rounds_failure_rows_stay_live, test_a_cancelled_row_inside_a_live_run_is_not_dropped, test_a_cancelled_rows_verdict_must_also_be_cancelled_to_drop_it, test_a_cancelled_newest_run_is_never_dropped, test_newest_is_the_greatest_run_id_not_the_latest_started_row, test_a_displaced_round_is_dropped_whole_not_row_by_row (the fix(security): quote-aware argv windows for the mint and pkill self-protection floors #16496 shape), test_status_contexts_are_never_collapsed, test_status_contexts_that_cannot_be_ordered_are_all_kept. The read: test_the_rollup_query_selects_every_field_the_collapse_keys_on pins the selection bytes and the -f/-F split of the variables (the fake answers whatever a fixture carries, so a field the query never asked for would otherwise pass every test), test_the_rollup_read_pages_and_re_checks_the_head_on_every_page, test_a_head_that_moves_mid_pagination_discards_the_rollup, test_a_rollup_whose_commit_is_not_the_reported_head_is_discarded, test_a_board_past_the_page_cap_reads_unknown_not_partial, test_a_malformed_rollup_page_reads_unavailable. The three degraded-read tests now fail the GraphQL call and assert the core read never names the rollup.
  • test/test_prepare_pr_findings.py -- TestRollupHelperParity now pins ROLLUP_QUERY, _MAX_ROLLUP_PAGES, flatten_rollup_row and fetch_check_rollup byte-identical across the two scripts; test_the_findings_collector_lists_every_failing_row_it_fetched pins the residual named above.
  • test/test_babysit_guidance_gates.py -- the spec/skill pin flips from "declares the divergence" to "names both enforcers and no divergence", and test_the_skill_states_the_collapse_rule_the_typed_provider_implements now feeds the same four rows to collapse_superseded and to the provider's _normalize_checks and asserts the same three answers.
  • test/skill_script_helpers.py -- rollup_graphql_response builds the host's nested GraphQL shape from flat fixtures; is_rollup_graphql_read routes the fake.

Mutation verification (each applied to the real script, tests run, file restored): keying on the label, dropping the event, ordering by startedAt, dropping every older-run row, ignoring the run conclusion, ignoring the row verdict, last-writer-wins within a run, grouping unidentified rows under a None key, re-introducing a createdAt collapse of StatusContexts, passing the string variables as typed -F, skipping the commit-oid head check, and returning the partial board at the page cap -- twelve mutations, each reddening at least one of the named tests.

Manual verification

Read-only runs of the modified scripts against live pull requests in this repository:

Screenshots / video

N/A -- no user-visible frontend surface; the change is in two CLI scripts and their docs.

Related Issues

Closes #11832

Pattern harvest

Rule candidate: review-prompt
Pattern: a dedup/collapse keyed on a display label (name, title) and ordered by a per-item timestamp, where the host offers a stable id for the thing being deduplicated; the displaced item is overwritten rather than kept when the order cannot be justified.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of bf7a8539b31380af69604cb8f9b496cc45f1fa93 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a re-run of failed jobs, which keeps the same run databaseId, doesn't leave the earlier attempt's FAILURE row in the rollup. Rows of one run never displace each other.

[DESIGN-REVIEWED] bf7a853

Description read

[DESCRIPTION-READ] 97740b0229415219fd5c24a3fa9521e6a1715564d4534b4c9daa750b8ce04e18

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of bf7a8539b31380af69604cb8f9b496cc45f1fa93 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The same label-keyed, startedAt-ordered overwrite still runs in the dashboard's GitHub check projection, and the description never says it was left unfixed.

Not justified as shipped

  • Item 1 — symptom-level (point patch, 1 unfixed sibling): the description gives the root cause as "the collapse used a display label as identity and a job start time as order, and overwrote rather than kept the row". I grepped for check-row collapses over statusCheckRollup and found three. monitoring/github_pull_request.py:_mark_superseded_rows is already correct, and this PR fixes pr_status.py. The third, src/kiro_crew/dashboard/source_providers/github.py:_github_checks (lines 97–127), keys on (kind, workflow label, name) and overwrites with _github_check_rank(check) >= …. It reads startedAt. So two live rows of one run, or two workflow files sharing one name:, still lose a row on the dashboard CI glyph. Fixing it needs the same GraphQL read, so it is larger than this change and can be deferred.
    Clears when: the description names dashboard/source_providers/github.py:_github_checks as the remaining sibling, left out of scope.

What this change ships

Inventory (7 items) — 6 justified

Intent: the PR readiness report must never call a board green while one of its checks is still failing. This is a FIX.

  1. The status report keeps both live rows of one run, rows from workflows sharing a name, and rows from a push run beside a PR run. It drops only cancelled rows of older cancelled runs — symptom-level (dashboard sibling unfixed)
  2. The check rollup is read through GraphQL with run, event and definition ids, instead of gh pr view --json — justified
  3. The rollup read pages up to 10 pages; a board past that reads UNAVAILABLE instead of partial — justified
  4. The read is discarded when the rollup's own commit differs from the head, on every page — justified
  5. Status contexts are no longer collapsed at all — justified
  6. pr_findings.py switches to the same GraphQL read under the existing parity pin, and its no-collapse behaviour is pinned — justified
  7. The spec, the babysit skill and rationale.md drop the "diverges" / "does NOT yet follow" wording — justified

[FIRST-PRINCIPLES-REVIEWED] bf7a853

Description read

[DESCRIPTION-READ] b0b884b9fb8e66821bc8f5f7e202b8c18d72f990c64dfc3d7bf9eda5b3aecc59

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed bf7a8539b31380af69604cb8f9b496cc45f1fa93 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] bf7a853

Verdict parsed from the review's SHA-scoped output markers for commit bf7a8539b31380af69604cb8f9b496cc45f1fa93.

False positive or not applicable? A repository writer can comment:
/ai-review override fable bf7a8539b31380af69604cb8f9b496cc45f1fa93: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of bf7a8539b31380af69604cb8f9b496cc45f1fa93 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] bf7a853

False positive or not applicable? A repository writer can comment:
/ai-review override gpt bf7a8539b31380af69604cb8f9b496cc45f1fa93: <one-sentence reason>

@NicholasRBowers
NicholasRBowers force-pushed the fix/collapse-superseded-live-runs-11832 branch from a7e19eb to 3754676 Compare October 3, 2026 12:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@NicholasRBowers
NicholasRBowers force-pushed the fix/collapse-superseded-live-runs-11832 branch 2 times, most recently from 93cc484 to 5edeeaf Compare October 3, 2026 12:45
@NicholasRBowers

Copy link
Copy Markdown
Contributor Author
  • fixed span=5e77434ee71a

StatusContexts are no longer collapsed at all: collapse_superseded keeps every status row, and createdAt is gone from ROLLUP_QUERY (both scripts, parity-pinned). Landed in 5edeeaf.
The subtraction is taken because the host reports one status per context (0 duplicate contexts measured across the rollups of #11783 and #16496-#16501) and the provider keeps every status row, so a rule here was a second place to be wrong. One premise corrected for the record: on base a gh pr view StatusContext row DOES carry startedAt (measured: the PR Readiness row on #11783 has startedAt: 2026-09-24T05:04:58Z, byte-identical to GraphQL createdAt), so base collapsed dated contexts too; the createdAt branch was a port of that, not new behaviour. Removing it is the smaller surface either way.
Pinned by test_status_contexts_are_never_collapsed (an older FAILURE beside a newer SUCCESS of one context still counts, both orders); re-introducing the collapse reddens it.

@NicholasRBowers

Copy link
Copy Markdown
Contributor Author
  • fixed span=236a9866855b

Owner, repository and cursor now travel through gh's -f (raw string); only n stays -F, since Int! needs the typed form. Applied in both scripts (the parity pin keeps them byte-identical). Landed in 5edeeaf.
Pinned by test_the_rollup_query_selects_every_field_the_collapse_keys_on (asserts -f o=, -f r=, -F n=) and the paging test (asserts -f c=); reverting to -F reddens the former.
The same shape exists on main in unresolved_thread_count (pr_status.py) and iter_unresolved_threads (pr_findings.py), which this PR does not touch; noted here rather than widened into this diff.

…label

collapse_superseded keyed a CheckRun on ("run", workflowName, name) and kept
the row with the latest startedAt; the displaced row was overwritten, never
added to `undecidable`. Three shapes dropped a LIVE row: two check runs of ONE
workflow run (a workflow publishing under its own job's name), two workflow
files sharing one `name:`, and one file dispatching on `push` and
`pull_request` for the same commit. The run fields the correct rule needs are
not in `gh pr view --json statusCheckRollup` rows (measured: no run id,
definition id or event), so the rollup is now read through GraphQL in its own
call, keeping the stated contract: soft degradation to UNAVAILABLE, the head
re-checked beside the rollup (now on every page, and against the rollup's own
commit), a parity-pinned copy in pr_findings.py.

The collapse adopts the rule the structured monitor's GitHub provider enforces
(_mark_superseded_rows, #11830/#13394): identity is (workflow definition id,
run event, check name); newest is the greatest run id; a row is removed only
when it belongs to an older run that itself concluded CANCELLED and the row is
COMPLETED+CANCELLED; rows of one run, rows without a run identity, a replaced
round's FAILURE rows, a cancelled row inside a live run and a cancelled newest
run all stay. StatusContexts collapse by context on createdAt, ties kept.

Tests: both arrival orders for the same-run pair (the issue's shape, red on
base), one fixture per clause of the rule, the GraphQL selection pinned by
bytes, paging / mid-pagination head move / page cap, and a cross-check that the
tool and the provider answer identically on the same rows. Eleven mutations
each redden at least one named test.

Closes #11832
@NicholasRBowers

Copy link
Copy Markdown
Contributor Author
  • accepted-and-deferred span=1836b9879cb6

Confirmed on bytes: dashboard/source_providers/github.py:_github_checks keys on (kind, workflow label, name) (_github_check_identity) and overwrites by startedAt (_github_check_rank), the same rule this PR removes from pr_status.py. It feeds the dashboard CI glyph and checks panel, not the merge decision, and the fix needs _github_rollup_read moved to the GraphQL read inside the provider's sandboxed runner -- larger than this change.
Tracked in #16519 (labels deferred-finding + needs-triage, assigned, Due: 2026-10-17), with the reference shape named. The PR description's "What changed" section now names it as the remaining sibling left out of scope; the frozen Goal / Not-a-goal sections were not edited.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@bolichen97
bolichen97 merged commit 9a4ae45 into main Oct 4, 2026
106 of 107 checks passed
@bolichen97
bolichen97 deleted the fix/collapse-superseded-live-runs-11832 branch October 4, 2026 03:14
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

prepare-pr status tool: collapse_superseded can discard one of two LIVE check runs of the same workflow run

2 participants