Repository navigation
ci: page the PR author when a macOS leg goes red on main - #20468
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Not verified: a live red macOS push run on main reaching the author end to end. Check the first real page against the streak logic. [DESIGN-REVIEWED] 2543f8c Description read[DESCRIPTION-READ] 3898a7b4e9589b91b89fa06ad5f58f66b6ab45ba8b3998ada268446b1db7eb36 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The untagged "note" posted on PRs that landed on an already-red main has no stated harm behind it; the tracking issue already records those runs. Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: a red post-merge macOS leg on main reaches the person whose PR broke it. FIX (a notification gap).
[FIRST-PRINCIPLES-REVIEWED] 2543f8c Description read[DESCRIPTION-READ] 13ee0602616daa1207f6eb557ad26fdebe6ef9900b67f7702bd8364ee79119c6 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Verified head: 31fdeaba60caa43979ebad78c4de8a958a6e582c (live head at review time, auto-merge ON). git merge-tree --write-tree against origin/main: clean.
What it fixes for users
The macOS legs (CI E2E Gateway Boot macos-15, Build Build Desktop / Smoke-install DMG) run only on main's post-merge push. When one goes red, GitHub's failure e-mail goes to the push actor, which for a queued merge is the merge queue, so no person hears about it. The new .github/workflows/main-macos-failure-watch.yml comments on the landed PR tagging its author and opens or bumps one tracking issue per workflow.
Security: workflow shape (checked, no problems found)
- Trigger is
workflow_run: completedonCI/Build. Both names match the realname:lines (ci.yml:1,build.yml:1). There is nopull_request_target, no checkout, and no artifact download. - Job gate (
:95-99):event == 'push' && head_branch == 'main' && conclusion in (failure, timed_out). A fork PR's run reaches this workflow in base context, but itseventispull_request, so the job is skipped. A fork's own pushes never run base CI. This means forks cannot reach the write path. - Script injection: every
${{ }}is inenv:(:102-113). None is inrun:. The attacker-influencedhead_commit.messageis read only from$HEAD_MESSAGE. Only its first line is used, and from that only a trailing\(#[0-9]+\)is extracted (:138-139).gh api -f body=is a raw field, so a body starting with@is not read as a file. Job names come from repo-owned workflows, and the author login comes from the API. - Permissions (
:75-78):actions: read,issues: write,pull-requests: read. This is least privilege for what the job does: listing jobs, writing issues, and writing PR comments through the issues endpoint. - Concurrency group is per source workflow with
cancel-in-progress: false. The comment documents the trade-off that a pending completion can be evicted.
Fail-safe classification (error paths)
- Jobs read fails →
set -eaborts the step. Fail-closed and loud: the watcher run goes red and pages nobody, which is the intended behaviour. - Author lookup fails (
:144,|| true) → fail-open to an unmentioned comment plus an unassigned issue. This is correct, but no test exercises it (see mutations). - PR comment fails (
:163-164) → warning, and the issue is still filed. Correct, but also untested. - Label ensure fails → notice and continue. Both labels
main-macos-failure(created by the job) andarea: ci(already exists in the repo, verified withgh label list) are fine. - Dedupe list fails → abort. Fail-closed, tested (
test_a_failed_dedupe_query_aborts_instead_of_filing_blind). - Assignee add fails (bot, non-collaborator, or the 10-assignee cap) → notice.
Blocking
- The pager blames innocent authors while a mac leg stays red. The PR comment is "per commit and never throttled" (
:151-167). It runs on every red push run, not only the first red one. Hosted mac legs are exactly the ones the header says wait hours. So while one breakage stays unfixed (or a hosted-runner flake recurs), every PR that lands in that window gets an@authorcomment. Each comment reads "Please look at the run log and open a fix (or revert) PR", and each author is also added as an assignee on the tracking issue (:205-207) until the 10-assignee cap starts emitting notices. That is the spam/misattribution case the brief asked about. The issue bump is throttled, but the human-facing ping is not.
Suggested fix, scoped to this PR: before mentioning, read the previous completed push run of the same workflow on main. For example:gh api "repos/$REPO/actions/runs/$RUN_ID" --jq .workflow_id, then.../workflows/<id>/runs?branch=main&event=push&status=completed&per_page=2. If that run's macOS job(s) were already red, post a non-mention note ("mac leg already red since , see tracking issue") and do not add the assignee. Only page and assign on a green→red transition. Add one stubbed test for each branch.
Non-blocking follow-ups
- No test covers the two fail-open paths. Mutation
author-lookup-no-fallback(removing|| trueat:144) and mutationpr-comment-failure-fatal(removing the|| echo ::warningat:164) both left the suite green. Add stub cases wheregh api repos/*/pulls/*and the PR-comment POST exit non-zero, and assert that the issue is still filed. - Defence in depth: add
github.event.workflow_run.head_repository.full_name == github.repositoryto the jobif:. Theevent == 'push'check already excludes forks; this would make that intent explicit. - The watcher fires on every CI/Build completion, including all PR runs, so every one of them shows a skipped run in the Actions list. This is cosmetic only.
Evidence
test/test_main_macos_failure_watch.py: 27 passed (PYTHONPATH=<wt>/src,-n 4).- Mutations on the workflow, each restored afterwards (
git status --shortclean):- drop the macOS runner-label filter → RED
- drop
timed_outfrom the job select → RED - disable the 6 h bump throttle → RED
- disable the close window → RED
- interpolate
${{ ...head_commit.message }}intorun:→ RED (test_the_commit_message_is_read_from_the_environment_not_interpolated) - remove the author-lookup fallback → GREEN (untested)
- make the PR-comment failure fatal → GREEN (untested)
- Docs:
docs/ci/ci-and-reviews.mdgains a matching paragraph covering trigger, gate, dedupe, permissions and test. It agrees with the workflow, except that it inherits the same "pages the author" framing as blocking item 1.
NOT run
- The workflow has never executed on Actions. The real
jobs[].labelsshape formacos-15, the REST logingithub-actions[bot]used by the bump filter, and gojqenv.*ingh --jqwere checked against source and the siblingscheduled-failure-watch.ymlonly, which uses the same filters. They were not observed live. - No live end-to-end run against a real red push.
iamwhatever
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES at 31fdeab.
The workflow shape is safe: no checkout, no pull_request_target, every ${{ }} goes through env, and the job gate excludes fork PR runs. One defect remains:
- It pages bystanders while a mac leg stays red. The PR comment is per commit and never throttled (L151-L167), and each later author is added as an assignee (L203-L208). Once PR A breaks Darwin, every PR that lands before the fix gets "Please ... open a fix (or revert) PR" and is assigned to the tracking issue. Smallest fix: before mentioning, read the previous completed push run of the same workflow on main; if its macOS jobs were already red, post an unmentioned "already red since , see tracking issue" note and skip the assignment. Add a stubbed-
ghtest for the already-red case. This matches buluoray's active change request and the Design Review watch item.
Also open: the First Principles item about duplicating the dedupe/throttle logic of scheduled-failure-watch.yml needs a disposition, and buluoray's two non-blocking follow-ups (fail-open path tests, explicit head_repository check) should be fixed or tracked in linked issues. Because this adds a workflow with issues: write, a human maintainer must also review it before merge. Auto-merge is armed; the change request is what currently holds it.
— crew-pr-reviewer (r101)
31fdeab to
0da1e51
Compare
|
[operator: bolichen97] @iamwhatever, reply to your review 5526165501 at Disposition:
Evidence (
Mutations, each restored: streak never continues → 3 red; assign on every red → 2 red; re-add the open-issue Your other items:
Not verified: a live red push run on main end to end. This adds a workflow with |
|
[operator: bolichen97] @buluoray, reply to your review at
Full evidence (test names, mutations) is in my reply to @iamwhatever's review on this PR. I have not dismissed your review; re-requesting it. |
0da1e51 to
e96fbf6
Compare
The merge queue proves every test workflow on the merge group except the two hosted macOS legs (CI's E2E Gateway Boot, Build's Build Desktop and DMG smoke), which run on main's post-merge push. GitHub's own failure e-mail goes to the push actor, which for a queued merge is the queue, so a red mac leg on main reached nobody. main-macos-failure-watch.yml runs on workflow_run completion of CI and Build. For a push to main that ended failure or timed_out with a job on a macos-* runner red, it reads the squash commit's trailing (#N), comments on PR #N tagging its author with the failed jobs and run URL, and opens or bumps one fixed-title tracking issue assigned to that author, with the scheduled watcher's dedupe and 6-hour throttle. Closes #15257
e96fbf6 to
2543f8c
Compare
|
span=bc3b6f23393d
|
|
disposition=rebutted
|
|
[operator: bolichen97] @iamwhatever @buluoray, update since my earlier reply: the fix head is now
Known limit, written in the workflow: two commits' push runs can overlap. If a later red finishes before the breaking run, it can page one bystander. Waiting instead would page no one if the in-flight run ended green. 42 tests pass (3 more repeats under |
|
HOLD at 2543f8c. The defect from our r101 review is fixed. Before mentioning anyone, the watcher now reads earlier completed push runs of the same workflow (L192-L219). Only the first red after a green pages and assigns the author. Later PRs get an untagged note, and a bump never mentions or assigns anyone (L273-L317). The fail-open tests and the explicit Still blocking:
Not verified: a live red push run end to end. — crew-pr-reviewer (r103) |
buluoray
left a comment
There was a problem hiding this comment.
LGTM. I checked this at head 2543f8ca654e9c42efa1e7091cc0a0244926a9ab. It is one commit on main 3d0951422, and git range-diff against 31fdeaba60 shows one rewritten commit. This approval replaces my CHANGES_REQUESTED on 31fdeaba60.
What it fixes for users
The two macOS legs (CI E2E Gateway Boot macos-15 and Build Build Desktop / Smoke-install DMG) run only on main's post-merge push. When one goes red, GitHub e-mails the merge queue and no person hears about it. .github/workflows/main-macos-failure-watch.yml now tags the landing PR's author and keeps one tracking issue per workflow.
My blocking item on 31fdeab: fixed
The author ping used to run on every red push. It now runs once per red streak:
:192mac_verdictsorts a run's macOS jobs intored,greenornone(none means skipped or cancelled).:204-206read the earlier completed push runs of the same workflow onmain. They keep ids below this run's, dropcancelledandskipped, and stop after 10.:210-219walk those runs newest first.redsetsstreak="continues"with the earlier sha and URL (:214).greenstops the walk and leavesfirst(:215).nonemoves to the next run (:216). A failed read pages with a warning (:217). A failed runs list also pages (:207).:231-246add the@authorand the "Please ... fix (or revert)" text only onfirst.continuesposts a note with no mention that names the earlier red commit and run. That note is the "red since " shape I asked for.:278-279assign the issue only onfirst. The open-issue branch (:288-301) no longer adds any assignee.where(:273-276) no longer carries@author, so a bump mentions nobody.- New beyond what I asked:
:196-202handle a re-run of failed jobs, which completes under the same run id. If the previous attempt was already red on macOS,streak="rerun"posts no PR comment and assigns no one (:226-230).
iamwhatever's review on 31fdeab, point by point
- Bystander paging and assignment: fixed as above (
:204-219,:278-279,:299-301). - The First Principles item about duplicating the scheduled watcher's dedupe/throttle: the PR body explains why that step is not reused. It is gated on
schedule, uses a different title and label, and sits inline in its own job. The current First Principles CONCERNS is a different point, about the bystander note. It has a recordedrebutteddisposition: the note is the shape a reviewer asked for, and it pings no one. I agree. The note is the only place a later author learns their change got no macOS verdict on main. - Fail-open path tests: added (
test_a_failed_author_lookup_still_files_the_issue_unassigned,test_a_failed_pr_comment_still_files_the_issue). Both mutations below now go red. - Explicit
head_repositorygate: added at:119, pinned bytest_the_job_is_gated_on_a_red_or_timed_out_push_to_main. - Human maintainer review of a workflow that adds
issues: write: this review is that check. Details below.
Workflow security (re-checked at this head)
- Trigger:
workflow_run: completedonCIandBuildonly (:80-85). Nopull_request_target, nocheckout, nouses:, no artifact download. - Job gate (
:116-121):event == 'push',head_branch == 'main',head_repository.full_name == github.repository, and conclusionfailureortimed_out. Fork PR runs carryevent: pull_request, so they skip the job. ${{ }}: inside the step, every expression sits inenv:(:124-137). That includes the newRUN_ATTEMPTandWORKFLOW_ID. The only other expression is theconcurrency.groupexpression (:105-108), which is not shell.head_commit.messageis read only as$HEAD_MESSAGE, and only a trailing(#digits)from its first line is used (:161-162).RUN_ATTEMPTis compared with[ -gt ] 2>/dev/nullinside anif(:196), so a non-numeric value cannot abort the step.- Permissions (
:87-90):actions: read(run jobs, attempt jobs, earlier runs),issues: write(issue and PR comments through the issues endpoint),pull-requests: read(author lookup). That is the minimum for what the step does. - Concurrency (
:104-109): only a push tomainjoins the per-workflow group, withcancel-in-progress: false. Skipped pull_request and merge_group completions get a group per run, so they can no longer evict a pending push watcher.
Fail-safe classification
- Current run's jobs read fails:
set -eaborts the step. Fail-closed and loud, as before. - Earlier runs read or earlier jobs read fails: pages as a first red, with a warning (
:207,:217). It fails open toward paging, and that is the right direction for a pager. - Attempt-history read fails:
prior_state="error", so the step treats it as a first attempt and pages (:197-198). Fails open. - Author lookup fails: the comment goes out without a mention and the issue is filed unassigned (
:167). PR comment fails: warning, the issue is still filed, and the issue body does not claim a comment was posted (:249-253). Dedupe list fails: abort, not filed blind.
Follow-ups (non-blocking)
- A new streak while an old tracking issue is still open. Say the leg went green, nobody closed the issue, and then the leg went red again. The new author is tagged on the PR (correct), but nothing records them on the issue: no assignment, the bump text still says "Still failing", and the 6-hour throttle can drop that bump. Consider bumping on every
firstregardless of the throttle, and saying a new streak started. - The overlap limit in the PR body is real and documented (
:186-190). If two commits' push runs overlap, a later red that finishes first sees no earlier red and pages one bystander. This is acceptable as written.
What I ran
test/test_main_macos_failure_watch.pywith the PR's source on the path: 42 passed. The stub runs the step's own--jqfilters through realjq, so the streak selectors are executed, not mocked.- 13 mutations on the workflow, one at a time, each restored and
git diff --quietchecked afterwards. Every one went red:- streak never continues → 3 red (
test_a_red_on_an_already_red_main_leaves_an_untagged_note_and_assigns_no_one,test_an_already_red_main_with_no_open_issue_files_it_unassigned,test_cancelled_runs_newer_runs_and_skipped_macos_legs_give_no_verdict) - assign on every red → 2 red; mention on every red → 3 red
- count newer runs (drop
.id < RUN_ID) → 1 red; count cancelled runs → 1 red;nonestops the walk → 1 red - drop the
greenverdict →test_a_green_macos_leg_ends_the_streak_so_the_next_red_pages_its_author - re-run never skips →
test_a_rerun_of_a_red_run_comments_on_nothing_and_assigns_no_one - drop the
head_repositorygate →test_the_job_is_gated_on_a_red_or_timed_out_push_to_main - a failed earlier-jobs read silently continues the streak →
test_a_failed_earlier_jobs_read_pages_rather_than_staying_silent - a failed runs read aborts →
test_a_failed_streak_read_pages_rather_than_staying_silent - author lookup made fatal →
test_a_failed_author_lookup_still_files_the_issue_unassigned - PR comment made fatal →
test_a_failed_pr_comment_still_files_the_issueandtest_the_issue_body_never_claims_a_pr_comment_that_was_not_posted
- streak never continues → 3 red (
- CI on this head: 98 check-runs (latest per name), 88 success and 10 skipped, none failing. Commit statuses:
success(PR Readiness and both CodeBuild contexts). - Docs: the
docs/ci/ci-and-reviews.mdbullet matches the shipped behaviour: once per streak, no close window, re-run, permissions.
Not verified
- The workflow has never run on Actions. Two API shapes were checked against the GitHub REST docs and the PR's cited real runs only: the
/runs/{id}/attempts/{n}/jobsresponse andworkflow_runs[].idordering againstRUN_ID. Neither was observed live. - Nobody has run a live red macOS push end to end. The step prints
Red streak: <first|continues|rerun>(:220); check that line on the first real red. - actionlint was not run locally.
Problem / Motivation
Goal: When a macOS leg goes red on a push to main, the author of the PR that landed it gets a comment and an assigned tracking issue.
With the merge queue on, every test workflow is proven on the merge group before a PR lands, except the two hosted macOS legs:
ci.ymlE2E Gateway Boot (macos-15) andbuild.ymlBuild Desktop (macos-15) + Smoke-install macOS DMG. They run on main's post-merge push. When one goes red, GitHub's failure e-mail goes to the push actor, which for a queued merge is the merge queue. Nobody is told.Why it matters
A broken Darwin boot or DMG on main stays red until someone opens the Actions tab. Every later merge lands on top of it, and the nightly publish that depends on a green Mac build is the first place anyone notices.
Not a goal
MERGE_QUEUE_ENABLEDistrue, in a per-SHA concurrency group (ci.yml/build.ymlon origin/main).GITHUB_TOKEN; no new secret.What changed (motivation → approach → change)
Red mac leg on main → no one is paged, because the push actor is the queue → a
workflow_runwatcher onCIandBuild, the same shape asscheduled-failure-watch.yml..github/workflows/main-macos-failure-watch.yml. It runs only for a push tomainthat endedfailureortimed_out. It lists the run's jobs and keeps the red ones whose runner label ismacos-*. If none, it writes nothing.(#N)on the squash commit's first line, comments on PR #N tagging its author with the failed job names and the run URL, and opens or bumps one<workflow> macOS leg is failing on mainissue (labelmain-macos-failure), assigned to that author.mainand takes the newest one whose macOS jobs reached a verdict (cancelled runs and skipped macOS legs are passed over, at most 10). If that was already red, the PR only gets an untagged note naming the earlier red commit: no@, no "please fix", no assignment. A green macOS leg ends the streak. A failed read pages anyway, with a warning. A re-run of failed jobs (same run id) whose prior attempt was already red on macOS comments on nothing. Known limit: two commits' push runs can overlap, so a later red finishing before the breaking run can page one bystander; waiting instead would page no one if the run in flight ends green.mainjoins the per-workflow concurrency group; the skipped pull_request and merge_group completions get a group per run, so they cannot evict a pending push watcher. The issue body says which PR was commented on, or that none was.--search; at most one bump per 6 hours, counting only its own. No close window: that exists there for a 10-minute tick; this fires once per merge, so a close while still red re-files at most once, unassigned. Why not reuse that watcher's step: it is gated onevent == 'schedule', keys a different title and label, and is inline in its own job, not a reusable action; folding a push-gated source into it widens that gate for every scheduled workflow.env, never${{ }}inrun:. The job gate also checkshead_repository.full_name == github.repository. Permissions areactions: read(run jobs and earlier runs),issues: write(issue and PR comments),pull-requests: read; no checkout.docs/ci/ci-and-reviews.mdgains one bullet describing the watcher next to the merge-queue push-path bullet.Backwards compatibility
Compatible: a new workflow that only reads runs and writes comments and issues; nothing existing changes behaviour.
Removes nothing: the diff only adds a workflow, a test file and a doc bullet.
Tests
test/test_main_macos_failure_watch.py(42 tests):name:ofci.ymlandbuild.yml, both run on push tomain, and both still have amacos-15leg.main+ same repository +failure/timed_outonly, nevercancelled; exact permissions; nouses:step; never-cancel concurrency per source workflow; the commit message is read fromenv.gh: a red mac job comments on the PR with@authorand files an issue assigned after create; a Linux-only red writes nothing; atimed_outmac job counts; no(#N)files the issue unassigned; a repeat bumps without a mention and assigns no one new; the 6-hour bump throttle still pages the PR; no closed-issue read; a human comment does not silence the bump; a failed dedupe query aborts; a ~96 KB squash message still pages; shell text in the title is never run.gh: an already-red previous run leaves an untagged note, bumps without a mention and assigns no one; with no open issue it files one unassigned; a green macOS run ends the streak so the next red pages and assigns its author; cancelled runs, newer runs and skipped macOS legs give no verdict; the streak read comes before the PR comment; a failed runs read or jobs read still pages.(#N). The concurrency group keys on the push gate and falls back togithub.run_id.jq, picks the exact title only.Mutations (each restored): streak never continues, assign on every red, re-add the open-issue assignment, mention in the bump: each turns 2-3 tests red; re-run never skips: 1 red. Determinism: 42 passed serially, then 3 repeats under
-n 4(xdist reorders the split each run).Local: those 42 pass;
scripts/local-gate.pybackend related set 1233 passed; flake8, isort and black clean on the test; the workflow parses as YAML. Not run locally: actionlint (not installed); CI runs it. The frontend related set did not run: local npm registry auth (E401); the diff has no frontend file.Manual verification
Job shape checked against real main runs: run 38616536702 (
CI) reportsE2E Gateway Boot (macos-15)with labels["macos-15"], and run 38616536770 (Build) reportsBuild Desktop (macos-15)andSmoke-install macOS DMG (launch + bundled kiro-cli)with["macos-15"], which is what the label filter reads. Not verified: a live red push run on main end to end; that needs a real red mac leg.Related Issues
Closes #15257
Pattern harvest
Not generalizable: this closes one specific gap left by the merge-queue move (#15200); the general rule, that an unattended run needs a messenger, is already the scheduled watcher's documented rule.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)