Skip to content

ci: page the PR author when a macOS leg goes red on main - #20468

Merged
iamwhatever merged 1 commit into
mainfrom
fix/macos-red-main-pager-15257
Oct 16, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/macos-red-main-pager-15257

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When a macOS leg goes red on a push to main, the author of the PR that landed it gets a comment and an assigned tracking issue.

With the merge queue on, every test workflow is proven on the merge group before a PR lands, except the two hosted macOS legs: ci.yml E2E Gateway Boot (macos-15) and build.yml Build Desktop (macos-15) + Smoke-install macOS DMG. They run on main's post-merge push. When one goes red, GitHub's failure e-mail goes to the push actor, which for a queued merge is the merge queue. Nobody is told.

Why it matters

A broken Darwin boot or DMG on main stays red until someone opens the Actions tab. Every later merge lands on top of it, and the nightly publish that depends on a green Mac build is the first place anyone notices.

Not a goal

  • Changing which jobs run on a push to main or on the merge group. Main already runs only the two macOS legs on push when MERGE_QUEUE_ENABLED is true, in a per-SHA concurrency group (ci.yml / build.yml on origin/main).
  • Paging on Linux or Windows reds on main. Those run on the PR and the merge group first.
  • Slack, e-mail or any external endpoint. The page is a GitHub comment and issue under GITHUB_TOKEN; no new secret.

What changed (motivation → approach → change)

Red mac leg on main → no one is paged, because the push actor is the queue → a workflow_run watcher on CI and Build, the same shape as scheduled-failure-watch.yml.

  • New .github/workflows/main-macos-failure-watch.yml. It runs only for a push to main that ended failure or timed_out. It lists the run's jobs and keeps the red ones whose runner label is macos-*. If none, it writes nothing.
  • It reads the trailing (#N) on the squash commit's first line, comments on PR #N tagging its author with the failed job names and the run URL, and opens or bumps one <workflow> macOS leg is failing on main issue (label main-macos-failure), assigned to that author.
  • It pages once per red streak. First it reads the earlier completed push runs of the same workflow on main and takes the newest one whose macOS jobs reached a verdict (cancelled runs and skipped macOS legs are passed over, at most 10). If that was already red, the PR only gets an untagged note naming the earlier red commit: no @, no "please fix", no assignment. A green macOS leg ends the streak. A failed read pages anyway, with a warning. A re-run of failed jobs (same run id) whose prior attempt was already red on macOS comments on nothing. Known limit: two commits' push runs can overlap, so a later red finishing before the breaking run can page one bystander; waiting instead would page no one if the run in flight ends green.
  • Later authors are never added to an open tracking issue, and a bump never mentions anyone. The untagged note stays because a PR landed on a red main never gets a macOS verdict of its own; without it, its author has no sign their change shipped unverified on Darwin.
  • Only a push to main joins the per-workflow concurrency group; the skipped pull_request and merge_group completions get a group per run, so they cannot evict a pending push watcher. The issue body says which PR was commented on, or that none was.
  • Dedupe and throttle follow the scheduled watcher: exact-title match by label, never --search; at most one bump per 6 hours, counting only its own. No close window: that exists there for a 10-minute tick; this fires once per merge, so a close while still red re-files at most once, unassigned. Why not reuse that watcher's step: it is gated on event == 'schedule', keys a different title and label, and is inline in its own job, not a reusable action; folding a push-gated source into it widens that gate for every scheduled workflow.
  • The commit message reaches the script through env, never ${{ }} in run:. The job gate also checks head_repository.full_name == github.repository. Permissions are actions: read (run jobs and earlier runs), issues: write (issue and PR comments), pull-requests: read; no checkout.
  • docs/ci/ci-and-reviews.md gains one bullet describing the watcher next to the merge-queue push-path bullet.

Backwards compatibility

Compatible: a new workflow that only reads runs and writes comments and issues; nothing existing changes behaviour.

Removes nothing: the diff only adds a workflow, a test file and a doc bullet.

Tests

test/test_main_macos_failure_watch.py (42 tests):

  • Wiring: the two watched names equal the real name: of ci.yml and build.yml, both run on push to main, and both still have a macos-15 leg.
  • Gating: push + main + same repository + failure/timed_out only, never cancelled; exact permissions; no uses: step; never-cancel concurrency per source workflow; the commit message is read from env.
  • Executed with a stubbed gh: a red mac job comments on the PR with @author and files an issue assigned after create; a Linux-only red writes nothing; a timed_out mac job counts; no (#N) files the issue unassigned; a repeat bumps without a mention and assigns no one new; the 6-hour bump throttle still pages the PR; no closed-issue read; a human comment does not silence the bump; a failed dedupe query aborts; a ~96 KB squash message still pages; shell text in the title is never run.
  • Red streak, stubbed gh: an already-red previous run leaves an untagged note, bumps without a mention and assigns no one; with no open issue it files one unassigned; a green macOS run ends the streak so the next red pages and assigns its author; cancelled runs, newer runs and skipped macOS legs give no verdict; the streak read comes before the PR comment; a failed runs read or jobs read still pages.
  • Re-run: attempt 2 after a red attempt comments on nothing and assigns no one; after a green attempt it pages; attempt 1 never reads attempt history.
  • Fail-open paths: a failed author lookup still files the issue unassigned; a failed PR comment still files and assigns the issue.
  • The issue body names the commented PR, and says none was when the comment failed or there is no (#N). The concurrency group keys on the push gate and falls back to github.run_id.
  • The dedupe selector, run with jq, picks the exact title only.

Mutations (each restored): streak never continues, assign on every red, re-add the open-issue assignment, mention in the bump: each turns 2-3 tests red; re-run never skips: 1 red. Determinism: 42 passed serially, then 3 repeats under -n 4 (xdist reorders the split each run).

Local: those 42 pass; scripts/local-gate.py backend related set 1233 passed; flake8, isort and black clean on the test; the workflow parses as YAML. Not run locally: actionlint (not installed); CI runs it. The frontend related set did not run: local npm registry auth (E401); the diff has no frontend file.

Manual verification

Job shape checked against real main runs: run 38616536702 (CI) reports E2E Gateway Boot (macos-15) with labels ["macos-15"], and run 38616536770 (Build) reports Build Desktop (macos-15) and Smoke-install macOS DMG (launch + bundled kiro-cli) with ["macos-15"], which is what the label filter reads. Not verified: a live red push run on main end to end; that needs a real red mac leg.

Related Issues

Closes #15257

Pattern harvest

Not generalizable: this closes one specific gap left by the merge-queue move (#15200); the general rule, that an unattended run needs a messenger, is already the scheduled watcher's documented rule.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 14, 2026 20:11
@bolichen97
bolichen97 enabled auto-merge (squash) October 14, 2026 20:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2543f8ca654e9c42efa1e7091cc0a0244926a9ab touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2543f8ca654e9c42efa1e7091cc0a0244926a9ab — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Not verified: a live red macOS push run on main reaching the author end to end. Check the first real page against the streak logic.

[DESIGN-REVIEWED] 2543f8c

Description read

[DESCRIPTION-READ] 3898a7b4e9589b91b89fa06ad5f58f66b6ab45ba8b3998ada268446b1db7eb36

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 2543f8ca654e9c42efa1e7091cc0a0244926a9ab — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The untagged "note" posted on PRs that landed on an already-red main has no stated harm behind it; the tracking issue already records those runs.

Not justified as shipped

  • Item 4 — oversized: the Goal is that "the author of the PR that landed it gets a comment". It never asks for a comment on bystander PRs, yet pr_body="Note: the \$SOURCE_NAME` macOS leg on `main` was already red before this PR's squash commit…"posts one on every later red PR during a streak. Subtraction: delete theelsebranch that builds the note, and skip the PR comment whenstreak = continues. The issue bump at $where (PR #N)` already names the PR.
    Clears when: the bystander-note branch is removed, or a human amends the Goal to include it.

What this change ships

Inventory (8 items) — 7 justified

Intent: a red post-merge macOS leg on main reaches the person whose PR broke it. FIX (a notification gap).

  1. A red macOS job on a main push now comments on the landing PR and tags its author — justified
  2. A fixed-title tracking issue per workflow is opened and assigned to the first red's author — justified
  3. Only the first red of a streak tags anyone; later authors are never tagged or assigned — justified
  4. Bystander PRs during a red streak get an untagged explanatory note — oversized
  5. A re-run of an already-red attempt comments on nothing — justified
  6. Issue bumps are throttled to one per 6 hours — justified
  7. A new main-macos-failure label is created automatically — justified
  8. timed_out counts as red, cancelled does not — justified

[FIRST-PRINCIPLES-REVIEWED] 2543f8c

Description read

[DESCRIPTION-READ] 13ee0602616daa1207f6eb557ad26fdebe6ef9900b67f7702bd8364ee79119c6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2543f8ca654e9c42efa1e7091cc0a0244926a9ab and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2543f8c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2543f8ca654e9c42efa1e7091cc0a0244926a9ab: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2543f8ca654e9c42efa1e7091cc0a0244926a9ab — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2543f8c

Verdict parsed from the review's SHA-scoped output markers for commit 2543f8ca654e9c42efa1e7091cc0a0244926a9ab.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2543f8ca654e9c42efa1e7091cc0a0244926a9ab: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 14, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified head: 31fdeaba60caa43979ebad78c4de8a958a6e582c (live head at review time, auto-merge ON). git merge-tree --write-tree against origin/main: clean.

What it fixes for users

The macOS legs (CI E2E Gateway Boot macos-15, Build Build Desktop / Smoke-install DMG) run only on main's post-merge push. When one goes red, GitHub's failure e-mail goes to the push actor, which for a queued merge is the merge queue, so no person hears about it. The new .github/workflows/main-macos-failure-watch.yml comments on the landed PR tagging its author and opens or bumps one tracking issue per workflow.

Security: workflow shape (checked, no problems found)

  • Trigger is workflow_run: completed on CI/Build. Both names match the real name: lines (ci.yml:1, build.yml:1). There is no pull_request_target, no checkout, and no artifact download.
  • Job gate (:95-99): event == 'push' && head_branch == 'main' && conclusion in (failure, timed_out). A fork PR's run reaches this workflow in base context, but its event is pull_request, so the job is skipped. A fork's own pushes never run base CI. This means forks cannot reach the write path.
  • Script injection: every ${{ }} is in env: (:102-113). None is in run:. The attacker-influenced head_commit.message is read only from $HEAD_MESSAGE. Only its first line is used, and from that only a trailing \(#[0-9]+\) is extracted (:138-139). gh api -f body= is a raw field, so a body starting with @ is not read as a file. Job names come from repo-owned workflows, and the author login comes from the API.
  • Permissions (:75-78): actions: read, issues: write, pull-requests: read. This is least privilege for what the job does: listing jobs, writing issues, and writing PR comments through the issues endpoint.
  • Concurrency group is per source workflow with cancel-in-progress: false. The comment documents the trade-off that a pending completion can be evicted.

Fail-safe classification (error paths)

  • Jobs read fails → set -e aborts the step. Fail-closed and loud: the watcher run goes red and pages nobody, which is the intended behaviour.
  • Author lookup fails (:144, || true) → fail-open to an unmentioned comment plus an unassigned issue. This is correct, but no test exercises it (see mutations).
  • PR comment fails (:163-164) → warning, and the issue is still filed. Correct, but also untested.
  • Label ensure fails → notice and continue. Both labels main-macos-failure (created by the job) and area: ci (already exists in the repo, verified with gh label list) are fine.
  • Dedupe list fails → abort. Fail-closed, tested (test_a_failed_dedupe_query_aborts_instead_of_filing_blind).
  • Assignee add fails (bot, non-collaborator, or the 10-assignee cap) → notice.

Blocking

  1. The pager blames innocent authors while a mac leg stays red. The PR comment is "per commit and never throttled" (:151-167). It runs on every red push run, not only the first red one. Hosted mac legs are exactly the ones the header says wait hours. So while one breakage stays unfixed (or a hosted-runner flake recurs), every PR that lands in that window gets an @author comment. Each comment reads "Please look at the run log and open a fix (or revert) PR", and each author is also added as an assignee on the tracking issue (:205-207) until the 10-assignee cap starts emitting notices. That is the spam/misattribution case the brief asked about. The issue bump is throttled, but the human-facing ping is not.
    Suggested fix, scoped to this PR: before mentioning, read the previous completed push run of the same workflow on main. For example: gh api "repos/$REPO/actions/runs/$RUN_ID" --jq .workflow_id, then .../workflows/<id>/runs?branch=main&event=push&status=completed&per_page=2. If that run's macOS job(s) were already red, post a non-mention note ("mac leg already red since , see tracking issue") and do not add the assignee. Only page and assign on a green→red transition. Add one stubbed test for each branch.

Non-blocking follow-ups

  • No test covers the two fail-open paths. Mutation author-lookup-no-fallback (removing || true at :144) and mutation pr-comment-failure-fatal (removing the || echo ::warning at :164) both left the suite green. Add stub cases where gh api repos/*/pulls/* and the PR-comment POST exit non-zero, and assert that the issue is still filed.
  • Defence in depth: add github.event.workflow_run.head_repository.full_name == github.repository to the job if:. The event == 'push' check already excludes forks; this would make that intent explicit.
  • The watcher fires on every CI/Build completion, including all PR runs, so every one of them shows a skipped run in the Actions list. This is cosmetic only.

Evidence

  • test/test_main_macos_failure_watch.py: 27 passed (PYTHONPATH=<wt>/src, -n 4).
  • Mutations on the workflow, each restored afterwards (git status --short clean):
    • drop the macOS runner-label filter → RED
    • drop timed_out from the job select → RED
    • disable the 6 h bump throttle → RED
    • disable the close window → RED
    • interpolate ${{ ...head_commit.message }} into run: → RED (test_the_commit_message_is_read_from_the_environment_not_interpolated)
    • remove the author-lookup fallback → GREEN (untested)
    • make the PR-comment failure fatal → GREEN (untested)
  • Docs: docs/ci/ci-and-reviews.md gains a matching paragraph covering trigger, gate, dedupe, permissions and test. It agrees with the workflow, except that it inherits the same "pages the author" framing as blocking item 1.

NOT run

  • The workflow has never executed on Actions. The real jobs[].labels shape for macos-15, the REST login github-actions[bot] used by the bump filter, and gojq env.* in gh --jq were checked against source and the sibling scheduled-failure-watch.yml only, which uses the same filters. They were not observed live.
  • No live end-to-end run against a real red push.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES at 31fdeab.

The workflow shape is safe: no checkout, no pull_request_target, every ${{ }} goes through env, and the job gate excludes fork PR runs. One defect remains:

  1. It pages bystanders while a mac leg stays red. The PR comment is per commit and never throttled (L151-L167), and each later author is added as an assignee (L203-L208). Once PR A breaks Darwin, every PR that lands before the fix gets "Please ... open a fix (or revert) PR" and is assigned to the tracking issue. Smallest fix: before mentioning, read the previous completed push run of the same workflow on main; if its macOS jobs were already red, post an unmentioned "already red since , see tracking issue" note and skip the assignment. Add a stubbed-gh test for the already-red case. This matches buluoray's active change request and the Design Review watch item.

Also open: the First Principles item about duplicating the dedupe/throttle logic of scheduled-failure-watch.yml needs a disposition, and buluoray's two non-blocking follow-ups (fail-open path tests, explicit head_repository check) should be fixed or tracked in linked issues. Because this adds a workflow with issues: write, a human maintainer must also review it before merge. Auto-merge is armed; the change request is what currently holds it.

— crew-pr-reviewer (r101)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 15, 2026
@bolichen97
bolichen97 force-pushed the fix/macos-red-main-pager-15257 branch from 31fdeab to 0da1e51 Compare October 15, 2026 04:23
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] @iamwhatever, reply to your review 5526165501 at 31fdeaba60.

Disposition: fixed in 0da1e510d29c1c65750b9ffe00c0cff4233beb51 (one commit, rebased on main 3d0951422).

Pages once per red streak: before any mention the step reads earlier completed push runs of the same workflow on main (actions/workflows/$WORKFLOW_ID/runs?branch=main&event=push&status=completed), ids below this run, cancelled skipped.
It takes the newest run whose macOS jobs reached a verdict (skipped legs passed over, at most 10). Red there means main was already red: the PR gets an untagged note naming that commit, no @, no "please fix", no assignment.
A green macOS leg ends the streak, so the next red pages its own author. A failed runs/jobs read pages anyway with a warning (silence on a real first red is the failure this exists for).
The open-issue branch no longer runs --add-assignee at all; where no longer carries @author, so a bump mentions nobody. A new issue is assigned only on a first red.
Kept: no checkout, no pull_request_target, every ${{ }} via env (new WORKFLOW_ID too), fork gate, plus an explicit head_repository.full_name == github.repository in if:.

Evidence (test/test_main_macos_failure_watch.py, 36 pass, stubbed gh):

  • test_a_red_on_an_already_red_main_leaves_an_untagged_note_and_assigns_no_one: your already-red case. No @, no Please, no issue edit, bump has no mention.
  • test_an_already_red_main_with_no_open_issue_files_it_unassigned
  • test_a_green_macos_leg_ends_the_streak_so_the_next_red_pages_its_author
  • test_cancelled_runs_newer_runs_and_skipped_macos_legs_give_no_verdict
  • test_the_streak_check_runs_before_the_pr_is_commented_on
  • test_a_failed_streak_read_pages_rather_than_staying_silent, test_a_failed_earlier_jobs_read_pages_rather_than_staying_silent
  • test_a_repeat_failure_bumps_the_open_issue_instead_of_filing_a_second now asserts no issue edit on the open issue.

Mutations, each restored: streak never continues → 3 red; assign on every red → 2 red; re-add the open-issue --add-assignee → 2 red; @author back in the bump → 3 red.

Your other items:

  • First Principles duplication: disposition posted separately (target=first-principles), reason in the PR body: the scheduled watcher's step is gated on event == 'schedule' and inline, so reusing it means widening that gate for every scheduled workflow.
  • buluoray's follow-ups are fixed in the same commit, not deferred: test_a_failed_author_lookup_still_files_the_issue_unassigned, test_a_failed_pr_comment_still_files_the_issue, and the head_repository check (pinned in test_the_job_is_gated_on_a_red_or_timed_out_push_to_main).

Not verified: a live red push run on main end to end. This adds a workflow with issues: write, so it still needs a human maintainer other than the pushing account to review it; I am not signing it. I have not dismissed your review; re-requesting it now.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] @buluoray, reply to your review at 31fdeaba60. Disposition: fixed in 0da1e510d29c1c65750b9ffe00c0cff4233beb51.

Blocking item 1 (bystander paging): the step now reads earlier completed push runs of the same workflow on main before any mention; if the newest one with a macOS verdict was red, the PR gets an untagged note and no assignment. Only a green-to-red transition pages and assigns. Later authors are never added to an open issue.
Fail-open paths now tested: test_a_failed_author_lookup_still_files_the_issue_unassigned (your author-lookup-no-fallback mutation) and test_a_failed_pr_comment_still_files_the_issue (pr-comment-failure-fatal).
Defence in depth: github.event.workflow_run.head_repository.full_name == github.repository added to the job if:, pinned by the gate test.
The skipped-run noise in the Actions list is cosmetic, as you note; left as is.

Full evidence (test names, mutations) is in my reply to @iamwhatever's review on this PR. I have not dismissed your review; re-requesting it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 15, 2026
@bolichen97
bolichen97 force-pushed the fix/macos-red-main-pager-15257 branch from 0da1e51 to e96fbf6 Compare October 15, 2026 04:36
The merge queue proves every test workflow on the merge group except the
two hosted macOS legs (CI's E2E Gateway Boot, Build's Build Desktop and
DMG smoke), which run on main's post-merge push. GitHub's own failure
e-mail goes to the push actor, which for a queued merge is the queue, so
a red mac leg on main reached nobody.

main-macos-failure-watch.yml runs on workflow_run completion of CI and
Build. For a push to main that ended failure or timed_out with a job on
a macos-* runner red, it reads the squash commit's trailing (#N),
comments on PR #N tagging its author with the failed jobs and run URL,
and opens or bumps one fixed-title tracking issue assigned to that
author, with the scheduled watcher's dedupe and 6-hour throttle.

Closes #15257
@bolichen97
bolichen97 force-pushed the fix/macos-red-main-pager-15257 branch from e96fbf6 to 2543f8c Compare October 15, 2026 04:49
@bolichen97

Copy link
Copy Markdown
Contributor Author

span=bc3b6f23393d
disposition=rebutted

  • Item 4: untagged note on a PR that landed on an already-red main

Kept because a human reviewer's change request named it as the fix: iamwhatever's review 5526165501 at 31fdeab asks to "post an unmentioned 'already red since , see tracking issue' note and skip the assignment".
It carries no @, no ask and no assignment, so it pages nobody; it is the only place a bystander learns their own change got no macOS verdict on main.
The Goal is unchanged; the note is the reviewer's chosen shape for the "already red" branch. If the reviewer prefers removal, it is one else branch and three assertions.
self-added: no
mechanism: reviewer-requested untagged note on the already-red branch

@bolichen97

Copy link
Copy Markdown
Contributor Author

disposition=rebutted

  • Not verified live: first real page against the streak logic

True, and stated in the PR body's Manual verification. A live red macOS push run on main needs a real red leg, which this PR cannot produce on demand.
The jobs and runs shapes the step reads were checked against real main runs (38616536702, 38616536770). Every streak branch (first, continues, rerun, green ends it, failed read) is pinned by stubbed-gh tests.
For the first real red after merge, the watcher's run log prints Red streak: <first|continues|rerun>, so the check against run history is one line to read.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] @iamwhatever @buluoray, update since my earlier reply: the fix head is now 2543f8ca654e9c42efa1e7091cc0a0244926a9ab (still one commit). Your blocking item is fixed as described above. Review-bot rounds added four more changes on top:

  • Re-run: re-running failed jobs completes again under the same run id. If the prior attempt was already red on macOS, the watcher comments on nothing and assigns no one (test_a_rerun_of_a_red_run_comments_on_nothing_and_assigns_no_one).
  • Concurrency: only a push to main joins the per-workflow group. Skipped pull_request / merge_group completions get a group per run, so they cannot evict a pending push watcher.
  • The issue body no longer says "the PR above carries a comment" when none was posted.
  • The 6 h close window is gone: it existed in the scheduled watcher for a 10-minute tick. This watcher fires once per merge, so a close while still red re-files at most once, unassigned.

Known limit, written in the workflow: two commits' push runs can overlap. If a later red finishes before the breaking run, it can page one bystander. Waiting instead would page no one if the in-flight run ended green.

42 tests pass (3 more repeats under -n 4). Your reviews are not dismissed; both are re-requested.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 15, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 15, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 2543f8c.

The defect from our r101 review is fixed. Before mentioning anyone, the watcher now reads earlier completed push runs of the same workflow (L192-L219). Only the first red after a green pages and assigns the author. Later PRs get an untagged note, and a bump never mentions or assigns anyone (L273-L317). The fail-open tests and the explicit head_repository gate are added. The First Principles bystander-note concern and the Design note have reasonable dispositions, and all checks on this head are green.

Still blocking:

  • buluoray's CHANGES_REQUESTED review is still active. It clears when buluoray re-reviews or it is dismissed.
  • This adds a workflow with issues: write (L87-L90). New write permissions in workflows are reserved for a human maintainer's review.

Not verified: a live red push run end to end.

— crew-pr-reviewer (r103)

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked this at head 2543f8ca654e9c42efa1e7091cc0a0244926a9ab. It is one commit on main 3d0951422, and git range-diff against 31fdeaba60 shows one rewritten commit. This approval replaces my CHANGES_REQUESTED on 31fdeaba60.

What it fixes for users

The two macOS legs (CI E2E Gateway Boot macos-15 and Build Build Desktop / Smoke-install DMG) run only on main's post-merge push. When one goes red, GitHub e-mails the merge queue and no person hears about it. .github/workflows/main-macos-failure-watch.yml now tags the landing PR's author and keeps one tracking issue per workflow.

My blocking item on 31fdeab: fixed

The author ping used to run on every red push. It now runs once per red streak:

  • :192 mac_verdict sorts a run's macOS jobs into red, green or none (none means skipped or cancelled).
  • :204-206 read the earlier completed push runs of the same workflow on main. They keep ids below this run's, drop cancelled and skipped, and stop after 10.
  • :210-219 walk those runs newest first. red sets streak="continues" with the earlier sha and URL (:214). green stops the walk and leaves first (:215). none moves to the next run (:216). A failed read pages with a warning (:217). A failed runs list also pages (:207).
  • :231-246 add the @author and the "Please ... fix (or revert)" text only on first. continues posts a note with no mention that names the earlier red commit and run. That note is the "red since " shape I asked for.
  • :278-279 assign the issue only on first. The open-issue branch (:288-301) no longer adds any assignee. where (:273-276) no longer carries @author, so a bump mentions nobody.
  • New beyond what I asked: :196-202 handle a re-run of failed jobs, which completes under the same run id. If the previous attempt was already red on macOS, streak="rerun" posts no PR comment and assigns no one (:226-230).

iamwhatever's review on 31fdeab, point by point

  1. Bystander paging and assignment: fixed as above (:204-219, :278-279, :299-301).
  2. The First Principles item about duplicating the scheduled watcher's dedupe/throttle: the PR body explains why that step is not reused. It is gated on schedule, uses a different title and label, and sits inline in its own job. The current First Principles CONCERNS is a different point, about the bystander note. It has a recorded rebutted disposition: the note is the shape a reviewer asked for, and it pings no one. I agree. The note is the only place a later author learns their change got no macOS verdict on main.
  3. Fail-open path tests: added (test_a_failed_author_lookup_still_files_the_issue_unassigned, test_a_failed_pr_comment_still_files_the_issue). Both mutations below now go red.
  4. Explicit head_repository gate: added at :119, pinned by test_the_job_is_gated_on_a_red_or_timed_out_push_to_main.
  5. Human maintainer review of a workflow that adds issues: write: this review is that check. Details below.

Workflow security (re-checked at this head)

  • Trigger: workflow_run: completed on CI and Build only (:80-85). No pull_request_target, no checkout, no uses:, no artifact download.
  • Job gate (:116-121): event == 'push', head_branch == 'main', head_repository.full_name == github.repository, and conclusion failure or timed_out. Fork PR runs carry event: pull_request, so they skip the job.
  • ${{ }}: inside the step, every expression sits in env: (:124-137). That includes the new RUN_ATTEMPT and WORKFLOW_ID. The only other expression is the concurrency.group expression (:105-108), which is not shell. head_commit.message is read only as $HEAD_MESSAGE, and only a trailing (#digits) from its first line is used (:161-162). RUN_ATTEMPT is compared with [ -gt ] 2>/dev/null inside an if (:196), so a non-numeric value cannot abort the step.
  • Permissions (:87-90): actions: read (run jobs, attempt jobs, earlier runs), issues: write (issue and PR comments through the issues endpoint), pull-requests: read (author lookup). That is the minimum for what the step does.
  • Concurrency (:104-109): only a push to main joins the per-workflow group, with cancel-in-progress: false. Skipped pull_request and merge_group completions get a group per run, so they can no longer evict a pending push watcher.

Fail-safe classification

  • Current run's jobs read fails: set -e aborts the step. Fail-closed and loud, as before.
  • Earlier runs read or earlier jobs read fails: pages as a first red, with a warning (:207, :217). It fails open toward paging, and that is the right direction for a pager.
  • Attempt-history read fails: prior_state="error", so the step treats it as a first attempt and pages (:197-198). Fails open.
  • Author lookup fails: the comment goes out without a mention and the issue is filed unassigned (:167). PR comment fails: warning, the issue is still filed, and the issue body does not claim a comment was posted (:249-253). Dedupe list fails: abort, not filed blind.

Follow-ups (non-blocking)

  • A new streak while an old tracking issue is still open. Say the leg went green, nobody closed the issue, and then the leg went red again. The new author is tagged on the PR (correct), but nothing records them on the issue: no assignment, the bump text still says "Still failing", and the 6-hour throttle can drop that bump. Consider bumping on every first regardless of the throttle, and saying a new streak started.
  • The overlap limit in the PR body is real and documented (:186-190). If two commits' push runs overlap, a later red that finishes first sees no earlier red and pages one bystander. This is acceptable as written.

What I ran

  • test/test_main_macos_failure_watch.py with the PR's source on the path: 42 passed. The stub runs the step's own --jq filters through real jq, so the streak selectors are executed, not mocked.
  • 13 mutations on the workflow, one at a time, each restored and git diff --quiet checked afterwards. Every one went red:
    • streak never continues → 3 red (test_a_red_on_an_already_red_main_leaves_an_untagged_note_and_assigns_no_one, test_an_already_red_main_with_no_open_issue_files_it_unassigned, test_cancelled_runs_newer_runs_and_skipped_macos_legs_give_no_verdict)
    • assign on every red → 2 red; mention on every red → 3 red
    • count newer runs (drop .id < RUN_ID) → 1 red; count cancelled runs → 1 red; none stops the walk → 1 red
    • drop the green verdict → test_a_green_macos_leg_ends_the_streak_so_the_next_red_pages_its_author
    • re-run never skips → test_a_rerun_of_a_red_run_comments_on_nothing_and_assigns_no_one
    • drop the head_repository gate → test_the_job_is_gated_on_a_red_or_timed_out_push_to_main
    • a failed earlier-jobs read silently continues the streak → test_a_failed_earlier_jobs_read_pages_rather_than_staying_silent
    • a failed runs read aborts → test_a_failed_streak_read_pages_rather_than_staying_silent
    • author lookup made fatal → test_a_failed_author_lookup_still_files_the_issue_unassigned
    • PR comment made fatal → test_a_failed_pr_comment_still_files_the_issue and test_the_issue_body_never_claims_a_pr_comment_that_was_not_posted
  • CI on this head: 98 check-runs (latest per name), 88 success and 10 skipped, none failing. Commit statuses: success (PR Readiness and both CodeBuild contexts).
  • Docs: the docs/ci/ci-and-reviews.md bullet matches the shipped behaviour: once per streak, no close window, re-run, permissions.

Not verified

  • The workflow has never run on Actions. Two API shapes were checked against the GitHub REST docs and the PR's cited real runs only: the /runs/{id}/attempts/{n}/jobs response and workflow_runs[].id ordering against RUN_ID. Neither was observed live.
  • Nobody has run a live red macOS push end to end. The step prints Red streak: <first|continues|rerun> (:220); check that line on the first real red.
  • actionlint was not run locally.

@iamwhatever
iamwhatever disabled auto-merge October 16, 2026 01:07
@iamwhatever
iamwhatever merged commit 0fafcf9 into main Oct 16, 2026
104 of 113 checks passed
@iamwhatever
iamwhatever deleted the fix/macos-red-main-pager-15257 branch October 16, 2026 01:07
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: page someone when a macOS leg goes red on main

3 participants