Skip to content

ci: turn main's audit red when the ruleset stops requiring the merge queue - #17224

Merged
chenmingwei23 merged 1 commit into
mainfrom
ci/audit-2026-10-merge-queue-ruleset
Oct 6, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
ci/audit-2026-10-merge-queue-ruleset

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When main's ruleset stops requiring the merge queue, main's own audit turns red, so the gap is seen instead of found later through cross-merge breaks.

docs/ci/ci-and-reviews.md says "The real merge gate is human approval plus the merge queue". ci.yml says a merge_group run is what makes "every PR was green" imply "main is green". But main's live rules have no merge_queue entry today:

gh api repos/kirodotdev/KiroCrew/rules/branches/main --jq '[.[].type]'
["repository_visibility","repository_visibility","repository_delete","repository_transfer","deletion","non_fast_forward","required_linear_history","pull_request","required_status_checks"]

merge_group CI runs exist only on 2026-10-02..03 (about 6 hours). A ruleset change is not a diff, so no check noticed.

Why it matters

Without the queue, two PRs each green alone land minutes apart and are red together. Every open PR then inherits the red until a third PR repairs it. Main push reds of this kind (run/attempt/job): 35293993344/1/105448930500, 34111349592/1/101713969407, 34115067847/1/101723629780, 35634906432/1/106450545989, 34658128710/1/103455742964, 35381992437/1/105720581106, 35166695064/1/105032457582, 35794581983/1/106971521230, 37182491418/1/111377998664, 36827891125/1/110258870559, 35266675992/1/105364805985. Known pairs: #11438 over #11018, #15078 over #14987, #15917 over #16427.

Not a goal

  • Re-enabling the queue. That is a ruleset setting a maintainer changes (tick "Require merge queue" with the documented values, set MERGE_QUEUE_ENABLED=true). No diff can do it.
  • Changing ruleset bypass actors (RepositoryRole 2/5, team 12896464). That is an owner decision.
  • Moving the two macOS legs into the queue. They stay post-merge by design.
  • Wiring this job into the tracking-issue report job.
  • Editing AGENTS.md / AUTOSDE.yaml / docs/README.md. The matching AGENTS/AUTOSDE parts come in a later integration PR.

What changed (motivation → approach → change)

Symptom: main goes red from cross-merges. Root cause: the queue that would test the integrated tree is not required, and turning it off leaves no trace in CI. Change: scripts/check_merge_queue_ruleset.py reads one rules/branches/main response and exits 1 unless it holds a merge_queue rule backed by a required_status_checks rule. "Branch up to date" (strict_required_status_checks_policy: true) does not count as a substitute: merges here land seconds apart (84 s, 42 s, 2.6 min). When MERGE_QUEUE_ENABLED is true but the queue is missing, the message also says ci.yml skips main's push run, so nothing tests the merged tree.

A new scheduled workflow, merge-queue-ruleset.yml (Merge Queue Ruleset Audit, every 6 hours plus manual dispatch, upstream repo only), runs main's copy of the script. It is listed in scheduled-failure-watch.yml, so a red scheduled run opens one tracking issue. It is not a job in main-ratchet-audit.yml: that run status is the per-commit ratchet signal, and a setting-level red on every push would hide a real ratchet regression (Design Review). docs/ci/ci-and-reviews.md names the workflow.

Runtime / CI impact: this is a workflow change, not docs-only. One hosted job every 6 hours (one API call). Until a maintainer ticks the queue, every scheduled run is red and the watcher keeps one issue open. That red is the point. No PR lane changes.

This PR's green does not prove the new workflow works. It never runs on a pull request. After merge, confirm it:

  1. Run gh workflow run merge-queue-ruleset.yml --repo kirodotdev/KiroCrew. The job Merge queue required on main should be red with ::error::main has no merge_queue rule.
  2. After the next scheduled red run, an issue titled Merge Queue Ruleset Audit scheduled run is failing should exist (label scheduled-failure).
  3. After a maintainer ticks "Require merge queue", the next run should be green with merge queue required on main (ruleset 20088190).

Backwards compatibility

Compatible: it adds a script, a test, a scheduled workflow and one watched name. No PR lane, input or setting changes.

Removes nothing: it only adds a check, a test, a watch entry and one doc sentence.

Tests

test/test_check_merge_queue_ruleset.py, deterministic, no network:

  • the live rule shape without a queue fails;
  • the same shape with strict_required_status_checks_policy: true still fails;
  • a merge_queue rule plus the required check passes;
  • a merge_queue rule with no required_status_checks rule fails;
  • a non-list body (e.g. {"message": "Not Found"}) fails closed;
  • the CLI exits 1 / 0 / 2 for off / on / unreadable;
  • MERGE_QUEUE_ENABLED=true without the queue names the skipped push run (function and CLI);
  • the workflow triggers only on schedule and workflow_dispatch, is fenced to the upstream repo, passes the variable, and is in the failure watch's list.

Local runs (exit codes from the tool itself, logged to file):

  • pytest test/test_check_merge_queue_ruleset.py test/test_scheduled_failure_watch.py test/test_main_ratchet_audit.py test/test_workflow_checkout_credentials.py test/test_ci_fleet_routing_expression_parity.py test/test_ci_runner_watchdog.py test/test_workflow_secret_and_cache_scope.py test/test_ai_review_workflows.py test/test_node_version_pins.py test/test_docs_lint_fact_checks.py → 0 (1763 passed)
  • scripts/run_scoped_tests.py --surface backend → 0
  • black, flake8, isort, comment-history, brand-name, subprocess-encoding, docs-lint → 0

Manual verification

gh api repos/kirodotdev/KiroCrew/rules/branches/main > rules.json; python3 scripts/check_merge_queue_ruleset.py rules.json → exit 1 with the ::error:: line; with MERGE_QUEUE_ENABLED=true → exit 1 plus the skipped-push-run sentence (read 2026-10-05 ~22:30 PDT).

Unvalidated: the workflow on main and the watcher issue. They cannot run before merge; see the three steps above.

Related Issues

Closes #17223

Pattern harvest

Rule candidate: review-prompt
Pattern: "a CI guarantee that rests on a repository setting (ruleset, variable), not a file, needs a scheduled live read of that setting; otherwise turning it off is invisible."

Guard record (audit G-U3-01)

Field Value
Failure evidence runs listed under Why it matters
Scope ruleset protected-branches (20088190) on refs/heads/main; every PR
Forbidden pattern rules/branches/main has no type == merge_queue entry
Correct alternative tick "Require merge queue" (max_entries_to_build 40, max_entries_to_merge 1, check timeout ≥ 180 min) + MERGE_QUEUE_ENABLED=true; strict up-to-date is not a substitute
Exceptions ruleset bypass actors; macOS legs stay post-merge
Canonical owner docs/ci/ci-and-reviews.md (contract), the ruleset (setting), scripts/check_merge_queue_ruleset.py (drift check)
Consumers merge-queue-ruleset.yml (schedule/dispatch, runs main's own script); scheduled-failure-watch.yml (files the issue); merge-queue-readiness.yml (runs only once the queue is required)
Positive examples live rules today → 1; same with strict=true → 1; queue without required check → 1; variable true without queue → 1 with the extra sentence
Negative example rules with merge_queue + required_status_checks → 0
Validation unit tests pass; live read → 1; on-main run unvalidated until merge

Folds two duplicate audit proposals: G-U5-01 (same script; its "queue without a required check" case and its scheduled cadence are taken here) and G-U7-06 (doc-level statement; its three cross-merge pairs and runs are cited above). G-U5-01's "strict up-to-date as fallback" is not taken, for the reason in What changed. The audit patch's push-only job in main-ratchet-audit.yml is replaced by the scheduled workflow, per Design Review.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 6, 2026 04:03
@bolichen97
bolichen97 requested a review from CrysisDeu October 6, 2026 04:03
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of cc25784461c8ea750896994cd18d1b04e4663b56 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

After merge, dispatch merge-queue-ruleset.yml once and check that github.token with contents: read can read rules/branches/main. No PR run exercises that call.

What this change ships

Inventory (6 items) — 6 justified

Intent: make it visible when main's ruleset stops requiring the merge queue, so cross-merge reds are no longer found only after they happen. This is an ADDITION. It detects the gap; re-enabling the queue is a ruleset setting no diff can change, and the description says so under ## Not a goal.

  1. A scheduled workflow (every 6 hours, plus manual dispatch) turns red when main's live rules have no merge_queue entry — justified
  2. The same check also fails when a queue exists but no required_status_checks rule backs it — justified
  3. The red message gets an extra sentence when MERGE_QUEUE_ENABLED is true, saying main's push run is skipped too — justified
  4. A "Branch up to date" (strict) rule does not count as a substitute for the queue — justified
  5. A red scheduled run opens one tracking issue through scheduled-failure-watch.yml — justified
  6. docs/ci/ci-and-reviews.md gets one sentence naming the audit — justified

[FIRST-PRINCIPLES-REVIEWED] cc25784

Description read

[DESCRIPTION-READ] e981bea067666496f619889f08ae2ef15d24b9c6a0db75cb68bcf187d211ea37

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of cc25784461c8ea750896994cd18d1b04e4663b56 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, check that github.token with only contents: read can read rules/branches/main. If it can't, every run is red for an auth error, not the missing queue.

[DESIGN-REVIEWED] cc25784

Description read

[DESCRIPTION-READ] 871e85fe2edeeb92ae875aa4b155c513d443ae035ebfcddca7df4ae791483ac4

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed cc25784461c8ea750896994cd18d1b04e4663b56 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] cc25784

Verdict parsed from the review's SHA-scoped output markers for commit cc25784461c8ea750896994cd18d1b04e4663b56.

False positive or not applicable? A repository writer can comment:
/ai-review override fable cc25784461c8ea750896994cd18d1b04e4663b56: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of cc25784461c8ea750896994cd18d1b04e4663b56 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] cc25784

False positive or not applicable? A repository writer can comment:
/ai-review override gpt cc25784461c8ea750896994cd18d1b04e4663b56: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 6, 2026 04:42
@bolichen97
bolichen97 force-pushed the ci/audit-2026-10-merge-queue-ruleset branch from 7d98007 to 72594c9 Compare October 6, 2026 05:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 6, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed, not approving (crew-pr-reviewer). The code is fine: a push-only job with contents: read, persist-credentials: false, one gh api read, and a small, tested parser. But today GET repos/kirodotdev/KiroCrew/rules/branches/main returns no merge_queue rule (active types: repository_visibility, repository_delete, repository_transfer, deletion, non_fast_forward, required_linear_history, pull_request, required_status_checks). So the new Merge queue required on main job goes red on the first push after merge and stays red until someone re-enables the merge queue in the ruleset. Whether the repo should require the merge queue again is a repository-settings decision for a maintainer, so I am leaving this to one. If the plan is to re-enable the queue first, merge this after that change.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 6, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@bolichen97
bolichen97 force-pushed the ci/audit-2026-10-merge-queue-ruleset branch from 72594c9 to cc25784 Compare October 6, 2026 06:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 6, 2026
@bolichen97
bolichen97 disabled auto-merge October 6, 2026 07:10
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 6, 2026 07:40
@bolichen97
bolichen97 disabled auto-merge October 6, 2026 12:40
@chenmingwei23
chenmingwei23 merged commit 2eeb9c4 into main Oct 6, 2026
146 of 147 checks passed
@chenmingwei23
chenmingwei23 deleted the ci/audit-2026-10-merge-queue-ruleset branch October 6, 2026 14:39
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: main's ruleset no longer requires the merge queue, and nothing notices

3 participants