Skip to content

ci: tell agents how to fix a PR body that skips the template - #14412

Merged
bolichen97 merged 1 commit into
mainfrom
ci/pr-template-agent-hint
Sep 27, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
ci/pr-template-agent-hint

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When PR Hygiene fails a PR body that skips the template, the author (often a coding agent) is told exactly what to fix and how.

Agents open PRs with gh pr create --body-file, which never loads .github/PULL_REQUEST_TEMPLATE.md. PR Hygiene then fails, but its error annotation only says "4 required description sections are missing". The list of missing parts is buried in the job log, and the fix text talks about fork approval, which does not apply to a same-repo PR. Example: #14322 uses ## Summary / ## Cross-OS instead of the template headings.

Why it matters

A failure that names no fix makes the agent guess, so it pads the body or rewrites it by hand, and the check stays red for another round. Most PRs here are opened by agents, so the failure message is effectively their instructions.

Not a goal

What changed (motivation → approach → change)

Where Before After
PR Hygiene error annotation a count the missing parts plus the fix
Job output log only log and job summary page
Fix text fork approval note for every PR numbered fix steps; fork note only on the fork check
AGENTS.md no word on the PR body gh pr create bypasses the template: build the body from it and self-check

.github/scripts/pr-description-check.sh now writes an annotation output: one line listing everything missing (sections, the Goal line if absent) and the fix (rebuild from the template, keep headings verbatim, save the description, no push). Its summary gains "How to fix it" steps, including a local self-check command. A new PR_CALLER input keeps the fork-approval sentence on fork-pr-description.yml (the default) and drops it for code-review.yml, which passes PR_CALLER=pr-hygiene. The template link now tolerates a missing GITHUB_REPOSITORY, so the self-check runs outside Actions. docs/ci/ci-and-reviews.md describes the new failure output.

Backwards compatibility

Compatible: the pass/fail rules and the required sections are unchanged; only failure wording and one extra output key are added. The fork check keeps its existing summary text by default.

Tests

test/test_fork_pr_description_workflow.py:

  • the annotation names the missing section and the fix, and is empty on success;
  • a missing Goal line is named in the annotation, and a missing section plus Goal line are listed together (no "1 missing" miscount);
  • the printed self-check recipe writes to a fresh mktemp file, in the script output and in AGENTS.md;
  • only the fork caller mentions fork approval;
  • the script runs without GITHUB_REPOSITORY;
  • the PR Hygiene step passes PR_CALLER=pr-hygiene, emits the annotation, and writes the job summary.

Manual verification

Ran the script on #14322's live body with PR_CALLER=pr-hygiene: failure, annotation lists ## Problem / Motivation, ## Why it matters, ## Not a goal, ## What changed plus the fix, no fork sentence. Ran the AGENTS.md self-check recipe twice in a row (failing body, then this body): each run reads one fresh conclusion.

Screenshots / video

Why no screenshot: CI script and docs only; no dashboard change.

no linked issue: follow-up to template-compliance feedback on #14173 and #14322.

@iamwhatever
iamwhatever requested a review from a team as a code owner September 27, 2026 07:43
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Intent: When PR Hygiene fails a body that skips the template, tell the author (usually an agent) exactly what is missing and how to fix it, and tell agents in AGENTS.md that gh pr create bypasses the template.
Not a goal: changing the pass/fail rules, checking section order, or changing prepare-pr (#14401).

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 57c240b7404c7cd8d71ac161a6c20e0a6aa61885 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the ::error:: annotation renders legibly in the live checks tab — manual verification ran the script locally, not through Actions.

[DESIGN-REVIEWED] 57c240b

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 57c240b7404c7cd8d71ac161a6c20e0a6aa61885 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 57c240b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 57c240b7404c7cd8d71ac161a6c20e0a6aa61885: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 57c240b7404c7cd8d71ac161a6c20e0a6aa61885 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify on one real fork PR that the fork check-run summary — which also gains the numbered "How to fix it" steps — still renders within GitHub's check-run limits.

What this change ships

Inventory (7 items) — 7 justified

Intent: FIX — a failing PR-body check told the author a count instead of what to fix (examples #14173, #14322).

  1. The failure annotation now names each missing section and the fix, not a count — justified
  2. The full fix steps now also appear on the job summary page, not only the log — justified
  3. The failure text gains numbered "How to fix it" steps with a local self-check command — justified
  4. The fork-approval sentence now appears only on the fork check (PR_CALLER, default keeps fork behavior) — justified
  5. The self-check now runs outside Actions (template link tolerates missing GITHUB_REPOSITORY) — justified
  6. AGENTS.md now warns that gh pr create bypasses the template and gives the self-check — justified
  7. docs/ci/ci-and-reviews.md describes the new failure output (same-commit doc rule) — justified

Consumer counts run: annotation — 1 consumer (code-review.yml:458, grep annotation under .github/); PR_CALLER — 1 explicit setter plus the default fork lane (grep PR_CALLER, 2 non-test sites). No deleted pin: grep fork workflow runs shows no pre-existing assertion of the old unconditional sentence.

[FIRST-PRINCIPLES-REVIEWED] 57c240b

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 57c240b7404c7cd8d71ac161a6c20e0a6aa61885 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 57c240b

Verdict parsed from the review's SHA-scoped output markers for commit 57c240b7404c7cd8d71ac161a6c20e0a6aa61885.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 57c240b7404c7cd8d71ac161a6c20e0a6aa61885: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the ci/pr-template-agent-hint branch from 4d0bcbd to b0e2f59 Compare September 27, 2026 07:56
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: annotation count ignores the Goal line span=dd53eec12443

The annotation no longer reuses $title; it reads "Missing from the PR description: ", listing sections and the Goal line together.
Test: test_the_annotation_counts_a_section_and_the_goal_line_together. Fixed in b0e2f59.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: AGENTS.md self-check dies without GITHUB_OUTPUT span=ee3e5ff81392

AGENTS.md now gives the full recipe with PR_BODY and GITHUB_OUTPUT set, the same one the script prints.
Ran it twice locally: failure then success, one conclusion each. Fixed in b0e2f59.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: self-check recipe appends to a stale output file span=dd53eec12443

The recipe starts with : > /tmp/pr-check.out, so each run reads only its own verdict.
Test: test_the_printed_self_check_recipe_truncates_its_output_file. Fixed in b0e2f59.

@iamwhatever
iamwhatever force-pushed the ci/pr-template-agent-hint branch from b0e2f59 to f46af48 Compare September 27, 2026 08:13
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/ci/ci-and-reviews.md since this head went green (6978ac0 -> a83d225); CI re-running. Same push formats test/test_fork_pr_description_workflow.py with black (Backend Lint was red on it).

@iamwhatever
iamwhatever force-pushed the ci/pr-template-agent-hint branch from f46af48 to 1a57392 Compare September 27, 2026 08:26
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
PR Hygiene's error annotation now names the missing sections and the fix, the full steps go to the job summary, the fork-only approval note stays on the fork check, and AGENTS.md says gh pr create bypasses the GitHub template.
@iamwhatever
iamwhatever force-pushed the ci/pr-template-agent-hint branch from 1a57392 to 57c240b Compare September 27, 2026 09:18
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main 0864035 (it touched docs/ci/ci-and-reviews.md, same file as this PR). No conflicts; 46/46 tests in test/test_fork_pr_description_workflow.py pass. New head 57c240b. GPT, Design and First Principles passed on the previous head with no findings.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97
bolichen97 merged commit af86ede into main Sep 27, 2026
97 checks passed
@bolichen97
bolichen97 deleted the ci/pr-template-agent-hint branch September 27, 2026 16:42
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants