ci: add a guidelines bot that learns from code review - #25628
Open
mshabarov wants to merge 3 commits into
Open
Conversation
Reviewers here spend part of every review repeating themselves, and some of that repetition exists only because a rule everyone holds in their head was never written down. This adds an agentic workflow that finds that second kind once a week and proposes it as a change to CONVENTIONS.md or a guidelines chapter. It reads the review comments on every pull request merged into main in the past week, keeps the ones that state a rule rather than report a defect, and verifies that each point actually landed — the merged code obeys it, the hunk it was anchored to was rewritten afterwards, a commit followed it, or the reviewer approved after it. An author's "done" is never enough on its own. A rule then has to survive a counter-example check: grep has to show that main mostly follows it already. What reaches a pull request is gated on evidence rather than on the model's enthusiasm. A rule qualifies when it recurs across two pull requests or two reviewers, or when repeating the mistake once more would be expensive — a correctness or thread-safety trap, a break in public API compatibility, a security consequence, a broken build or release. Three rules per pull request at most, one pull request per run, and one open at a time. The bot's own pull requests are its memory: closing one without merging tells it never to propose those rules again. Backports are out of scope, so that one review is not counted twice, and only dependency bumps are filtered by title. Measured over three weeks of merges, refactor: and ci: pull requests draw more review comments per pull request than fix: does, so the commit type says nothing about whether the review taught anything. The workflow only ever edits CONVENTIONS.md and the chapters under guidelines/ — never CLAUDE.md, never guidelines/overview.md, never code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mshabarov
marked this pull request as ready for review
September 10, 2026 09:31
Contributor
|
platosha
self-requested a review
September 11, 2026 14:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Reviewers here spend part of every review repeating themselves, and some of that repetition exists only because a rule everyone holds in their head was never written down. This adds an agentic workflow that looks for that second kind once a week and proposes it as a change to
CONVENTIONS.mdor aguidelines/chapter.CONVENTIONS.mdalready contains rules that arrived this way by hand — #25529 (static imports in tests) is the archetype. The bot automates the harvesting, not the decision: every run ends either in one draft pull request or in nothing at all.How it decides
Material. Every pull request merged into
mainin the past week — about seventy, of which roughly half carry an inline comment. Release branches are skipped so a backport is not counted as a second occurrence. Only dependency bumps are filtered by title: measured over three weeks of merges,refactor:(2.3 comments per PR) andci:(2.0) draw more review thanfix:(1.3), so the commit type says nothing about whether the review taught anything.Evidence that a point landed — at least one signal the agent checked itself:
mainobeys the point (sufficient on its own);An author's "done" is never enough. Pushback that stands, deferrals to a ticket, and unresolved disagreements between reviewers are dropped.
Counter-example check. Before a cluster becomes a rule,
grephas to show thatmainmostly follows it already. If violations dominate, it is one reviewer's preference or an unfinished migration, not a convention.The gate. Generalizable, landed, matching the codebase, not already covered, not proposed and rejected before — and then either recurring across two pull requests or two reviewers, or load-bearing on its own (correctness, thread safety, public API compatibility, security, a broken build or release).
Volume control
Three rules per pull request at most, one pull request per run, one open at a time. The bot's own pull requests are its memory: closing one without merging tells it never to propose those rules again. Proposals expire after 14 days. The pull request body lists what was weighed and dropped, which is where the behaviour gets tuned.
It only ever edits
CONVENTIONS.mdand the chapters underguidelines/— neverCLAUDE.md, neverguidelines/overview.md, never code.What changed
.github/workflows/guidelines-bot.md.github/workflows/guidelines-bot.lock.ymlgh aw compile..github/workflows/README.mdSecurity review
ANTHROPIC_API_KEY, the standard key for theclaudeengine. Already used bydiagram-bot.lock.ymlandclaude.yml, and excluded from the agent sandbox bygh-awin the same way.gh aw compileleft.github/aw/actions-lock.jsonuntouched, so every action this workflow pins was already pinned by an existing one. Compiled withgh awv0.80.4, matching the other lock files, so the diff is this workflow alone.contents,pull-requests,issues: read). Writing is done by thesafe_outputsjob, whose only enabled outputs arecreate-pull-request(max 1, draft, basemain) andnoop.gh-aw's own XPIA prompt.scheduleandworkflow_dispatchonly — no path from a pull request or a comment to a run.Trying it
Actions → Guidelines Bot → Run workflow, optionally with
lookback-days: 30for a first look at a wider window. A run that finds nothing records the reason in the log and touches nothing.🤖 Generated with Claude Code