Skip to content

ci: add a guidelines bot that learns from code review - #25628

Open
mshabarov wants to merge 3 commits into
mainfrom
update-guidelines-on-comment
Open

ci: add a guidelines bot that learns from code review#25628
mshabarov wants to merge 3 commits into
mainfrom
update-guidelines-on-comment

Conversation

@mshabarov

Copy link
Copy Markdown
Contributor

Summary

Reviewers here spend part of every review repeating themselves, and some of that repetition exists only because a rule everyone holds in their head was never written down. This adds an agentic workflow that looks for that second kind once a week and proposes it as a change to CONVENTIONS.md or a guidelines/ chapter.

CONVENTIONS.md already contains rules that arrived this way by hand — #25529 (static imports in tests) is the archetype. The bot automates the harvesting, not the decision: every run ends either in one draft pull request or in nothing at all.

How it decides

Material. Every pull request merged into main in the past week — about seventy, of which roughly half carry an inline comment. Release branches are skipped so a backport is not counted as a second occurrence. Only dependency bumps are filtered by title: measured over three weeks of merges, refactor: (2.3 comments per PR) and ci: (2.0) draw more review than fix: (1.3), so the commit type says nothing about whether the review taught anything.

Evidence that a point landed — at least one signal the agent checked itself:

  1. the merged code on main obeys the point (sufficient on its own);
  2. the comment is outdated, so the hunk it was anchored to was rewritten after it;
  3. a commit touching that file followed the comment;
  4. the reviewer approved after it.

An author's "done" is never enough. Pushback that stands, deferrals to a ticket, and unresolved disagreements between reviewers are dropped.

Counter-example check. Before a cluster becomes a rule, grep has to show that main mostly follows it already. If violations dominate, it is one reviewer's preference or an unfinished migration, not a convention.

The gate. Generalizable, landed, matching the codebase, not already covered, not proposed and rejected before — and then either recurring across two pull requests or two reviewers, or load-bearing on its own (correctness, thread safety, public API compatibility, security, a broken build or release).

Volume control

Three rules per pull request at most, one pull request per run, one open at a time. The bot's own pull requests are its memory: closing one without merging tells it never to propose those rules again. Proposals expire after 14 days. The pull request body lists what was weighed and dropped, which is where the behaviour gets tuned.

It only ever edits CONVENTIONS.md and the chapters under guidelines/ — never CLAUDE.md, never guidelines/overview.md, never code.

What changed

File
.github/workflows/guidelines-bot.md The agentic workflow. Edit this one.
.github/workflows/guidelines-bot.lock.yml Generated by gh aw compile.
.github/workflows/README.md Table row and a section describing the bot.

Security review

  • New secret: ANTHROPIC_API_KEY, the standard key for the claude engine. Already used by diagram-bot.lock.yml and claude.yml, and excluded from the agent sandbox by gh-aw in the same way.
  • New actions: none. gh aw compile left .github/aw/actions-lock.json untouched, so every action this workflow pins was already pinned by an existing one. Compiled with gh aw v0.80.4, matching the other lock files, so the diff is this workflow alone.
  • Permissions: the agent job is read-only (contents, pull-requests, issues: read). Writing is done by the safe_outputs job, whose only enabled outputs are create-pull-request (max 1, draft, base main) and noop.
  • Untrusted input: the workflow ingests arbitrary review comment bodies. The prompt states that comment text is data and never instruction, on top of gh-aw's own XPIA prompt.
  • Triggers: schedule and workflow_dispatch only — no path from a pull request or a comment to a run.

Trying it

Actions → Guidelines Bot → Run workflow, optionally with lookback-days: 30 for a first look at a wider window. A run that finds nothing records the reason in the log and touches nothing.


🤖 Generated with Claude Code

Reviewers here spend part of every review repeating themselves, and some of
that repetition exists only because a rule everyone holds in their head was
never written down. This adds an agentic workflow that finds that second
kind once a week and proposes it as a change to CONVENTIONS.md or a
guidelines chapter.

It reads the review comments on every pull request merged into main in the
past week, keeps the ones that state a rule rather than report a defect, and
verifies that each point actually landed — the merged code obeys it, the
hunk it was anchored to was rewritten afterwards, a commit followed it, or
the reviewer approved after it. An author's "done" is never enough on its
own. A rule then has to survive a counter-example check: grep has to show
that main mostly follows it already.

What reaches a pull request is gated on evidence rather than on the model's
enthusiasm. A rule qualifies when it recurs across two pull requests or two
reviewers, or when repeating the mistake once more would be expensive — a
correctness or thread-safety trap, a break in public API compatibility, a
security consequence, a broken build or release. Three rules per pull
request at most, one pull request per run, and one open at a time. The
bot's own pull requests are its memory: closing one without merging tells it
never to propose those rules again.

Backports are out of scope, so that one review is not counted twice, and
only dependency bumps are filtered by title. Measured over three weeks of
merges, refactor: and ci: pull requests draw more review comments per pull
request than fix: does, so the commit type says nothing about whether the
review taught anything.

The workflow only ever edits CONVENTIONS.md and the chapters under
guidelines/ — never CLAUDE.md, never guidelines/overview.md, never code.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mshabarov
mshabarov marked this pull request as ready for review September 10, 2026 09:31
@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Test Results

 1 440 files  ±0   1 524 suites  ±0   1h 38m 2s ⏱️ +14s
12 039 tests ±0  11 971 ✅ ±0  68 💤 ±0  0 ❌ ±0 
12 357 runs  ±0  12 289 ✅ ±0  68 💤 ±0  0 ❌ ±0 

Results for commit 99a6910. ± Comparison against base commit edd1445.

♻️ This comment has been updated with latest results.

@sonarqubecloud

Copy link
Copy Markdown

@platosha
platosha self-requested a review September 11, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant