Explain selections: whole-suite rules, a why line, fuller console output - #8
Merged
Merged
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
leanest: 8 of 8 playwright test files selectedAll 8 run because the runner setup changed (.github/workflows/ci.yml). 8 test files
|
baronunread
force-pushed
the
selection-explained
branch
from
September 27, 2026 21:55
53969dd to
18e3073
Compare
Feedback from leanest's first real run (rdyrct #263: 37/37 selected, with no way to tell why from the log). - Whole-suite rules, decided before the judge: a change to runner config (playwright/vitest/vite config, package.json, lockfiles, CI workflows) runs everything; a Markdown-only change skips everything. No judge call, so matrix shards can't disagree on these. - "Why they run: N by rule, N judge unsure (c < 0.5), N judged affected." in the console and the PR comment. - Console: each RUN line carries its reason, and the RUN and Changed lists say "... and N more" past 20. - inspect during a judge outage selects every test and says why, instead of showing nothing selected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"37 by rule" read like a log line. Whole-suite rules now say their
reason once ("All 37 run because the runner setup changed (…)",
"Skipping all 37 tests: only Markdown changed.") instead of on every
test, and mixed runs get a sentence: "2 touch the change directly, the
judge wasn't sure enough to skip 9 and it thinks 1 is affected."
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every row repeats the reason the sentence above already gives, so the 37 rows go into a collapsed section like the skipped ones. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Markdown-only comment showed "0 of 37" and a collapsed table with no sentence; it now reads "Nothing runs because only Markdown changed." Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From a cold review of this PR: - "Only Markdown changed" skipped every test, overriding the rules that force a test to run: a test importing the changed .md file was skipped. Those tests now run with their own reason; the rest skip. - inspect never applied the whole-suite rules, so it could disagree with select and the real run. It now does, and says so. - Under --shadow/--full the comment said "Nothing runs because…" and then "The full suite runs anyway". The sentence is left out there. Adds an end-to-end test on a real git repo, which fails without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
baronunread
force-pushed
the
selection-explained
branch
from
September 27, 2026 21:57
18e3073 to
7754dd1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Acts on the feedback from leanest's first real run (rdyrct #263: 37/37 selected, with no way to tell why from the log).
package.json, lockfiles,.github/workflows/) runs everything. A Markdown-only change skips everything. No judge call, so matrix shards can't disagree on these.All 37 run because the runner setup changed (.github/workflows/test.yml).or2 touch the change directly, the judge wasn't sure enough to skip 9 and it thinks 1 is affected.... and N morepast 20.inspectduring a judge outage selects every test and gives the reason, instead of showing nothing selected.Checked against a local clone of rdyrct:
All 37 run because the runner setup changed (.github/workflows/test.yml)..Skipping all 37 tests: only Markdown changed..inspectwith an unknown provider:⚠ Judge unavailable (…), all 37 tests would run.Not in this PR: keeping shards consistent when the judge does decide (select once, then shards consume the list). That changes how the Action is wired up, so it'll be a separate change.
🤖 Generated with Claude Code