Repository navigation
Conversation
… kill them "P-58 keeps the first head's status" and "the impostor workflow posts from the Lens's own file" each named a test that passes with the mutant in place, so their legs ran the whole suite after it: 10 to 21 minutes of a 27-minute step, the slowest leg of five of the last six merge-group runs. Each now names the one test the whole suite fails on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… by the matrix On 2026-11-09 merge group pr-657-a81e6e58 was cut at the checks step's 26 minutes, and the ten runs before took 19 to 23.4. checks becomes a matrix of three legs, as the mutants are: each leg runs every check and every third test of the whole suite, told CHECKS_SHARD and CHECKS_SHARDS by strategy.job-index and job-total, never by a number written in, so across the legs every test runs once, the 300 seeds included. verify still needs checks, whose result is every leg's. tools/check_workflows.py holds the legs' job as it holds the shards' (_legs); tools/verify.py refuses a leg it cannot read before any check runs, and runs every test where neither is set, as at a desk. Each new rule has its mutant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll, never a run of the whole suite The mutant run holds each mutant to the tests it names, alone: when they pass on it, it is an ERROR that names them, where before the whole suite ran after them, 10 to 21 minutes of a 27-minute leg, and hid the wrong name. An edit tools/untested.py tries, through attempt without alone, still runs the whole suite after its tests. The verify workflow's comment, its example and the standard's copy say so; the guard has its test and four mutants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
 |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…y named test stays SURVIVED The Claude review of #666 found that the impostor mutant's new killer failed on it only at the mutant run's 40 seeds, not at the 300 make verify runs. The test the mutant named before now pins what the mutant breaks: the Lens's own App never posts from the Lens's own file, checked after every step, so it fails on the mutant at 5, 40 and 300 seeds; the mutant names it again. And a named test that failed once and then passed was told as named_test_does_not_kill: it is SURVIVED again, as before the guard, with a mutant for the difference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he legs gave one test_a_key_and_its_environment_are_named_by_their_one_file wrote an env: into the first job with contents: read, which was checks; checks now has an env: of its own, so the key is put in it, not in a second. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ant follows its rename On this pull request's run the guard told "exactly on the bar does not clear it" as named_test_does_not_kill: the gate test it named passes on it, and test_each_kind_is_held_to_its_own_bar_where_the_cases_are, which holds the bar exactly, fails on it. And "an unconfirmed failure is FLAKY with no second run" mutates the loop the review's fix renamed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On 2026-11-09 merge group pr-657-a81e6e58 was cut at the
checksstep's 26 minutes, and the queue ejected #657, a diff that only touched fixtures. In the same runmutants (7)finished at 26.2 of its 27. This PR gives both legs at least 40% headroom on merge-group runs. It skips and weakens nothing: every test, every seed and every mutant still runs in every merge group, and the required check is stillverify.needs_passedis unchanged.Protected edits (
.github/workflows/verify.yml,tools/check_workflows.py): pre-approved by the Maintainer 2026-11-08, ruled by the Factory Builder 2026-11-09. Design: option (a) with three legs, plus the mutant fix and its guard. No decision record, following #180's precedent; no rule of AGENTS.md's table changes.What was slow (17 runs to 2026-11-09)
checks, one jobmutantslegThe two mutants, found from the gaps in the job logs, with their killers found by running the whole suite on each:
ReachScript.test_a_head_ends_where_the_next_line_is_no_headfails on it, in 1 s.Simulation.test_the_seeds_serve_items_with_an_acceptance_contract_and_withoutfails on it, in 5 s.The change
make verifyin three legs.checksbecomes a matrix,CHECKS_SHARD/CHECKS_SHARDSfromstrategy.job-index/job-total, as the mutant shards are set. Every leg runs every check. Its tests are every third test method of the whole suite, so across the legs every test runs once, all 300 seeds included. With neither variable set,make verifystill runs everything, as at a desk. An unreadable leg (one variable without the other, not a number, or out of range) stops the run before anything else.Held by
test_the_legs_of_make_verify_run_every_test_once(the shares partition the suite for 1, 2, 3 and 5 legs; the front door) andcheck_workflows._legs/test_the_legs_of_make_verify_are_told_their_place_and_count_by_the_matrix: the index and count come from the matrix and are never written in; the job's shape is closed, as the shards' is. The example and the standard's copy are changed to match.guarded→attempt(..., alone=True)) now runs a mutant's named tests alone. If they pass on it, the mutant is anERROR,named_test_does_not_kill: … passed on it; name the test that fails on it, never a run of the whole suite.tools/untested.pystill callsattemptwithoutalone, so its edits still run the whole suite after their tests. Held bytest_a_mutant_its_named_tests_miss_is_an_error_and_never_runs_the_whole_suite.Each new rule has its mutant: 7 for
_legs, 5 for the legs intools/verify.py, and 4 for the guard. Two existing targets moved with the line they mutate. A filtered run of the 21 new and moved mutants killed all of them.tools/AGENTS.mdlost a line of prose, so that the guard's row fits its 40 lines.Runner-minutes per run
checksleg took 9.1 of 26 (65% headroom) and the slowestmutantsleg 14.4 of 27 (47%). The first runs dealt the legs unevenly: tests took 309, 355 and 675 s at 5524e9d, because round-robin put the slow simulator tests on two legs. The slowest leg there was still 11.9 of 26 (54%).Verification
make verifyon an export of 35673e9: 1,719 tests, OK (8 expected failures), exit 0.python3 -m tools.mutants: CI's twelve shards, all green at 35673e9. On the first run, the guard caught a third misnamed mutant. "exactly on the bar does not clear it" now namestest_each_kind_is_held_to_its_own_bar_where_the_cases_are, which holds the bar exactly.Reviews
tests/test_simulator.py, a protected path, and only strengthens it.named_test_does_not_kill. It is SURVIVED again, as before the guard, with a mutant for the difference.checks' ownenv:, since the legs gave it one. 35673e9 has the guard's catch above and a mutant target moved with the renamed loop.After the merge, three consecutive merge-group runs will be reported, each with the slowest
checksleg and the slowestmutantsleg against their limits.🤖 Generated with Claude Code