Skip to content

test: split test_ai_review_workflows.py, part 3 of 3 - #20654

Merged
chenmingwei23 merged 1 commit into
mainfrom
refactor/linecap-t02-test-ai-review-workflows-p3
Oct 15, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
refactor/linecap-t02-test-ai-review-workflows-p3

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Bring test/test_ai_review_workflows.py under the 2,000-line cap on hand-written files without changing a single test, in a three-PR series; this PR is part 3.

test/test_ai_review_workflows.py is still 7,367 lines after parts 1 and 2 (#20282, #20347). It is past the 2,000-line cap on hand-written files, and nearly every review-workflow change collides in it.

Why it matters

Every change to a review lane edits this one file, so unrelated PRs conflict in it and a reader has to search thousands of lines for the pin that guards a workflow. Splitting by the behaviour under test gives each lane's pins a findable home.

Not a goal

  • Changing, renaming, rewriting or de-flaking any test. Bodies move verbatim.
  • Changing any workflow, script or product behaviour. The workflow edits are comments.

What changed (motivation → approach → change)

Like parts 1 and 2, this part moves ONE contiguous run of the original, so git's default diff equals git diff --minimal byte for byte (679,041 bytes; per file, git diff --numstat and --minimal --numstat agree, the original at +1 / -6,374). It moves everything between the module helpers and TestPrReadiness, and the run from TestFirstPrinciplesShellSyntax through TestProtectedCheckNameHasOnePublisherPerPrType: 30 classes, 428 tests. The original ends at 1,247 lines and keeps one contiguous run: TestPrReadiness (which test/test_review_lane_prompt_parity.py reads by path), TestDesignReviewPresentation, and TestFirstPrinciplesReview, which holds an unsplit PEM private-key header as a case literal that the content scan has no merged precedent for, so it is not moved.

File Lines What it holds
test/test_ai_review_workflows_ux.py 53 -> 1,374 + the UX scope gate, ...Pipeline (the blind pass) and ...DescriptionEvidence
test/test_ai_review_workflows_ux_committed_evidence.py (new) 824 ...CommittedEvidence
test/test_ai_review_workflows_opus.py (new) 1,413 the Opus lane, its spent turn budget, and the prepare-pr local review
test/test_ai_review_workflows_fork_stage2.py (new) 970 fork Stage-2 trigger, identity, check runs, check names
test/test_ai_review_workflows_human_override.py 300 -> 1,014 + the override handler and the fail-closed comment reads
test/test_ai_review_workflows_gpt.py 496 -> 823 + PR-intent grounding, the media filter, the falsification pass, framing parity
test/test_ai_review_workflows_first_principles.py 270 -> 668 + shell syntax, the scope gate, the PR-intent read and cap, the fork contract state
test/test_ai_review_workflows_whole_design_verdicts.py 821 -> 966 + the observed cause of an incomplete review, the current-head marker
test/ai_review_workflows_helpers.py 673 -> 1,074 + UxEvidenceGateHarness and the helpers the new parts share with the original

TestUxReviewReadsTheScreenshotsBlindFirst is 2,298 lines, over the 1,600-line cap for one part, so it is cut into three topic classes: ...Pipeline (the blind pass and its steps) and ...DescriptionEvidence in ux, and ...CommittedEvidence in ux_committed_evidence. The evidence-gate harness both evidence classes drive (fixture bytes, the gh/jq/curl stubs, the git fixture repository and the step runner) moves once into test/ai_review_workflows_helpers.py as UxEvidenceGateHarness: no Test prefix and no test method, so pytest collects nothing from it, and both evidence classes inherit it. The three-line _steps accessor is copied into the two topic classes that call it.

The workflow, doc and test comments that cite the moved pins name their new files.

Backwards compatibility

Compatible: only test modules move; no product import path, entry point or API changes, and every test keeps its name and parameters.

Removes nothing: the 428 moved tests still run, from sibling files in the same directory, and nothing imports from test/test_ai_review_workflows.py.

Move audit

test/test_ai_review_workflows.py: 7620 -> 1247 lines; moved 99.99% (python-ast)
   changed: class TestUxReviewReadsTheScreenshotsBlindFirst (header) [class-header] 1 lines

Tests

No test is added or changed. The proof that nothing was lost:

base this PR
collected tests 1,501 1,501
(test, params) multiset — identical
(class, test, params) multiset — identical once the three UX topic classes below map back to TestUxReviewReadsTheScreenshotsBlindFirst

Class mapping (57 collected tests, by method; [...] is every parametrization):

Old (TestUxReviewReadsTheScreenshotsBlindFirst::) New class Collected
test_the_fork_lane_reads_the_media_a_no_write_access_contributor_commits TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_committed_symlink_is_refused_by_its_mode_before_its_bytes_are_read TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_rejected_committed_media_still_consumes_the_blob_read_budget TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_description_attachment_is_never_displaced_by_a_committed_file TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_the_committed_script_runs_under_the_bash_3_of_macos TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_glob_metacharacter_in_a_committed_name_names_that_one_entry TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_committed_name_with_a_control_character_reaches_no_sink TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_committed_format_the_reviewer_cannot_read_is_named_not_dropped TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_refused_committed_format_consumes_no_read_budget TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_screenshot_that_was_only_moved_is_named_as_the_base_not_admitted TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_a_failed_enumeration_fails_the_step_instead_of_reading_as_no_media TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_both_fork_lanes_read_the_committed_media_from_the_object_store TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_the_same_repo_design_lane_sources_the_shared_script_and_keeps_no_loop_of_its_own TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_both_design_lanes_admit_a_renamed_screenshot_and_refuse_by_mode_and_bytes TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 2
test_a_design_lane_attachment_is_counted_apart_from_the_committed_media TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 2
test_neither_fork_lane_tells_its_reviewer_to_discount_a_committed_image TestUxReviewReadsTheScreenshotsBlindFirstCommittedEvidence 1
test_the_blind_pass_can_see_only_the_screenshots TestUxReviewReadsTheScreenshotsBlindFirstPipeline 1
test_the_blind_pass_runs_before_the_review_and_feeds_it_a_data_file TestUxReviewReadsTheScreenshotsBlindFirstPipeline 1
test_each_ux_model_call_has_its_own_credential_assume TestUxReviewReadsTheScreenshotsBlindFirstPipeline 1
test_a_failed_blind_read_degrades_to_an_evidence_gap_not_a_red_lane TestUxReviewReadsTheScreenshotsBlindFirstPipeline 1
test_review_prompts_carry_no_expression_so_the_21000_cap_cannot_bite TestUxReviewReadsTheScreenshotsBlindFirstPipeline 2
test_the_five_second_proxy_is_replaced_not_duplicated TestUxReviewReadsTheScreenshotsBlindFirstPipeline 2
test_the_fork_lane_says_it_has_no_blind_read_rather_than_faking_one TestUxReviewReadsTheScreenshotsBlindFirstPipeline 1
test_each_case_gets_its_own_job_temp_not_the_shard_s TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_the_evidence_step_copies_regular_images_under_opaque_names TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_the_evidence_step_reports_no_screens_for_a_code_only_ui_change TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_every_download_failing_fails_the_same_repo_evidence_step TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_the_evidence_step_reports_unfetched_false_when_nothing_was_offered TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_a_recording_only_description_is_missing_evidence_not_a_fetch_failure TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_a_404_attachment_is_the_authors_url_not_a_fetch_failure TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_a_partial_fetch_fails_the_step_like_a_total_one TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_all_four_evidence_lanes_admit_the_same_two_evidence_classes TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_a_recording_that_downloaded_beside_a_failed_still_fails_the_step TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_the_fork_lane_fails_an_all_failed_fetch_and_reddens_its_check_run TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_the_evidence_step_downloads_attachments_from_the_pr_description TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_a_transient_api_failure_is_retried_and_the_evidence_still_collected TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_three_api_failures_fail_the_step_closed_not_as_no_evidence TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_every_download_skipped_is_an_error_annotation_not_a_quiet_gap TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_description_recordings_are_capped_and_never_count_as_screens TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_description_images_share_the_committed_cap_and_downloads_are_bounded TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 2
test_the_pr_description_reaches_the_evidence_step_only_as_grep_input TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_the_fork_lane_fetches_attachments_from_the_api_body_with_the_same_guards TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1
test_both_lanes_name_the_description_as_the_normal_home_for_evidence TestUxReviewReadsTheScreenshotsBlindFirstDescriptionEvidence 1

The harness is not collected on its own: no node id names UxEvidenceGateHarness.

  • Floor on this head (every test_ai_review_workflows* file, the tests that read an edited workflow, doc or test, the tests that scan every workflow, and the doc/ratchet readers; 75 files): 4,899 passed, 10 skipped.
  • test/test_flake_pattern_ratchet.py, test/test_wait_discipline_ratchet.py, comment-history and brand gates, black, isort, flake8, docs_lint.py, semgrep 1.78 with CI's configs (0 findings): all pass.

Determinism: the moved tests are unchanged, so their determinism is what it was on main.

CI path references

Comment lines only, no logic: "test_ai_review_workflows.py asserts the guard is on every step" -> test_ai_review_workflows_fork_stage2.py (where TestProtectedCheckNameHasOnePublisherPerPrType now lives), in claude-review.yml, codex-review.yml, design-review.yml, first-principles-review.yml and ux-review.yml.

No workflow path filter or file list names the target.

Citations updated

Eight comment and docstring citations (L29), each naming the part that now holds the pin it cites; per file, git diff --numstat and --minimal --numstat agree:

File +/- What it now names
docs/ci/ci-and-reviews.md (2 edits) +2 / -2 TestOpusDiscoveryCaptureExecutes -> test_ai_review_workflows_opus.py; the override-handler pins -> test/test_ai_review_workflows_human_override.py
docs/system-specs/common/testing-conventions.md +2 / -2 the UX evidence-step cases -> the TestUxReviewReadsTheScreenshotsBlindFirst* classes in test_ai_review_workflows_ux*.py
test/test_pr_quality_gates.py (two docstrings, one comment) +3 / -3 the evidence-script pins -> test_ai_review_workflows_ux_committed_evidence.py and test_ai_review_workflows_ux*.py
test/test_fork_stage2_checkout.py (one comment) +2 / -2 the base-ref literal pin -> test_ai_review_workflows*.py; "pins" becomes "pin" to agree with the plural subject
test/test_issue_triage_workflow.py (one comment) +1 / -1 the nt guard -> "the test_ai_review_workflows_*.py parts"

The five workflow comment lines are listed under ## CI path references above (+1 / -1 each).

Line-cap baseline

Deletes this target's row (7367 test/test_ai_review_workflows.py) from .github/file-line-cap-baseline.txt: the file ends at 1,247 lines. python3 scripts/check_file_line_cap.py: 0 failures.

Upstream changes carried

Three PRs edited the original after this PR's first push. The recipe was re-run on main 213339f671, so their text moves verbatim; the move audit and the collected multiset above are measured on that base.

Upstream What it changed Where it now lives
#20583 MERGE_BASE asserts in TestFirstPrinciplesReview; the UX evidence harness env; TestClaudeReviewCodeOnlyScope TestFirstPrinciplesReview stays in the original; the env in UxEvidenceGateHarness (helper module); TestClaudeReviewCodeOnlyScope in opus
#20619, #20702 a new test in TestOpusTwoStageArchitecture, the new TestOpusTurnBudgetOutcome and _max_turns_transcript all in opus, beside the class they extend

Notes for reviewers

  • The move audit lists one changed unit, the TestUxReviewReadsTheScreenshotsBlindFirst class header: the class is cut into topic classes. Every member is AST-identical in its new class; the original class docstring stays on ...Pipeline.
  • Every part's docstring now describes its full contents.

Manual verification

N/A — test-only move; the collected multiset, the move audit and the passing runs above are the verification.

Related Issues

Part of #20192

Follow-up: #20383 (pre-existing shapes moved verbatim; de-flake there)

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 15, 2026 06:29
@bolichen97
bolichen97 requested a review from CrysisDeu October 15, 2026 06:29
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 5861761c041423e7c7af057fb17ed7df77c046e1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5861761c041423e7c7af057fb17ed7df77c046e1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that test_ai_review_workflows.py (now 1,247 lines) still holds the pin .github/workflows/pr-readiness.yml:58 points at.

[DESIGN-REVIEWED] 5861761

Description read

[DESCRIPTION-READ] 8da986b46474cc42c3e34c99fefa8a14162cf083a308df9109243683abfa30da

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 5861761c041423e7c7af057fb17ed7df77c046e1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that CI collects the same 1,501 tests on this head. The description claims parity, but the claim is all it gives.

What this change ships

Inventory (6 items) — 6 justified

Intent: bring test/test_ai_review_workflows.py under the 2,000-line cap on hand-written files without changing any test, as part 3 of 3. This is a FIX for a documented invariant (AGENTS.md: "No hand-written file over 2,000 lines").

  1. 428 tests in 30 classes move verbatim into sibling test_ai_review_workflows_*.py files, three of them new — justified
  2. The original file shrinks to 1,247 lines and its row leaves .github/file-line-cap-baseline.txt — justified
  3. TestUxReviewReadsTheScreenshotsBlindFirst (2,298 lines) is split into three topic classes: Pipeline, DescriptionEvidence and CommittedEvidence — justified
  4. The evidence-gate fixtures move once into UxEvidenceGateHarness in ai_review_workflows_helpers.py, which the two evidence classes inherit — justified
  5. The _steps accessor is copied into the two topic classes that call it — justified
  6. Comments in the five review workflows, two docs and three tests now name the files the moved pins live in — justified

[FIRST-PRINCIPLES-REVIEWED] 5861761

Description read

[DESCRIPTION-READ] c3d833ab8de4a9aca9919fe39b69d4fcc7c7ab82e1d70ab34c796cb83a901d81

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5861761c041423e7c7af057fb17ed7df77c046e1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5861761

Verdict parsed from the review's SHA-scoped output markers for commit 5861761c041423e7c7af057fb17ed7df77c046e1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5861761c041423e7c7af057fb17ed7df77c046e1: <one-sentence reason>

@bolichen97
bolichen97 enabled auto-merge (squash) October 15, 2026 06:35
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 5861761c041423e7c7af057fb17ed7df77c046e1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5861761

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5861761c041423e7c7af057fb17ed7df77c046e1: <one-sentence reason>

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 15, 2026
The last part of the split moves everything between the module
helpers and TestPrReadiness, and the run from the First Principles
shell-syntax tests through the protected check-name tests (30
classes, 428 tests). Three new sibling parts take them, beside five
parts that parts 1 and 2 started. No file is now over 1,600 lines.

The original keeps TestPrReadiness, which another test reads by path,
and the two classes beside it: TestFirstPrinciplesReview holds an
unsplit PEM header literal the content scan has no precedent for, and
TestDesignReviewPresentation keeps the kept run contiguous.

TestUxReviewReadsTheScreenshotsBlindFirst is 2,298 lines, so it is
cut into three topic classes (Pipeline, DescriptionEvidence,
CommittedEvidence). The evidence-gate harness both evidence classes
drive moves once into ai_review_workflows_helpers.py as
UxEvidenceGateHarness, a base class that holds no test.

Test bodies move verbatim; only the UX tests change class. The
workflow, doc and test comments that cite the moved pins name their
new files.

Part of #20192
@bolichen97
bolichen97 force-pushed the refactor/linecap-t02-test-ai-review-workflows-p3 branch from 70bacad to 5861761 Compare October 15, 2026 11:15
@bolichen97

Copy link
Copy Markdown
Contributor Author

Re-cut on main ef81402: #20583, #20619 and #20702 edited the original after the first push, so the split recipe was re-run and their text moves verbatim (see "Upstream changes carried"). 1,501 = 1,501 collected; default diff == --minimal (679,041 bytes); CI re-running on 5861761.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 15, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • test_ai_review_workflows.py still holds the pin pr-readiness.yml:58 cites

Outcome: rebutted (checked at this head; the citation is accurate).
pr-readiness.yml:58 ("pins the rule so a future lane cannot be added back") and :218 ("test/test_ai_review_workflows.py refuses") name two TestPrReadiness tests: test_no_monitored_lane_is_itself_workflow_run_triggered (test_ai_review_workflows.py:593) and test_the_job_gate_refuses_a_workflow_run_upstream (:634).
TestPrReadiness (lines 127-713) is one of the three classes the original keeps, so both pins stay where the workflow comments say; test_review_lane_prompt_parity.py reads that class by path too.
docs/ci/ci-and-reviews.md:3330 ("pins both", the trigger allowlist and job gate) names the same two tests.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The 1,501 collected tests on this head match main

Outcome: rebutted (checked on this head; holds).
pytest --collect-only over the 18 test_ai_review_workflows*.py files collects 1,501 tests at this head and 1,501 with main 213339f's own copies of those files restored; the rebase onto ef81402 changed no test file (range-diff "=").
The (class, test, params) multiset is identical once the 57 tests of the three TestUxReviewReadsTheScreenshotsBlindFirst* topic classes map back to that class; UxEvidenceGateHarness yields no node.
The 12 tests #20619 added (TestOpusTurnBudgetOutcome and one TestOpusTwoStageArchitecture test) are in that count, now in test_ai_review_workflows_opus.py.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by a maintainer after reading the review lanes: every lane is clean on this head, and the automated triage held it only on a size, path, category or doc rule.

@chenmingwei23
chenmingwei23 merged commit 7b4ce83 into main Oct 15, 2026
109 of 110 checks passed
@chenmingwei23
chenmingwei23 deleted the refactor/linecap-t02-test-ai-review-workflows-p3 branch October 15, 2026 15:42
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants