Skip to content

ci(design-review): verify readers of anything a PR takes away - #16092

Merged
chenmingwei23 merged 1 commit into
mainfrom
ci/design-review-take-away
Oct 2, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
ci/design-review-take-away

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Design Review checks, against the code, every reader of anything a PR takes away, and blocks on a broken reader the author did not list.

#13273 hid <config_dir>/apps from cron children. Every app cron that runs code out of its own apps/<app>/ bundle then failed before its first line (reverted by #16072). #12798 pruned synced crewmate config.agents rows after checking only the Crew page. Chat resume, subagent continue and spawn, and the prompt builder still read those records (fixed by #16016 and #16033). Both PRs passed every review lane. The author's "Compatible" claim was never checked against code.

Why it matters

A take-away with an unchecked reader ships a break on a path the PR never mentions. Both cases above broke core paths (app crons, chat resume, subagents) for every user who had the data, and needed follow-up PRs to repair.

Not a goal

  • Changing the PR template (a sibling PR adds the Reader: line format there).
  • CI format checks on the PR body.
  • First Principles (.github/review-prompts/first-principles.md): it guards over-building, and this check is out of its scope.

What changed (motivation → approach → change)

The Design Review lane took the author's compatibility line on trust, so a reader outside the PR's own surface was never looked at.

The reviewer now does the reader sweep itself. It lists what the patch takes away: a removed or renamed key or row, a migration that deletes or rewrites records, a newly hidden path, a stricter validator, a removed public function. It greps the tree for each one across every entry point: crew page, chat, subagent, cron, app bundles, prompt builder, release. Then it compares with the description's ## Backwards compatibility list, one Reader: <path>:<symbol> -- <entry> -- <why | test> line each, or Removes nothing: <why>. A reader a Compatible: line names by path:symbol counts as listed, and one a Breaking: line names counts as listed and accepted; a Compatible: naming no reader lists nothing. When a description cut at the 8000-byte capture cap is missing the section, or the section is the last one before the cut, the check caps at CONCERNS.

The prompt in .github/workflows/design-review.yml and its fork twin .github/workflows/fork-design-review.yml (same text in both) gains this TAKE-AWAY CHECK and a new BLOCK TRIGGER. A reader that breaks under the patch and is not listed blocks, named as path:symbol plus its entry point. Removes nothing lists no reader, so under it every breaking reader is unlisted. A listed reader with a weak reason is CONCERNS. The trigger joins the existing falsification rule with its own three items, and the one-BLOCK budget is unchanged. docs/ci/ci-and-reviews.md records the trigger and the section shape it reads.

Replay over the two diffs. The same-repo lane runs this PR's own prompt, but this PR takes nothing away, so its own review never exercises the new BLOCK; the replay is the evidence:

PR Take-away Reader the sweep finds Entry
#13273 <config_dir>/apps passed as extra_hidden_dirs in run_script_sandboxed / run_command_sandboxed src/kiro_crew/apps/bridges.py:_cron_defs_from_manifest hands the app's script / command, which live in and import from apps/<app>/, to src/kiro_crew/cron_script.py:resolve_script_path app, cron
#12798 crewmate_prune_migration.py deletes synced config.agents rows src/kiro_crew/config/loader.py:_resolve_agent_selection (chat resume), src/kiro_crew/subagent_persistence.py and subagent_manager/admission/gate.py ("selected member is unavailable"), src/kiro_crew/execution_context.py:execution_from_record (prompt builder member lookup) chat, subagent, crew page

Neither PR listed those readers, so both would have been a BLOCK naming them.

Backwards compatibility

Compatible: only review-prompt text changes. A PR that takes nothing away is reviewed as before. One that does now needs its readers listed, or it gets a Design Review BLOCK, which a human can still override.

Tests

test_ai_review_workflows.py::TestDesignTakeAwayCheck pins the TAKE-AWAY CHECK byte-identical across both design lanes (the fork lane is the one outside contributors get), and pins that an unlisted broken reader is BLOCK, a weak reason is CONCERNS, the Reader: shape, and the truncated-description cap to CONCERNS. It also checks that every entry-point path the sweep names still exists, so a moved file fails a test instead of leaving the sweep grepping a ghost path. test_ai_review_workflows.py, test_fork_stage2_checkout.py, test_pr_quality_gates.py and test_ci_fleet_routing_expression_parity.py pass locally.

Manual verification

Both workflow files parse as YAML, and the added lines are identical in the two. The replay above was checked against the base trees of #13273 (0496784bd1~1) and #12798 (9abfda6fe2~1).

Related Issues

Refs #13273, #12798, #16072, #16016, #16033

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner October 2, 2026 07:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 909dc4635ca6cd3fadb090f5d3e2b377e59e9f3b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 909dc46

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 909dc4635ca6cd3fadb090f5d3e2b377e59e9f3b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 909dc4635ca6cd3fadb090f5d3e2b377e59e9f3b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The replay table is the only evidence the new BLOCK fires correctly — author-asserted against #13273/#12798 base trees, never exercised by this PR's own run; verify it.

What this change ships

Inventory (7 items) — 7 justified

Intent: Design Review itself verifies every reader of anything a PR removes, hides or prunes, and blocks on an unlisted broken one — an ADDITION, derived from two linked incidents (#13273→#16072, #12798→#16016/#16033).

  1. Both Design lanes now run a reader sweep on every review — justified
  2. An unlisted broken reader is now a Design Review BLOCK — justified
  3. Descriptions gain a recognized Reader: / Removes nothing: listing grammar — justified
  4. A truncated description caps the check at CONCERNS — justified
  5. FALSIFY-BEFORE-BLOCK extended with the trigger's three facts — justified
  6. docs/ci/ci-and-reviews.md records the trigger — justified
  7. Tests pin cross-lane identity, verdict wiring, and that the 12 named entry-point paths exist — justified

No prior mechanism does this job: grepped compatib|caller|reader|sweep across .github/workflows/ — gate 6 asks the question but verifies nothing, and the template's writer sweep is author-side, Breaking-only. All 12 entry-point paths exist; the [description TRUNCATED notice is really emitted by pr-description-capture.sh:143, sourced by both lanes.

[FIRST-PRINCIPLES-REVIEWED] 909dc46

Description read

[DESCRIPTION-READ] 0523c33b2682d12907c966830bf0629d5895bb1445eaa47f1fd8ced5aea1a29b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 909dc4635ca6cd3fadb090f5d3e2b377e59e9f3b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Land the sibling PR-template PR promptly, so authors learn the Reader: line format from the template rather than from their first BLOCK.

[DESIGN-REVIEWED] 909dc46

Description read

[DESCRIPTION-READ] 27389d37b76aa2c9e544f7bdae9cfecc087b08425b19a65f78fd05abb413bb63

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 909dc4635ca6cd3fadb090f5d3e2b377e59e9f3b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 909dc46

Verdict parsed from the review's SHA-scoped output markers for commit 909dc4635ca6cd3fadb090f5d3e2b377e59e9f3b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 909dc4635ca6cd3fadb090f5d3e2b377e59e9f3b: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the ci/design-review-take-away branch from b33012d to d9c3753 Compare October 2, 2026 07:29
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: TAKE-AWAY CHECK block has no cross-lane parity pin span=67af5c2e131a

Fixed in d9c3753: test/test_ai_review_workflows.py::TestDesignTakeAwayCheck pins the whole TAKE-AWAY CHECK block byte-identical across design-review.yml and fork-design-review.yml, the same extraction the calibration-block pin uses.
It also pins the BLOCK / CONCERNS split, the Reader: line shape and the truncated-description cap in both lanes.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: a prose-disclosed accepted break reads as an empty list span=c0c457a215aa

Fixed in d9c3753: step 3 now counts a reader named in a Breaking: line (the shape .github/PULL_REQUEST_TEMPLATE.md already prescribes) as listed and accepted, so step 4's accepted-break branch is reachable before the sibling template PR lands.
A bare Compatible: stays an empty list on purpose: it is the unchecked claim #13273 and #12798 both shipped on.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: five-take-away sweep budget span=f61f8da7d37c

The lane runs with --max-turns 60 and only Read/Grep/Glob; each take-away costs several greps across seven entry points, so an uncapped sweep on a large PR exhausts turns and the lane emits no verdict at all, which is worse than a partial sweep.
The cap is not silent: step 2 requires every skipped take-away to be named in Watch, so a human sees exactly which readers were not checked.
The cap picks the five most likely to have readers outside the patch, so the remainder is the low-risk tail; on main today no review lane sweeps any of them.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: section shape, truncated description, and spec record (3 findings) span=a6115be27566

Fixed in d9c3753, all three:
(1) step 3 reads a Breaking: line's readers as listed and accepted, the template's existing vocabulary; the Reader: / Removes nothing: lines are the sibling template PR's format, read alongside it. A bare Compatible: stays an empty list on purpose: it is the unchecked claim this check exists to test.
(2) when pr-intent.txt carries the [description TRUNCATED notice, absence of the section is unconfirmed and the check caps at CONCERNS, mirroring the evidence trigger.
(3) docs/ci/ci-and-reviews.md now records the take-away trigger and the section shape it reads, beside the missing-evidence rule.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@iamwhatever
iamwhatever force-pushed the ci/design-review-take-away branch from d9c3753 to 45d11da Compare October 2, 2026 07:54
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: Removes nothing branch disagreed with the trigger and FALSIFY span=5c1d6032b843

Fixed in 45d11da: the separate Removes nothing BLOCK branch is gone. Step 4, the trigger and docs/ci/ci-and-reviews.md now say one thing: Removes nothing lists no reader, so a reader that BREAKS under it is unlisted and blocks; an intact reader under it does not.
FALSIFY's existing third item (the breaking reader's path:symbol, its line and entry point, its absence from the list) now covers every take-away BLOCK unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 2, 2026
@iamwhatever
iamwhatever force-pushed the ci/design-review-take-away branch from 45d11da to 448191d Compare October 2, 2026 08:22
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: truncation cap pinned presence-only span=05b7ffa004a9

Fixed in 448191d: the test now pins the whole clause "[description TRUNCATED notice: then absence is unconfirmed, so cap this check at CONCERNS", so flipping the cap to BLOCK fails the suite.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: entry-point paths in the sweep are unpinned span=1176f59e7e6b

Fixed in 448191d: TestDesignTakeAwayCheck::test_every_entry_point_the_sweep_names_exists asserts that every entry-point path the TAKE-AWAY CHECK names is still in the block and still exists in the tree, so a moved file fails a test instead of leaving the sweep grepping a ghost path.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: Removes nothing: spelling duplicates a bare Compatible: span=4546d2d240c7

The two do read the same in step 4, by design: neither names a reader, so a breaking reader under either is unlisted. The spelling is kept because it is the author format the sibling template PR introduces for the ## Backwards compatibility section (Reader: ... lines or Removes nothing: <why>).
Deleting it here would make the reviewer treat a template-conforming section as an unknown shape the day that template lands. The cost is one clause in step 3; it adds no branch, mechanism or test.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 2, 2026
@iamwhatever
iamwhatever force-pushed the ci/design-review-take-away branch from 448191d to a4925d7 Compare October 2, 2026 08:49
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: sweep list omits messaging and the CLI span=3138632dcc6d

The list is the floor, not the scope: step 2 says "Cover every entry point, not only the one the PR is about", and the reviewer greps the whole tree for each take-away, so a messaging or CLI reader is found by the same grep.
The named paths are the entry points this PR's motivating breaks went through; growing the list for surfaces no incident has hit adds prompt and pin surface without changing what the grep reaches.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: entry label outside the set would unlist a reader span=f61f8da7d37c

Fixed in a4925d7: step 3 now says a reader is listed by its <path>:<symbol>, and an entry label outside the set (e.g. prompt builder) does not unlist it, so no false BLOCK on a label.
The label set itself stays as the sibling template defines it; TestDesignTakeAwayCheck pins the new sentence in both lanes.

@iamwhatever
iamwhatever force-pushed the ci/design-review-take-away branch from a4925d7 to 9d40419 Compare October 2, 2026 09:16
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • accepted: land the sibling template PR first or together span=86ba5f6eb7d3

Agreed on order; that is a merge-sequencing call for the maintainer, not a code change here. Until the template lands, step 3 already reads the template's existing Breaking: lines as listed readers, so an author following today's template is not blocked for format alone.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: truncation cap reached a section the reviewer read span=a6115be27566

Fixed in 9d40419: the cap now applies only when NO ## Backwards compatibility heading is in the description file AND the file carries the TRUNCATED notice; "A section you read is judged on its text." So a bare Compatible: the reviewer read can still reach BLOCK on a long description.
TestDesignTakeAwayCheck pins both halves in both lanes, and docs/ci/ci-and-reviews.md says the same.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: add messaging surfaces to the named sweep list span=2bedf71e94b2

Same answer as span 3138632dcc6d: the named list is a floor and step 2 says to cover every entry point; the reviewer greps the whole tree for each take-away, so a channel handler that reads it is found by that grep.
Growing the list for a surface no incident has hit adds prompt and pin surface without changing what the grep reaches.

Design Review now runs a TAKE-AWAY CHECK: it lists what the patch removes,
renames, hides, tightens or migrates, greps the tree for readers across every
entry point (crew page, chat, subagent, cron, app bundles, prompt builder,
release), compares them with the description's Backwards compatibility list,
and BLOCKs on a broken reader the author did not list, or on 'Removes nothing'
when a take-away has readers outside the patch. A weakly justified listed
reader is CONCERNS. The new trigger joins the falsification rule and the
one-BLOCK budget. The fork twin carries the same text.

#13273 and #12798 both passed every lane while breaking readers their authors
never checked (app crons importing their own bundle; chat resume, subagent
continue and the prompt builder reading pruned crewmate rows).
@iamwhatever
iamwhatever force-pushed the ci/design-review-take-away branch from 9d40419 to 909dc46 Compare October 2, 2026 09:46
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: "cannot review itself" holds only for the fork lane span=f094eb001faa

Fixed in 909dc46: the description now says the same-repo lane runs this PR's own prompt, and that this PR takes nothing away, so the replay is the evidence for the new BLOCK.
docs/ci/ci-and-reviews.md now records the lane asymmetry beside the TAKE-AWAY CHECK paragraph: same-repo runs the PR's prompt (pull_request), fork runs the default branch's (workflow_run).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • accepted: land the sibling PR-template PR first or together span=5f5c1f4b6187

Merge order is the maintainer's call. In 909dc46 step 3 also reads a reader named by path:symbol in today's Compatible: or Breaking: line as listed, so an author on today's template is not blocked for format alone.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • rebutted: mark the entry-point list as a floor span=30fa7afe4ab7

Step 2 already says "Cover every entry point, not only the one the PR is about", and the reviewer greps the whole tree for each take-away, so a messaging, connections or workflows reader is found by that grep. Same answer as spans 3138632dcc6d and 2bedf71e94b2; no further list growth in this PR.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • fixed: Compatible: readers unlisted; heading-present truncation edge span=a6115be27566

Fixed in 909dc46, both:
(1) a reader a Compatible: line names by <path>:<symbol> now counts as listed, so the template's own spelling can list readers and step 4's CONCERNS bucket is reachable; a Compatible: naming no reader stays an empty list, the unchecked claim this check exists for.
(2) the CONCERNS cap now applies when the description carries the TRUNCATED notice and the section is missing OR is the last section before the notice; only a section that ends before the notice is judged on its text.
Both pinned in TestDesignTakeAwayCheck for both lanes; docs/ci/ci-and-reviews.md updated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@chenmingwei23
chenmingwei23 merged commit e64668b into main Oct 2, 2026
97 checks passed
@chenmingwei23
chenmingwei23 deleted the ci/design-review-take-away branch October 2, 2026 15:56
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants