Skip to content

docs(prepare-pr): make two silent preflight failures loud - #20972

Merged
iamwhatever merged 2 commits into
mainfrom
replace/19453
Oct 16, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
replace/19453

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A prepare-pr preflight that reports a clean file-overlap scan has actually scanned every open PR, and a contributor on an older python3 is told which interpreter to use instead of meeting a confusing crash.

Two Phase 0 gates in kirocrew-dev/kirocrew-prepare-pr pass while having checked little or nothing.

The file-overlap gate is specified as gh pr list --state open --limit 500. Its own rationale says the --limit is load-bearing because the default of 30 rows "silently checks about a sixth of them and the gate reads as passing", and that reasoning now applies to the cap itself: this repository has more than 700 open PRs, so --limit 500 returns a subset, and nothing in the output tells a truncated scan from a clean one.

Separately, every script in the skill is run as python3 $SKILL_DIR/scripts/..., and the scripts carry PEP-604 annotations (str | None) that an interpreter below 3.12 evaluates at class-definition time. On a host whose python3 is 3.9 the first script dies with unsupported operand type(s) for |: 'type' and 'NoneType'. The 3.12 floor lives only in profiles/kirocrew.json's gate-check command, which runs after Phase 0's first script call.

Why it matters

A preflight gate's only value is that a clean result can be trusted. The overlap gate exists to catch another open PR rewriting a file you are about to change. A gate that quietly examines two thirds of the candidates prints the same clean result, so the contributor proceeds with the false confidence the gate was written to remove.

The interpreter floor is paid by every contributor on a distribution whose python3 is older than 3.12. The error surfaces inside a skill script, so it reads as a broken script rather than a wrong interpreter.

Not a goal

  • Does not change any script. Both failures are in what the skill instructs, and both are fixed there.
  • Does not add a version guard to the scripts themselves.
  • Does not change the overlap gate's decision rule (the >50% line-delta threshold, and stopping to ask which PR hosts the work). Only its scope and the proof that the scope was complete.
  • Does not touch other skills that ship a similar scan.

What changed (motivation → approach → change)

Overlap gate. Symptom: a clean scan that examined 500 of 700+ open PRs. Root cause: the gate's correctness depends on a hardcoded cap staying ahead of the open-PR count. A bigger cap only moves the cliff, and at this size the single gh pr list ... --json number,files request answers HTTP 504 anyway. So the gate no longer uses a cap: it pages with cursors (gh api graphql --paginate --slurp, pullRequests(states: OPEN, first: 50)) and asserts the last page's hasNextPage is false. Truncation has a second axis that survives any outer cap: each PR's own files connection caps at 100, so the gate also checks each PR's files.pageInfo.hasNextPage and re-reads a truncated one with gh api --paginate repos/<owner>/<repo>/pulls/<n>/files.

Python floor. Root cause is placement, not absence. The floor is now stated where the scripts are introduced, with the exact error text, so the failure reads as a wrong interpreter.

Size budget. SKILL.md has a 56000-byte ceiling (test_slim_skills_do_not_regrow_duplicate_guidance_or_wire_model_ids), and main sat 49 bytes under it. The original commit inlined the full procedure and went to 57382 bytes. Following the review on #19453, the second commit moves the detail out and leaves pointers:

  • SKILL.md File overlap bullet: one line, pointing to a new File-overlap scan section in references/ci.md that carries the paged command and both truncation checks.
  • SKILL.md Scripts paragraph: Python >=3.12 plus the error text in one clause.
  • references/rationale.md: why a cap is not the fix (both axes, the 504), and why the floor must be stated before the first script call.

SKILL.md is now 55961 bytes.

Backwards compatibility

Compatible: documentation-only. No script, profile, schema or command signature changes, so nothing that works today stops working.

Removes nothing: the gh pr list --limit 500 instruction is replaced by the paged scan in the same gate; no script reads that literal (git grep -- '--limit' src/kiro_crew/builtin_skills/kirocrew-dev/kirocrew-prepare-pr/scripts finds no PR-list call).

Tests

No new test: the change is skill prose. The existing size and wording guards were run locally on Python 3.12: the 38 test files that read the prepare-pr skill (including test/test_review_repair_routing_skill.py and test/test_prepare_pr_profiles.py) give 2471 passed.

Manual verification

On 2026-10-15 against kirodotdev/KiroCrew, the exact command now in references/ci.md returned 749 open PRs over 15 pages with a final hasNextPage: false, and 15 PRs with files.pageInfo.hasNextPage: true (#5274, #7157, #9201, #11100, #11397, #14089, #15720, #15957, #16481, #17554, #19245, #19274, #19833, #19883, #20965). This host's /usr/bin/python3 is 3.9.25, which is the interpreter the floor warns about.

Related Issues

Fixes #19452
Replaces #19453

This is a same-repo carry of @kidbrax's #19453 (the fork's red was the SKILL.md size budget, plus main-side reds since fixed). The first commit is cherry-picked unchanged with @kidbrax's authorship; the second only moves text to fit the size budget, as the review on #19453 asked. All credit for the fix is theirs.

Pattern harvest

Rule candidate: review-prompt
Pattern: a paginated API read whose completeness is assumed from a hardcoded cap, rather than asserted from the response's own truncation signal, so growth past the cap silently narrows the check while its output still reads as complete.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

Braxton Beyer and others added 2 commits October 15, 2026 22:07
Both of these failed the same way: the gate produced a pass while having
checked little or nothing.

File overlap. The gate's cap was `--limit 500`, documented as load-bearing
because `gh pr list` returns 30 rows by default. The repo has since passed
500 open PRs (723 on 2026-10-12), so the cap itself began truncating
silently in exactly the way the default did, and a truncated scan is
indistinguishable from a clean one. Raising the cap alone would not fix it,
because the cap is a number someone has to keep ahead of the repo, so the
gate now asserts instead: if the row count equals the `--limit` passed,
read that as truncation rather than as a pass. Two further facts the gate
did not account for, both measured while fixing it: at this size the single
request 504s, so it pages with cursors and asserts the final `hasNextPage`
is false; and truncation has a second axis that survives any outer cap,
because each PR's own `files` connection caps at 100 and 12 open PRs exceed
that, so an overlap past file 100 of a large PR was invisible. Each PR's
`files.pageInfo.hasNextPage` is now checked, and a truncated one re-read in
full.

Python floor. The scripts carry PEP-604 annotations (`str | None`) that an
interpreter below 3.12 evaluates at class-definition time, so on a host
whose `python3` is 3.9 the first script invoked dies with
`unsupported operand type(s) for |` before doing any work. The floor was
only asserted inside the kirocrew profile's gate-check command, which runs
well after Phase 0's first `python3 $SKILL_DIR/scripts/...` call, so the
reader met the crash before the requirement. State it where the scripts are
introduced, with the error text, so the failure is recognisable as a wrong
interpreter rather than a broken script.
The first commit pushed kirocrew-prepare-pr/SKILL.md to 57382 bytes, past
the 56000-byte ceiling in test_slim_skills_do_not_regrow. Move the detail
out and leave short pointers behind, as the review on the original PR
suggested.

- File overlap: SKILL.md keeps one line and points to a new File-overlap
  scan section in references/ci.md. That section has the paged GraphQL
  command and the two truncation checks (last hasNextPage, and each PR's
  files.pageInfo.hasNextPage). It drops the gh pr list --limit form, which
  504s at this repo's size.
- Python floor: SKILL.md states 3.12 and the error text in one clause;
  the PEP-604 explanation moves to references/rationale.md.

SKILL.md is now 55961 bytes.

Co-authored-by: Braxton Beyer <bebraxto@amazon.com>
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision fb69896117d839536063dc92e256a069b84a4faf touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of fb69896117d839536063dc92e256a069b84a4faf — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check before merge: rationale.md says "12 open PRs exceed 100 files" and "723 on 2026-10-12", but the description measured 15 PRs and 749 on 2026-10-15. Dated counts are already stale at merge.

What this change ships

Inventory (4 items) — 4 justified

Intent: a contributor running prepare-pr preflight can trust a clean overlap scan, and gets told the Python version they need before a script crashes. FIX (#19452).

  1. The file-overlap gate pages through every open PR with cursors and confirms the last page has no next page, instead of stopping at 500 — justified
  2. A PR with more than 100 files gets its full file list re-read, so an overlap past file 100 is no longer missed — justified
  3. SKILL.md now states the Python >=3.12 floor, with the crash text, where the scripts are introduced — justified
  4. rationale.md explains why a raised cap doesn't fix the scan and why the floor has to be stated early — justified

[FIRST-PRINCIPLES-REVIEWED] fb69896

Description read

[DESCRIPTION-READ] 721636b74eddab7a46891eb5ba17616aac5bfc9518d6175ba08e29d9e4c30631

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of fb69896117d839536063dc92e256a069b84a4faf — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] fb69896

Description read

[DESCRIPTION-READ] 0cb40b38bc89a092ce28224c8e3c2bba468c68a2a731a0f9a9d6949039043698

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fb69896117d839536063dc92e256a069b84a4faf — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fb69896

Verdict parsed from the review's SHA-scoped output markers for commit fb69896117d839536063dc92e256a069b84a4faf.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fb69896117d839536063dc92e256a069b84a4faf: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fb69896117d839536063dc92e256a069b84a4faf and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fb69896

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fb69896117d839536063dc92e256a069b84a4faf: <one-sentence reason>

@iamwhatever
iamwhatever marked this pull request as ready for review October 15, 2026 23:03
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 15, 2026 23:03
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 15, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head fb69896117d839536063dc92e256a069b84a4faf.

Both halves of #19452 are fixed, checked locally:

  • Overlap scan. The references/ci.md query is valid and carries both truncation signals: a small page (first: 2, files(first: 3)) returns pullRequests.pageInfo { hasNextPage endCursor } ahead of nodes, which is the cursor gh api graphql --paginate follows, plus a per-PR files.pageInfo.hasNextPage. No --limit 500 remains as an instruction in the skill; the only remaining literal is the rationale's explanation of why a cap fails.
  • Python floor. On a python3 3.9 host, resolve_profile.py (the first script SKILL.md calls, at :304) dies with exactly TypeError: unsupported operand type(s) for |: 'type' and 'NoneType', from its -> str | None method annotations. That is the error text the new Scripts line names.
  • Size budget. SKILL.md is 55961 bytes, under the 56000 ceiling, and main has not touched this skill since the merge base. Locally, test_review_repair_routing_skill.py (which holds the size guard), test_prepare_pr_profiles.py, test_prepare_pr_skill_shape.py, test_skills.py and 8 neighbouring skill-guard files: 470 passed.

Non-blocking: references/rationale.md pins dated counts ("723 on 2026-10-12", "12 open PRs exceed 100 files") that are already behind the 749 and 15 measured in the PR body, and will keep drifting. Wording that does not depend on the count (e.g. "the repo passed 500 open PRs", "some open PRs exceed 100 files") would stay true.

@chenmingwei23 chenmingwei23 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving: readiness: passed, all lanes green, no outstanding change requests.

@iamwhatever
iamwhatever merged commit 539c835 into main Oct 16, 2026
106 checks passed
@iamwhatever
iamwhatever deleted the replace/19453 branch October 16, 2026 05:54
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

prepare-pr: the file-overlap gate truncates silently (723 open PRs vs --limit 500), and the Python >=3.12 floor is stated after the first script runs

3 participants