Repository navigation
docs(tests): flaky guard: change a faked host condition with the check that reads it - #37573
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Docs-only convention change that matches the incident it describes. Nothing blocks. Notes
[DESIGN-REVIEWED] 4147790 Description read[DESCRIPTION-READ] 82f6427b92cbba05af43512f86c9ea4b3bbca05182d55b10d4f566a4404e5783 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS This PR only adds docs. It adds one rule, backed by a measured incident (#37546), in its spec section and the skill's short form. No existing doc already covers it. What this change shipsInventory (3 items) — 3 justifiedIntent: test authors change a faked host condition together with the production check that reads it, so two parallel PRs can't turn
[FIRST-PRINCIPLES-REVIEWED] 4147790 Description read[DESCRIPTION-READ] 307a1065070f2c5da8941b2c71fbe62438a99b6969f0fab86bd6217a125b781e That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 3% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This PR only adds documentation: a testing-conventions section and the matching Rule 1b paragraph and checklist item in the writing-tests skill. They say a faked host condition and the production check that reads it must change in the same PR. The incident it cites matches the code: off_offer_due() checks beacon.is_default_home() (kiro_export.py:238), TestOffOffer fakes it on its own class (test_kiro_export_destination.py:394-398), and its "home" case proves the "no" branch. The new link's relative path matches the 20 links of the same form already in SKILL.md. No code, state, limits or trust boundaries change, so nothing blocks. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Reviewed the full diff. Docs only: a testing-conventions section and a matching writing-tests skill note plus checklist line on keeping a faked host condition and the check that reads it in one PR. The relative link from the skill resolves to the new anchor. PR Readiness green on this head.
Problem / Motivation
Goal: Test authors change a faked host condition and the production check that reads it together, so two parallel PRs cannot turn
mainred on every run.Two PRs merged 26 minutes apart on 2026-11-15. #37460 removed the shared
homefixture'sis_default_home() -> Truefake, becausedestinations()no longer read the home. #37463 then madeoff_offer_due()returnFalseoff the default home. Each was green on its own base. Together, threeTestOffOffertests intest/metrics/test_kiro_export_destination.pyfailed on everymainrun withassert False is True(#37546), and they blocked every open PR until #37545 put the fake back on that one class.Why it matters
The red was not flaky: it failed every time, so repeat runs and the flake ledger cannot catch this shape. It held 16 open PRs red. Nothing in the test conventions told the author of either PR to search for the other side of the contract.
Not a goal
TestOffOffer.What changed (motivation → approach → change)
The convention now says a faked host condition (default home, platform, privilege) and the production check that reads it are one contract, changed in the same PR.
docs/system-specs/common/testing-conventions.md, new § A faked host condition changes with the check that reads ittest/for the tests that reach it. Removing a fake means agit grepfor every assertion that still relies on it. Keep the fake on the class that needs it, plus one test that proves the "no" branch. The #37460 + #37463 case is told as the measured example.writing-tests/SKILL.md, Rule 1b and the checklistBackwards compatibility
Compatible: documentation only.
Removes nothing: the diff only adds prose.
Tests
N/A — docs only. The related backend set (449 tests) passed locally.
Proof: not provable -- docs or config only
Manual verification
N/A — docs only; the new section anchor matches the link in
SKILL.md.Related Issues
Part of #37546
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)