Skip to content

docs(tests): flaky guard: change a faked host condition with the check that reads it - #37573

Merged
bolichen97 merged 1 commit into
mainfrom
flaky/fake-env-paired-with-check
Nov 15, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
flaky/fake-env-paired-with-check

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Test authors change a faked host condition and the production check that reads it together, so two parallel PRs cannot turn main red on every run.

Two PRs merged 26 minutes apart on 2026-11-15. #37460 removed the shared home fixture's is_default_home() -> True fake, because destinations() no longer read the home. #37463 then made off_offer_due() return False off the default home. Each was green on its own base. Together, three TestOffOffer tests in test/metrics/test_kiro_export_destination.py failed on every main run with assert False is True (#37546), and they blocked every open PR until #37545 put the fake back on that one class.

Why it matters

The red was not flaky: it failed every time, so repeat runs and the flake ledger cannot catch this shape. It held 16 open PRs red. Nothing in the test conventions told the author of either PR to search for the other side of the contract.

Not a goal

What changed (motivation → approach → change)

The convention now says a faked host condition (default home, platform, privilege) and the production check that reads it are one contract, changed in the same PR.

Where What it says
docs/system-specs/common/testing-conventions.md, new § A faked host condition changes with the check that reads it Adding a check means searching test/ for the tests that reach it. Removing a fake means a git grep for every assertion that still relies on it. Keep the fake on the class that needs it, plus one test that proves the "no" branch. The #37460 + #37463 case is told as the measured example.
writing-tests/SKILL.md, Rule 1b and the checklist The short form of the same rule, linking the section above.

Backwards compatibility

Compatible: documentation only.

Removes nothing: the diff only adds prose.

Tests

N/A — docs only. The related backend set (449 tests) passed locally.

Proof: not provable -- docs or config only

Manual verification

N/A — docs only; the new section anchor matches the link in SKILL.md.

Related Issues

Part of #37546

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 15, 2026 08:52
@bolichen97
bolichen97 requested a review from CrysisDeu November 15, 2026 08:52
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4147790b2979dd4a1db252e144c282ca72bc72fa touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4147790b2979dd4a1db252e144c282ca72bc72fa — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Docs-only convention change that matches the incident it describes. Nothing blocks.

Notes

  • SKILL.md ships in the package, but its new link goes up five levels to docs/system-specs/, which is outside the package. Installed copies get a dead link, so the short form in the skill has to make sense without it.

[DESIGN-REVIEWED] 4147790

Description read

[DESCRIPTION-READ] 82f6427b92cbba05af43512f86c9ea4b3bbca05182d55b10d4f566a4404e5783

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 4147790b2979dd4a1db252e144c282ca72bc72fa — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This PR only adds docs. It adds one rule, backed by a measured incident (#37546), in its spec section and the skill's short form. No existing doc already covers it.

What this change ships

Inventory (3 items) — 3 justified

Intent: test authors change a faked host condition together with the production check that reads it, so two parallel PRs can't turn main red on every run. This is an ADDITION (a convention).

  1. A new testing-conventions section says a host-condition fake and its check change in the same PR, and tells the feat(metrics): send product metrics from non-default data homes too #37460 + feat(privacy): offer Turn on once to installs off only by an old written false #37463 incident as the measured example. — justified
  2. The writing-tests skill's Rule 1b gets the short form of the same rule, linking to that section. — justified
  3. The writing-tests checklist gets a new item: keep the fake on the tests that need it, and git grep when adding a check or removing a fake. — justified

[FIRST-PRINCIPLES-REVIEWED] 4147790

Description read

[DESCRIPTION-READ] 307a1065070f2c5da8941b2c71fbe62438a99b6969f0fab86bd6217a125b781e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4147790b2979dd4a1db252e144c282ca72bc72fa — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4147790

Verdict parsed from the review's SHA-scoped output markers for commit 4147790b2979dd4a1db252e144c282ca72bc72fa.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4147790b2979dd4a1db252e144c282ca72bc72fa false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4147790b2979dd4a1db252e144c282ca72bc72fa and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4147790

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4147790b2979dd4a1db252e144c282ca72bc72fa false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 3% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This PR only adds documentation: a testing-conventions section and the matching Rule 1b paragraph and checklist item in the writing-tests skill. They say a faked host condition and the production check that reads it must change in the same PR. The incident it cites matches the code: off_offer_due() checks beacon.is_default_home() (kiro_export.py:238), TestOffOffer fakes it on its own class (test_kiro_export_destination.py:394-398), and its "home" case proves the "no" branch. The new link's relative path matches the 20 links of the same form already in SKILL.md. No code, state, limits or trust boundaries change, so nothing blocks.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 4147790b29 · 44 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 15, 2026 09:58

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff. Docs only: a testing-conventions section and a matching writing-tests skill note plus checklist line on keeping a faked host condition and the check that reads it in one PR. The relative link from the skill resolves to the new anchor. PR Readiness green on this head.

@bolichen97
bolichen97 merged commit 85bd1d6 into main Nov 15, 2026
100 of 101 checks passed
@bolichen97
bolichen97 deleted the flaky/fake-env-paired-with-check branch November 15, 2026 10:01
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants