Skip to content

test: give every test the cold source-link settings snapshot - #39475

Merged
bolichen97 merged 1 commit into
mainfrom
crew/linuxflackyguard/source-link-case-flake
Nov 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/linuxflackyguard/source-link-case-flake

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: No test can leave the source-link settings snapshot (session-card chip switch and provider host allowlists) changed for the tests after it on the same xdist worker.

test_source_link_github_case.py::TestOneChipPerObject::test_case_variants_collapse_to_one_chip fails with source_links_total assert 0 == 1 on Backend Tests shard 7/8, on main (runs 44519135612, 44514765616) and on four PRs that do not touch source links (#39453, #39450, #39301, #36477).

Why it matters

Shard 7 goes red on unrelated PRs, so their authors rerun or chase a failure they did not cause.

Not a goal

  • Changing how the product publishes or reads the chip switch.
  • Extending test_process_cache_floor_ratchet.py to scalar globals.

What changed (motivation → approach → change)

dashboard.source_providers.hosts keeps the chip switch and five host allowlists as module globals. The real /api/dashboard/config PUT publishes the switch. test_session_card_source_links_knob.py::TestConfigEndpoint::test_put_persists_off sends false and puts nothing back, so every later to_dict on that worker reports no chips. #39470, now on main, pins the switch in that file's fixture. This PR adds the floor entry, so the next test in any file that publishes these settings cannot leak them either.

  • test/conftest_resets.py: new autouse _cold_source_link_settings pins the switch to True, the allowlists to frozenset() and the load stamp to 0.0 through _floor_monkeypatch, only when the module is already imported. The generation counter stays unpinned because the per-slot link cache keys on it.
  • docs/system-specs/common/testing-conventions.md § 4 and src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md: this leak shape, a second pre-push grep for True / False / frozenset() / number initializers, and a wrong/right pair.

Backwards compatibility

Compatible: test-only floor and docs; no product code changes.

Removes nothing: the diff adds a test fixture and documentation.

Tests

No test added: the change is the floor fixture itself.

Determinism proof on main 8301e68663 (which already has #39470): a throwaway test that calls hosts._publish_session_card_chips(False) and restores nothing, run before test/test_source_link_github_case.py at -n0, x20: main's conftest 20/20 red (4 failed, 17 passed), this head 0/20 red (21 passed). The probe file was not committed.

Proof: not provable -- test-only change

Manual verification

N/A — covered by the real-machine runs below.

Real-machine verification

Machine: DevSpace (KiroCrewBlueprint), Amazon Linux 2023, kernel 6.1, x86_64, 8 vCPU / 30 GB, Python 3.12.16
Before (main 8c917a5): pytest -n0 -q test/test_session_card_source_links_knob.py::TestConfigEndpoint::test_put_persists_off test/test_source_link_github_case.py x20 -> 20/20 red, 4 failed, 17 passed (assert 0 == 1)
After (head f8ea3ca7380d, this change before the rebase onto #39470): same command x20 -> 0/20 red, 21 passed; both files at -n 4 x10 -> 0/10 red
Full-suite regression: CI's Backend Tests shards on head ac44e2d were green except test_process_cache_floor_ratchet.py::test_every_new_module_level_cache_has_a_per_test_reset (_NOT_DUE_MEMO), which is red on main itself and is fixed by #39479.
Cleanup: DevSpace ds-chip-floor-verify-jlsk056l deleted (devspaces delete reported removed)

Related Issues

Part of #39460

Relationship to #39470 (merged): #39470 pins the chip switch in test_session_card_source_links_knob.py's handler_app fixture, which stops the one polluter found. This PR adds the per-test floor reset in test/conftest_resets.py for every snapshot in dashboard.source_providers.hosts, plus the convention text. The two touch different files; with both on main, the fixture pin is redundant but harmless.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 432e233d81ca86461bf495820208c9a469e48fd5 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 432e233d81ca86461bf495820208c9a469e48fd5 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only floor reset, it follows the existing _floor_monkeypatch pattern, and it is easy to revert.

Notes

  • Resetting the snapshots without bumping _gitlab_hosts_generation means a test can see snapshot True at the same generation an earlier test cached under False. This is safe only while no generation-keyed projection cache outlives a test. slot_projection.py:222 and slots_push.py:590 key on that generation.

[DESIGN-REVIEWED] 432e233

Description read

[DESCRIPTION-READ] 532c6a9f3771b48a3bfe173c3537a6c49c7f62b1e812fbef3d47d7295e4ad1ad

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 432e233d81ca86461bf495820208c9a469e48fd5 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The floor entry fixes the cause: every snapshot that hosts publishes gets reset for each test, which goes beyond the one polluter #39470 pinned. Nothing in the change blocks.

What this change ships

Inventory (4 items) — 4 justified

Intent: stop tests from leaving the source-link chip switch or host allowlists changed for later tests on the same worker, which reddens unrelated shards. This is a FIX.

  1. Every test now starts with the chip switch on, the five host allowlists empty and the load stamp at zero (new autouse floor fixture) — justified
  2. The generation counter is deliberately left unreset, because the link cache keys on it — justified
  3. The testing-conventions § 4 spec gains this leak shape, a second pre-push grep for scalar and frozen initializers, and a wrong/right pair — justified
  4. The writing-tests skill gains a pointer to that section and to the new fixture — justified

[FIRST-PRINCIPLES-REVIEWED] 432e233

Description read

[DESCRIPTION-READ] 0803547db2b637edc1adaba0d82843fcbbd2bc31e870ca1905a6ff664829c1f3

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 432e233d81ca86461bf495820208c9a469e48fd5 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 432e233

Verdict parsed from the review's SHA-scoped output markers for commit 432e233d81ca86461bf495820208c9a469e48fd5.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 432e233d81ca86461bf495820208c9a469e48fd5 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 432e233d81ca86461bf495820208c9a469e48fd5 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 432e233

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 432e233d81ca86461bf495820208c9a469e48fd5 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 marked this pull request as ready for review November 19, 2026 13:24
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 19, 2026 13:24
@bolichen97
bolichen97 requested a review from CrysisDeu November 19, 2026 13:24
@bolichen97
bolichen97 enabled auto-merge (squash) November 19, 2026 13:25
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/source-link-case-flake branch from ac44e2d to 4bc1f9d Compare November 19, 2026 15:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/source-link-case-flake branch from 4bc1f9d to 58248f4 Compare November 19, 2026 15:28
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@bolichen97
bolichen97 force-pushed the crew/linuxflackyguard/source-link-case-flake branch from 58248f4 to 432e233 Compare November 19, 2026 20:01
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@bolichen97
bolichen97 merged commit 1029fe4 into main Nov 20, 2026
152 of 173 checks passed
@bolichen97
bolichen97 deleted the crew/linuxflackyguard/source-link-case-flake branch November 20, 2026 00:00
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant