Skip to content

test: reset the vault read audit's process caches in the test floor - #37754

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/read-audit-floor-reset
Nov 16, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/linuxflackyguard/read-audit-floor-reset

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: no test inherits the vault read audit's cached state from an earlier test on the same worker.

#37659 added two process globals to kiro_crew.secrets.read_audit. _loaded holds the audit switch read from config.json, reused for CONFIG_TTL_SECS (30 s) when there is no live config snapshot. _seen holds the per-secret rate-limit windows. Every vault read that returns a secret consults both. Only test/test_vault_read_audit.py reset them, in a file-local autouse fixture. So a vault read in one test decided whether a later test on the worker loaded config once or twice. That made test_status_loads_config_off_the_event_loop red by order (#37737; that test was pinned cold in #37726).

Why it matters

#37726 fixed one victim. Any other test that counts config loads or audit rows on a vault-read path can still pass or fail by order.

Not a goal

  • No change to read_audit or the vault.

What changed (motivation -> approach -> change)

The state is process-wide, so its reset belongs in the shared floor, not one file -> reset both globals before every test/ test, through _floor_monkeypatch, only when the module is already imported (the _fresh_aws_bin_dir_trust pattern) ->

  • test/conftest_resets.py: new autouse _cold_vault_read_audit sets _loaded = None and _seen = {}.
  • test/conftest.py: binds it in the conftest_resets import block.
  • test/test_vault_read_audit_floor.py: two cases in one xdist_group. Each asserts the cold state, then warms both caches the way a vault read does. Whichever case runs second fails when the reset is missing.

Backwards compatibility

Compatible: test-only.

Tests

Linux, 8 cores. "base" = main at 5c8ce43638 plus only the new guard file. "floor" = this branch.

Run base floor
guard, serial 1 failed 2 passed
guard, -n 4 --dist loadgroup 1 failed 2 passed
guard, --random-order seeds 1-5 5/5 red 5/5 green
71 files that touch the vault (SecretVault / read_audit / vault.get), -n 6 --random-order seed 7 1 failed: guard [first] saw _loaded == (65488.9, True) left by an unrelated test 3239 passed
pre-#37726 test_status_loads_config_off_the_event_loop (asserts one load) shuffled with test_vault_read_audit.py and test_connections_status.py, seeds 1-6 3 red, 3 green (order-dependent) 6/6 red (same verdict every order)
current test_wakatime_handlers.py + test_vault_read_audit.py 68 passed 68 passed

The base row for the 71 files is the leak this closes, seen without any planted warm-up.

Proof: forced condition red on the parent and green on the fix, shown above.

Manual verification

N/A: test-only.

Screenshots / video

N/A: no user-visible UI change.

Related Issues

Part of #37737

secrets.read_audit keeps _loaded (the audit switch read from config.json,
reused for 30 s with no live snapshot) and _seen (per-secret windows) for
the process. #37659 reset them only in test_vault_read_audit.py, so any
other test on the worker inherited a warm cache, which made
test_status_loads_config_off_the_event_loop count config loads by order.
Reset both in test/conftest_resets.py and add a two-case guard that fails
on whichever case runs second when the reset is missing.

Part of #37737
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 16, 2026 01:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3f6fe0f617466474a39dec4a8d93a9f4d5044157 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 3f6fe0f617466474a39dec4a8d93a9f4d5044157 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3f6fe0f617466474a39dec4a8d93a9f4d5044157 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. This is a test-only reset of process-wide audit caches, and it sits in the shared conftest floor, which is where that kind of reset belongs.

[DESIGN-REVIEWED] 3f6fe0f

Description read

[DESCRIPTION-READ] 36304fb96ae5ac45d4efb639b4220716f9f4dee372a3fcc2e78a5b63230324ee

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

bolichen97 added a commit that referenced this pull request Nov 16, 2026
…7659

testing-conventions §4 already says a new process-global ships its floor
entry. #37659's secrets.read_audit._loaded did not, and made
test_status_loads_config_off_the_event_loop red by order (fixed in #37726,
floor entry in #37754). Record why the rule did not stop it and add a
pre-push grep that flags new module-level state, plus a writing-tests
pointer and checklist item.

Part of #37737
@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3f6fe0f617466474a39dec4a8d93a9f4d5044157 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3f6fe0f

Verdict parsed from the review's SHA-scoped output markers for commit 3f6fe0f617466474a39dec4a8d93a9f4d5044157.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3f6fe0f617466474a39dec4a8d93a9f4d5044157 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 3f6fe0f617466474a39dec4a8d93a9f4d5044157 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3f6fe0f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3f6fe0f617466474a39dec4a8d93a9f4d5044157 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 16, 2026 02:53

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head the review checked several things. The new _cold_vault_read_audit floor fixture resets read_audit._loaded and read_audit._seen through _floor_monkeypatch, and only when the module is already imported. Those names and value shapes match src/kiro_crew/secrets/read_audit.py. Teardown restores the original values, and the fixture's undo order with the _rig fixture in test_vault_read_audit.py is correct. No production code changes. The guard file was run serially and under -n 2 --dist loadgroup, and both passed. With both reset lines removed, the second case fails. Removing only the _loaded line, or only the _seen line, also fails it, so each line has its own pin.

Done well: the reset sits in the shared test floor where the process-global lives, not in one more victim file, so every test that reaches a vault read starts cold.

Optional:

  • The local resets in test/test_vault_read_audit.py:51-52 and test/test_wakatime_handlers.py:507 now duplicate the floor. They do no harm and can stay, or a follow-up can drop them.

@KiroCrewCaptain
KiroCrewCaptain merged commit 9997e80 into main Nov 16, 2026
98 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/linuxflackyguard/read-audit-floor-reset branch November 16, 2026 02:54
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 16, 2026
bolichen97 added a commit that referenced this pull request Nov 16, 2026
…7659 (#37750)

testing-conventions §4 already says a new process-global ships its floor
entry. #37659's secrets.read_audit._loaded did not, and made
test_status_loads_config_off_the_event_loop red by order (fixed in #37726,
floor entry in #37754). Record why the rule did not stop it and add a
pre-push grep that flags new module-level state, plus a writing-tests
pointer and checklist item.

Part of #37737
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants