Skip to content

test: ignore other threads' lockdowns in the remaining lockdown spies - #35491

Merged
bolichen97 merged 1 commit into
mainfrom
fix/33379-thread-scoped-lockdown-spies
Nov 19, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/33379-thread-scoped-lockdown-spies

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop the remaining restrict_to_owner spies in tests from failing when another thread locks down its own file while the test runs.

Many tests swap the process-global platform_compat.restrict_to_owner for a spy that records each call, then assert the exact list, its first entry, its length, or that it is empty. The spy records calls from every thread, so a lockdown on another thread lands in the list and fails a test whose code is fine. #33300 was this failure on macOS CI (475797 bytes); #33394 fixed the probes it named and left these spies to #33379.

Why it matters

Any backend PR can hit one of these reds, pay a rerun and sit blocked on PR Readiness for a defect that is not in its diff. Nineteen tests in fourteen files carry the same exposure.

Not a goal

  • No product code change: every writer under test already locks its file down before content reaches it.
  • No change to the SEL writer thread or to the directory it writes.
  • No change to spies that already keep only their own target's calls, or that only check membership.
  • No new assertion, and no assertion loosened.

What changed (motivation → approach → change)

The SEL writer is a daemon thread (sel-writer) that, on every append, creates its directory, re-locks its non-empty audit log (src/kiro_crew/sel_append.py:255) and opens it with os.fdopen. When it flushes events an earlier test queued while a test's spy is installed, its call is recorded beside the test's own and breaks an exact-list, first-entry, count or empty check.

Each affected spy now records only calls made on the test's own thread, the shape #33394 gave the lockdown-order probes. The writers under test lock down on that thread, so each assertion keeps its strength. Two writers run on worker threads (the offloaded config write, and the revoked-nonce store in its stuck-write test), so those spies act only on files in the test's own directory instead. Where a spy shares its recorded order with a Path.mkdir or os.fdopen spy that the SEL writer also reaches, that spy is scoped the same way.

Assertion another thread's call broke Spies, now keeping the test thread's calls only
an exact list of recorded calls test/test_atomic_write_capabilities.py, test/test_weixin_qr.py, test/test_handlers_messaging_coverage_channels.py, test/test_cli_setup.py, test/test_computer_use_capture.py, test/test_computer_use_capture_windows.py, test/test_cron_script_more_coverage.py (fixture script_run)
the first entry, or the count test/test_config_rmw_preserves_settings.py (classify-first test with its Path.mkdir spy, local-volume and synchronous-caller tests), test/test_token_auth_session_lifecycle.py (lockdown-size test)
an empty list, or no True test/test_platform_compat_coverage_filesystem.py, test/test_platform_compat_owner_only.py (wrong == []), test/test_lockdown_before_publish.py (shared helper, two tests)
a subset of allowed paths test/test_vector_memory.py
restrict:tmp before the first fdopen test/test_session_image_repair.py, with its os.fdopen spy
a count, or the first is_revoked check, where the writer runs on a worker thread (spy acts only on files in tmp_path) the offloaded-write test in test/test_config_rmw_preserves_settings.py, the stuck-write test in test/test_token_auth_session_lifecycle.py

Backwards compatibility

Compatible: test-only change; no product code touched.

Removes nothing: every spy still records each lockdown the code under test makes; only other threads' calls are left out, or, for the two worker-thread writers, calls on files outside tmp_path.

Tests

No test is added. The changed tests keep every assertion they had.

Determinism proof, on Linux, parent 80cf3dfc64e8 and this change on it (b9165612e818, the same patch as this head). The 19 changed tests run with two unchanged spies as negative controls: test_memory_backup.py's path-filtered stage spy and code_review_sage's store spy, which patches a binding the SEL writer never calls.

Run Parent This change
plain, -n0 21 passed 21 passed
forced: a thread named sel-writer does the SEL flush (mkdir, os.open, restrict_to_owner, os.fdopen, append) on a 4 KB log right after each spy is installed, after each tempfile.mkstemp and after each os.replace; each RevokedNonceStore.revoke starts 0.3 s late 19 failed (every changed test), 2 passed (the controls) 21 passed
20 repeats at -n0 n/a 0 failures
10 repeats at -n 4 --dist loadgroup --max-worker-restart=2, with test/test_aws_consent.py and test/test_webhooks_store.py as co-tenants n/a 0 failures (184 passed each)
the changed tests, shuffled (seed 27286), plain and forced n/a all passed
every test in the fourteen changed files, shuffled (seed 7421), -n 4 --dist loadgroup n/a 978 passed, 6 skipped

The forced condition is a throwaway pytest plugin, not committed.

Manual verification

N/A — test-only change; the forced-condition runs under Tests are the verification.

Real-machine verification

Machine: EC2 m7i.8xlarge (the author's Linux development host), Amazon Linux 2023 x86_64, 32 vCPU / 123 GiB
Before (main 80cf3df): python -m pytest -n0 -p <the forced-flush plugin under Tests> <the 19 changed tests, the 2 controls> -> 19 failed (every changed test), 2 passed (the controls)
After (head b916561): the same command -> 21 passed; 20 repeats at -n0 and 10 at -n 4 --dist loadgroup -> 0 failures
After the re-sync (this patch on main 5672158; EC2 m7i.2xlarge, Ubuntu 24.04 x86_64, 8 vCPU / 32 GiB): python -m pytest -n 4 --dist loadgroup --max-worker-restart=2 <the 14 changed files> -> 986 passed, 6 skipped
After the rebase (this patch on main 3b772f5, head ab27757; the author's development host above): python -m pytest -n 4 --dist loadgroup --max-worker-restart=2 <the 14 changed files> -> 986 passed, 6 skipped
Cleanup: own development host for the forced runs; the m7i.2xlarge instance is terminated (describe returns nothing), with no key pair and no inbound rules

Each re-sync since b916561 moved only the base: the patch adds and removes the same lines.

Screenshots / video

N/A — test-only change, no UI.

Related Issues

Closes #33379

Sibling sweep

Swept: git grep -nE '"restrict_to_owner"|restrict_to_owner",|platform_compat.restrict_to_owner"' -- test/ src/kiro_crew/apps, then each replacement function and the asserts that read what it records
Sibling: test/test_memory_backup.py:329 -- not affected -- records only this test's own backup stages, by path
Sibling: test/test_memory_backup.py:1093 -- not affected -- keeps only .tmp/.partial files in the backup dir
Sibling: test/test_member_memory_backup.py:133 -- not affected -- keeps only .tmp/.partial files in the backup dir
Sibling: test/test_live_target.py:497 -- not affected -- keeps only calls under live-target-staging
Sibling: test/test_mcp_gateway_apps_spool.py:226 -- not affected -- keeps only calls under the spool temp dir
Sibling: test/test_sel_edge_cases.py:117 -- not affected -- keeps only files in the trust dir
Sibling: test/test_mcp_gateway_rewriter.py:883 -- not affected -- keeps only calls under the stubs and overlay dirs
Sibling: test/test_mcp_gateway_rewriter.py:977 -- not affected -- reads only the agents dir's own sizes
Sibling: test/test_snapshot.py:1294 -- not affected -- keeps only calls under the snapshot output dir
Sibling: test/test_snapshot.py:1366 -- not affected -- asserts one named path is absent
Sibling: test/test_beacon.py:1062 -- not affected -- asserts membership of the config path only
Sibling: test/test_mcp_gateway_oversize.py:155 -- not affected -- asserts one named path is absent
Sibling: test/test_mcp_gateway_prewarm.py:101 -- not affected -- asserts the spy saw at least one call
Sibling: test/test_discord_sessions_binding_state.py:633 -- not affected -- asserts the spy saw at least one call
Sibling: test/test_webhooks_store.py:664 -- not affected -- asserts membership of named calls only
Sibling: test/test_vector_memory.py:1148 -- not affected -- asserts membership of the WAL path only
Sibling: test/test_vector_memory.py:1177 -- not affected -- asserts membership of the FAISS path only
Sibling: test/test_vector_memory.py:1226 -- not affected -- compares the positions of restrict:mem.db and connect only
Sibling: test/test_vector_memory.py:1278 -- not affected -- keeps only paths that did not exist yet
Sibling: test/test_memory_index_owner_only.py:41 -- not affected -- patches the module's own binding, which the SEL writer never calls
Sibling: test/test_secrets_vault.py:202 -- not affected -- patches the vault's own binding, which the SEL writer never calls
Sibling: src/kiro_crew/apps/builtins/code_review_sage/tests/test_store_selfheal.py:307 -- not affected -- patches the app store's own restrict_to_owner, which the SEL writer never calls
Sibling: test/test_sel_log_lockdown.py:59 -- not affected -- _on_event_loop is patched process-wide, so every append in the test skips the lockdown
Sibling: test/test_acp_frame_record.py:1154 -- not affected -- a call on another thread raises in that thread, never in this test; the SEL writer is restarted on its next use
Sibling: test/test_cli_logging.py:1308 -- not affected -- raises only, and the test keeps only its own loggers' records
Sibling: test/test_mcp_gateway_rewriter_fingerprint_keep.py:653 -- not affected -- refuses only one named agent file
Sibling: test/test_workflows_run_identity.py:427 -- not affected -- refuses only the one target path
Sibling: test/test_atomic_write_capabilities.py:179 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_live_target.py:671 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_session_image_repair.py:836 -- not affected -- raises only; the asserts read this test's own descriptors
Sibling: test/test_snapshot.py:1309 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_pod_env.py:298 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_denied_commands_api.py:1067 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_computer_use_snapshot.py:499 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_webhooks_store.py:671 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_memory_backup.py:829 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_mcp_gateway_manager_coverage.py:317 -- not affected -- raises only; the asserts check membership in the log text
Sibling: test/test_memory_index_owner_only.py:66 -- not affected -- patches the module's own binding, which the SEL writer never calls
Sibling: test/test_aws_consent.py:95 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_kiro_prerequisite.py:134 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_tips.py:927 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_webhooks_store.py:515 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_workflows_store_lockdown.py:39 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_session_transfer_layer_b.py:1491 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/tests/test_policy_store.py:696 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/tests/test_security.py:337 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/conftest.py:872 -- not affected -- a no-op replacement that records nothing
Sibling: test/md_notebook_helpers.py:184 -- not affected -- restores the real function
Sibling: test/test_md_notebook_pat_read_closes.py:44 -- not affected -- a no-op on the store's own binding
Sibling: test/test_secret_file_owner_dacl.py:27 -- tracked in #37631 -- the seen fixture records every thread's calls under exact-list, count and empty asserts; the file landed on main after these sites were swept
Sibling: test/test_secret_file_owner_dacl.py:130 -- not affected -- raises only; the asserts read this test's own file and log text
Sibling: test/test_secret_file_owner_dacl.py:142 -- not affected -- raises only; the assert reads the helper's own return value

Pattern harvest

Rule candidate: review-prompt
Pattern: "a test patches a process-global and asserts on the order or count of its calls without keeping only the test thread's calls"

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license (CONTRIBUTING.md, Licensing).

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 11, 2026 18:56
@bolichen97
bolichen97 requested a review from buluoray November 11, 2026 18:56
@KiroCrewCaptain

KiroCrewCaptain commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

This PR only changes tests. It scopes 19 restrict_to_owner spies, plus the mkdir and fdopen spies that share their recorded order, so they record only calls made on the test's own thread. For the two writers that run on worker threads, the spies record only files in tmp_path. I checked that every writer under test really locks down on the thread the spy expects:

  • _inline_on_the_loop uses asyncio.run on the calling thread.
  • The cron fixture, RevokedNonceStore.revoke, _write_env_updates and apply_repair are all called synchronously.
  • The revoked-nonce temp file sits directly in tmp_path.
  • pytest's tmp_path is already a resolved path, so the offloaded config write's parent check matches.

No check that expects an empty list became vacuous. The lockdown-before-publish helper still asserts seen is non-empty. The platform_compat empty-list tests call make_owner_only_dir on the test thread, so a wrong call would still be recorded. Exact-list, count and first-entry checks still fail if the code stops locking down, because the code locks down on the recording thread. The four files whose diff doesn't add the import already import threading. No assertion is lo

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit b9165612e8 · 166 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97
bolichen97 enabled auto-merge (squash) November 11, 2026 19:18
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 11, 2026
@bolichen97
bolichen97 force-pushed the fix/33379-thread-scoped-lockdown-spies branch from 7a9dd25 to b916561 Compare November 12, 2026 07:02
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_cli_setup.py, test/test_platform_compat_owner_only.py, test/test_token_auth_session_lifecycle.py since this head's CI ran (ee7a026 -> 80cf3df); CI re-running.

The Backend Tests shard 8 red on Linux and Windows (test/test_mcp_remote_transport.py::TestRewriter::test_a_listed_remote_entry_passes_through_unwrapped and ::test_a_malformed_type_does_not_abort_the_rewrite, assert 0 == 1) was main's (#35535), fixed by #35541, which this base carries. The patch is unchanged; only context lines moved.
[operator: bolichen97]

@buluoray

Copy link
Copy Markdown
Contributor

Review passed on this head. Please rebase onto the latest main so CI can run fresh; this head is 922 commits behind main, and its red checks (the AI review lanes and the Dependency License Gate) come from runs created on 11-12 that hit the API rate limit. The approval will follow once the rebased head is green.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Rebased onto current main to resolve the merge conflict in test/test_cli_setup.py: main added import sys where this branch adds import threading, and both are kept. The patch is otherwise unchanged, and the description's line citations are updated to the new base.

@buluoray this is the rebase you asked for; CI is running fresh on the new head.

@bolichen97
bolichen97 force-pushed the fix/33379-thread-scoped-lockdown-spies branch from b916561 to 1d4cc12 Compare November 17, 2026 21:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 23f651f4204d86055b098c3cbb98528927edb7b8 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 23f651f4204d86055b098c3cbb98528927edb7b8 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 23f651f4204d86055b098c3cbb98528927edb7b8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change that keeps each spy to its own thread or tmp_path, so other threads' lockdowns stop causing false reds, and no assertion is loosened.

Notes

[DESIGN-REVIEWED] 23f651f

Description read

[DESCRIPTION-READ] 7757583ca409007411d8491e5ec5a9aad9364ac607dd8ade5d2b5946314d5d5d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 23f651f4204d86055b098c3cbb98528927edb7b8 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 23f651f

Verdict parsed from the review's SHA-scoped output markers for commit 23f651f4204d86055b098c3cbb98528927edb7b8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 23f651f4204d86055b098c3cbb98528927edb7b8 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 23f651f4204d86055b098c3cbb98528927edb7b8 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 23f651f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 23f651f4204d86055b098c3cbb98528927edb7b8 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 17, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_cron_script_more_coverage.py since this head's CI ran (bdc7e7d -> 3b772f5); CI re-running.

The rebase onto current main applied cleanly; the patch is unchanged (same lines added and removed), now at ab27757. The only red on the previous head was the Windows IPv6 backend job, which the runner cancelled while it was still fetching the repository (no test ran), plus the two checks that are derived from it (Coverage Gate and PR Readiness); there was no code failure to fix. The 14 changed test files pass on the rebased head (986 passed, 6 skipped, 4 workers), and the PR description now follows the current template (it adds the real-machine verification and license sections).

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/33379-thread-scoped-lockdown-spies branch from 1d4cc12 to ab27757 Compare November 19, 2026 14:30
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red test/test_process_cache_floor_ratchet.py::test_every_new_module_level_cache_has_a_per_test_reset on this head (Backend Tests (3.12, 1) and Backend Tests (Windows) (4)) is red on main too: https://github.com/kirodotdev/KiroCrew/actions/runs/44543577290. It names _NOT_DUE_MEMO in src/kiro_crew/subagent_persistence.py, which this PR does not touch. The fix is #39479 (tracking issue #39478). This PR waits for it to merge, then rebases onto it so CI runs again.

The other two Windows reds on this head (shards 5 and 8) are known flaky tests in files this PR does not change; both are already recorded in the flake ledger.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
Many tests replace the process-global platform_compat.restrict_to_owner
with a spy that records each call, then assert the exact list, its
first entry, its length, or that it is empty. The SEL writer is a
daemon thread that re-locks its non-empty audit log on every append, so
a flush of events an earlier test queued can land in the spy and fail a
test on a change that has nothing to do with it.

Each of these spies now records only calls made on the test's own
thread, as the lockdown-order tests already do. The two writers that
lock down on worker threads (an offloaded config write, and the
revoked-nonce store's stuck-write test) are scoped to the test's own
directory instead. Where a spy shares its recorded order with a
Path.mkdir or os.fdopen spy that the SEL writer also reaches, that spy
is scoped the same way. Every assertion is unchanged.

Closes #33379
@bolichen97
bolichen97 force-pushed the fix/33379-thread-scoped-lockdown-spies branch from ab27757 to 23f651f Compare November 19, 2026 19:41
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/conftest.py and test/conftest_resets.py (#39479, the fix for the red test_process_cache_floor_ratchet.py check) since this head's CI ran (3b772f5 -> 976e001); CI re-running.

The rebase applied cleanly and the patch is unchanged (the same lines added and removed), now at 23f651f. On the previous head, test/test_process_cache_floor_ratchet.py::test_every_new_module_level_cache_has_a_per_test_reset was red on main too, and #39479 has merged its fix; the two other Windows reds (shards 5 and 8) are flaky tests already recorded in the flake ledger, in files this PR does not change. The description's sibling list cites test/conftest.py:872, where that no-op replacement sits on the new base.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head 23f651f and ran all 14 changed test files in an isolated environment: 986 passed, 6 skipped. The change touches only tests. Each spy now records only calls made on the test's own thread. For the two writers that run on worker threads, the spies record only files in the test's own directory. I checked that the code under test still locks down where the spy records, so no assertion was weakened. Paths that run inline on the event loop, and the owner-only directory helper, still run on the test thread, so the asserts that expect no call still catch one. No blocking issue found.

@bolichen97
bolichen97 merged commit 492fa11 into main Nov 19, 2026
101 of 102 checks passed
@bolichen97
bolichen97 deleted the fix/33379-thread-scoped-lockdown-spies branch November 19, 2026 23:40
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky risk: restrict_to_owner spies in tests also record the SEL writer thread's calls

3 participants