Repository navigation
test: ignore other threads' lockdowns in the remaining lockdown spies - #35491
Conversation
Code change approver recommendation: ApproveWhy: the decider estimates a 6% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. This PR only changes tests. It scopes 19
No check that expects an empty list became vacuous. The lockdown-before-publish helper still asserts An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
7a9dd25 to
b916561
Compare
|
Rebased: main moved in test/test_cli_setup.py, test/test_platform_compat_owner_only.py, test/test_token_auth_session_lifecycle.py since this head's CI ran (ee7a026 -> 80cf3df); CI re-running. The Backend Tests shard 8 red on Linux and Windows ( |
|
Review passed on this head. Please rebase onto the latest main so CI can run fresh; this head is 922 commits behind main, and its red checks (the AI review lanes and the Dependency License Gate) come from runs created on 11-12 that hit the API rate limit. The approval will follow once the rebased head is green. |
|
[operator: bolichen97] Rebased onto current main to resolve the merge conflict in @buluoray this is the rebase you asked for; CI is running fresh on the new head. |
b916561 to
1d4cc12
Compare
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only change that keeps each spy to its own thread or Notes
[DESIGN-REVIEWED] 23f651f Description read[DESCRIPTION-READ] 7757583ca409007411d8491e5ec5a9aad9364ac607dd8ade5d2b5946314d5d5d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Rebased: main moved in test/test_cron_script_more_coverage.py since this head's CI ran (bdc7e7d -> 3b772f5); CI re-running. The rebase onto current main applied cleanly; the patch is unchanged (same lines added and removed), now at ab27757. The only red on the previous head was the Windows IPv6 backend job, which the runner cancelled while it was still fetching the repository (no test ran), plus the two checks that are derived from it (Coverage Gate and PR Readiness); there was no code failure to fix. The 14 changed test files pass on the rebased head (986 passed, 6 skipped, 4 workers), and the PR description now follows the current template (it adds the real-machine verification and license sections). [operator: bolichen97] |
1d4cc12 to
ab27757
Compare
|
[operator: bolichen97] The red The other two Windows reds on this head (shards 5 and 8) are known flaky tests in files this PR does not change; both are already recorded in the flake ledger. |
Many tests replace the process-global platform_compat.restrict_to_owner with a spy that records each call, then assert the exact list, its first entry, its length, or that it is empty. The SEL writer is a daemon thread that re-locks its non-empty audit log on every append, so a flush of events an earlier test queued can land in the spy and fail a test on a change that has nothing to do with it. Each of these spies now records only calls made on the test's own thread, as the lockdown-order tests already do. The two writers that lock down on worker threads (an offloaded config write, and the revoked-nonce store's stuck-write test) are scoped to the test's own directory instead. Where a spy shares its recorded order with a Path.mkdir or os.fdopen spy that the SEL writer also reaches, that spy is scoped the same way. Every assertion is unchanged. Closes #33379
ab27757 to
23f651f
Compare
|
Rebased: main moved in test/conftest.py and test/conftest_resets.py (#39479, the fix for the red The rebase applied cleanly and the patch is unchanged (the same lines added and removed), now at 23f651f. On the previous head, [operator: bolichen97] |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head 23f651f and ran all 14 changed test files in an isolated environment: 986 passed, 6 skipped. The change touches only tests. Each spy now records only calls made on the test's own thread. For the two writers that run on worker threads, the spies record only files in the test's own directory. I checked that the code under test still locks down where the spy records, so no assertion was weakened. Paths that run inline on the event loop, and the owner-only directory helper, still run on the test thread, so the asserts that expect no call still catch one. No blocking issue found.
Problem / Motivation
Goal: Stop the remaining
restrict_to_ownerspies in tests from failing when another thread locks down its own file while the test runs.Many tests swap the process-global
platform_compat.restrict_to_ownerfor a spy that records each call, then assert the exact list, its first entry, its length, or that it is empty. The spy records calls from every thread, so a lockdown on another thread lands in the list and fails a test whose code is fine. #33300 was this failure on macOS CI (475797 bytes); #33394 fixed the probes it named and left these spies to #33379.Why it matters
Any backend PR can hit one of these reds, pay a rerun and sit blocked on PR Readiness for a defect that is not in its diff. Nineteen tests in fourteen files carry the same exposure.
Not a goal
What changed (motivation → approach → change)
The SEL writer is a daemon thread (
sel-writer) that, on every append, creates its directory, re-locks its non-empty audit log (src/kiro_crew/sel_append.py:255) and opens it withos.fdopen. When it flushes events an earlier test queued while a test's spy is installed, its call is recorded beside the test's own and breaks an exact-list, first-entry, count or empty check.Each affected spy now records only calls made on the test's own thread, the shape #33394 gave the lockdown-order probes. The writers under test lock down on that thread, so each assertion keeps its strength. Two writers run on worker threads (the offloaded config write, and the revoked-nonce store in its stuck-write test), so those spies act only on files in the test's own directory instead. Where a spy shares its recorded order with a
Path.mkdiroros.fdopenspy that the SEL writer also reaches, that spy is scoped the same way.test/test_atomic_write_capabilities.py,test/test_weixin_qr.py,test/test_handlers_messaging_coverage_channels.py,test/test_cli_setup.py,test/test_computer_use_capture.py,test/test_computer_use_capture_windows.py,test/test_cron_script_more_coverage.py(fixturescript_run)test/test_config_rmw_preserves_settings.py(classify-first test with itsPath.mkdirspy, local-volume and synchronous-caller tests),test/test_token_auth_session_lifecycle.py(lockdown-size test)Truetest/test_platform_compat_coverage_filesystem.py,test/test_platform_compat_owner_only.py(wrong == []),test/test_lockdown_before_publish.py(shared helper, two tests)test/test_vector_memory.pyrestrict:tmpbefore the firstfdopentest/test_session_image_repair.py, with itsos.fdopenspyis_revokedcheck, where the writer runs on a worker thread (spy acts only on files intmp_path)test/test_config_rmw_preserves_settings.py, the stuck-write test intest/test_token_auth_session_lifecycle.pyBackwards compatibility
Compatible: test-only change; no product code touched.
Removes nothing: every spy still records each lockdown the code under test makes; only other threads' calls are left out, or, for the two worker-thread writers, calls on files outside
tmp_path.Tests
No test is added. The changed tests keep every assertion they had.
Determinism proof, on Linux, parent
80cf3dfc64e8and this change on it (b9165612e818, the same patch as this head). The 19 changed tests run with two unchanged spies as negative controls:test_memory_backup.py's path-filtered stage spy andcode_review_sage's store spy, which patches a binding the SEL writer never calls.-n0sel-writerdoes the SEL flush (mkdir,os.open,restrict_to_owner,os.fdopen, append) on a 4 KB log right after each spy is installed, after eachtempfile.mkstempand after eachos.replace; eachRevokedNonceStore.revokestarts 0.3 s late-n0-n 4 --dist loadgroup --max-worker-restart=2, withtest/test_aws_consent.pyandtest/test_webhooks_store.pyas co-tenants-n 4 --dist loadgroupThe forced condition is a throwaway pytest plugin, not committed.
Manual verification
N/A — test-only change; the forced-condition runs under Tests are the verification.
Real-machine verification
Machine: EC2 m7i.8xlarge (the author's Linux development host), Amazon Linux 2023 x86_64, 32 vCPU / 123 GiB
Before (main 80cf3df):
python -m pytest -n0 -p <the forced-flush plugin under Tests> <the 19 changed tests, the 2 controls>-> 19 failed (every changed test), 2 passed (the controls)After (head b916561): the same command -> 21 passed; 20 repeats at
-n0and 10 at-n 4 --dist loadgroup-> 0 failuresAfter the re-sync (this patch on main 5672158; EC2 m7i.2xlarge, Ubuntu 24.04 x86_64, 8 vCPU / 32 GiB):
python -m pytest -n 4 --dist loadgroup --max-worker-restart=2 <the 14 changed files>-> 986 passed, 6 skippedAfter the rebase (this patch on main 3b772f5, head ab27757; the author's development host above):
python -m pytest -n 4 --dist loadgroup --max-worker-restart=2 <the 14 changed files>-> 986 passed, 6 skippedCleanup: own development host for the forced runs; the m7i.2xlarge instance is terminated (describe returns nothing), with no key pair and no inbound rules
Each re-sync since b916561 moved only the base: the patch adds and removes the same lines.
Screenshots / video
N/A — test-only change, no UI.
Related Issues
Closes #33379
Sibling sweep
Swept: git grep -nE '"restrict_to_owner"|restrict_to_owner",|platform_compat.restrict_to_owner"' -- test/ src/kiro_crew/apps, then each replacement function and the asserts that read what it records
Sibling: test/test_memory_backup.py:329 -- not affected -- records only this test's own backup stages, by path
Sibling: test/test_memory_backup.py:1093 -- not affected -- keeps only .tmp/.partial files in the backup dir
Sibling: test/test_member_memory_backup.py:133 -- not affected -- keeps only .tmp/.partial files in the backup dir
Sibling: test/test_live_target.py:497 -- not affected -- keeps only calls under live-target-staging
Sibling: test/test_mcp_gateway_apps_spool.py:226 -- not affected -- keeps only calls under the spool temp dir
Sibling: test/test_sel_edge_cases.py:117 -- not affected -- keeps only files in the trust dir
Sibling: test/test_mcp_gateway_rewriter.py:883 -- not affected -- keeps only calls under the stubs and overlay dirs
Sibling: test/test_mcp_gateway_rewriter.py:977 -- not affected -- reads only the agents dir's own sizes
Sibling: test/test_snapshot.py:1294 -- not affected -- keeps only calls under the snapshot output dir
Sibling: test/test_snapshot.py:1366 -- not affected -- asserts one named path is absent
Sibling: test/test_beacon.py:1062 -- not affected -- asserts membership of the config path only
Sibling: test/test_mcp_gateway_oversize.py:155 -- not affected -- asserts one named path is absent
Sibling: test/test_mcp_gateway_prewarm.py:101 -- not affected -- asserts the spy saw at least one call
Sibling: test/test_discord_sessions_binding_state.py:633 -- not affected -- asserts the spy saw at least one call
Sibling: test/test_webhooks_store.py:664 -- not affected -- asserts membership of named calls only
Sibling: test/test_vector_memory.py:1148 -- not affected -- asserts membership of the WAL path only
Sibling: test/test_vector_memory.py:1177 -- not affected -- asserts membership of the FAISS path only
Sibling: test/test_vector_memory.py:1226 -- not affected -- compares the positions of restrict:mem.db and connect only
Sibling: test/test_vector_memory.py:1278 -- not affected -- keeps only paths that did not exist yet
Sibling: test/test_memory_index_owner_only.py:41 -- not affected -- patches the module's own binding, which the SEL writer never calls
Sibling: test/test_secrets_vault.py:202 -- not affected -- patches the vault's own binding, which the SEL writer never calls
Sibling: src/kiro_crew/apps/builtins/code_review_sage/tests/test_store_selfheal.py:307 -- not affected -- patches the app store's own restrict_to_owner, which the SEL writer never calls
Sibling: test/test_sel_log_lockdown.py:59 -- not affected -- _on_event_loop is patched process-wide, so every append in the test skips the lockdown
Sibling: test/test_acp_frame_record.py:1154 -- not affected -- a call on another thread raises in that thread, never in this test; the SEL writer is restarted on its next use
Sibling: test/test_cli_logging.py:1308 -- not affected -- raises only, and the test keeps only its own loggers' records
Sibling: test/test_mcp_gateway_rewriter_fingerprint_keep.py:653 -- not affected -- refuses only one named agent file
Sibling: test/test_workflows_run_identity.py:427 -- not affected -- refuses only the one target path
Sibling: test/test_atomic_write_capabilities.py:179 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_live_target.py:671 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_session_image_repair.py:836 -- not affected -- raises only; the asserts read this test's own descriptors
Sibling: test/test_snapshot.py:1309 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_pod_env.py:298 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_denied_commands_api.py:1067 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_computer_use_snapshot.py:499 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_webhooks_store.py:671 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_memory_backup.py:829 -- not affected -- raises only; the asserts read this test's own files
Sibling: test/test_mcp_gateway_manager_coverage.py:317 -- not affected -- raises only; the asserts check membership in the log text
Sibling: test/test_memory_index_owner_only.py:66 -- not affected -- patches the module's own binding, which the SEL writer never calls
Sibling: test/test_aws_consent.py:95 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_kiro_prerequisite.py:134 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_tips.py:927 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_webhooks_store.py:515 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_workflows_store_lockdown.py:39 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/test_session_transfer_layer_b.py:1491 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/tests/test_policy_store.py:696 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: src/kiro_crew/apps/builtins/ops_mission_control/tests/test_security.py:337 -- not affected -- already keeps only the test thread's calls (#33394)
Sibling: test/conftest.py:872 -- not affected -- a no-op replacement that records nothing
Sibling: test/md_notebook_helpers.py:184 -- not affected -- restores the real function
Sibling: test/test_md_notebook_pat_read_closes.py:44 -- not affected -- a no-op on the store's own binding
Sibling: test/test_secret_file_owner_dacl.py:27 -- tracked in #37631 -- the
seenfixture records every thread's calls under exact-list, count and empty asserts; the file landed on main after these sites were sweptSibling: test/test_secret_file_owner_dacl.py:130 -- not affected -- raises only; the asserts read this test's own file and log text
Sibling: test/test_secret_file_owner_dacl.py:142 -- not affected -- raises only; the assert reads the helper's own return value
Pattern harvest
Rule candidate: review-prompt
Pattern: "a test patches a process-global and asserts on the order or count of its calls without keeping only the test thread's calls"
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsContribution License Agreement
Contributed under the project's Apache-2.0 license (CONTRIBUTING.md, Licensing).