Skip to content

test(apps): move the settle-window clock only on the listing's own walk - #39010

Merged
bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/apps-walk-cache-flaky
Nov 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/LinuxFlackyGuard/apps-walk-cache-flaky

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: TestListingReusesTheWalk::test_an_unchanged_folder_is_not_walked_again passes whatever ran before it on the xdist worker.

test/test_app_manager_installed_record.py::TestListingReusesTheWalk::test_an_unchanged_folder_is_not_walked_again failed with assert 2 == 1 (the apps folder was "walked" twice) on Backend Tests (3.12, 8) for five open PRs: #38902, #38891, #38803, #37604, #33979.

Why it matters

Shard 8 holds this file, so every PR whose shard 8 put this test after the wrong neighbour went red with no relation to its own diff.

Not a goal

What changed (motivation → approach → change)

Symptom: the spy recorded two scans of the apps folder for two list_apps() calls on an unchanged folder.

Root cause: the spy did monkeypatch.setattr(manager_records.os, "scandir", counting). manager_records.os is the stdlib os module, so the spy counted every os.scandir of that folder in the worker, from every module and thread. list_apps() caused the second scan itself. When detect_orphaned_builtins had no cached result, list_apps validated the built-in manifests. The path gate then listed the home's credential masks on the shared mc-pathres pool (kiro_crew.apps.credential_masks._installed_manifests, which runs its own os.scandir(apps)). The second scan's stack is in credential_masks.py, not in manager_records.py, so the product cache behaved correctly.

No thread leaks across tests. mc-pathres is the process-wide shared executor, and this test's own list_apps() call submitted the listing. Whether the orphan cache was cold depended on the tests that ran before on the worker. The CI progress artifacts of all four failing shard-8 jobs show the same order: test/test_app_backend.py::TestBootSpawnLatency, then a test_app_backend_stale_reap.py test, then this test, within about 0.2 s.

Change: the unchanged-folder counter itself was fixed on main by #38980 (it counts only calls whose caller frame is _recorded_app_names). This PR, rebased on it, covers the rest of the class:

  • test_a_scan_started_inside_the_settle_window_is_not_reused had the same spy shape: it advanced its ManualClock on every os.scandir of the apps folder, from any module or thread. It now moves the clock only when the caller frame is _recorded_app_names, the same filter test(apps): count only the listing walk in the unchanged-folder test #38980 uses.
  • docs/system-specs/common/testing-conventions.md gets a new rule: a counting spy counts only the calls the code under test makes. It shows the wrong form and the caller-frame form. It also gives the module-binding form, and says when test/test_facade_mirror_guards.py rules that form out (os is a name the kiro_crew.apps.manager facade mirrors). The rule carries the numbers below.
  • AUTOSDE.yaml item 11 now flags a newly added counting or recording spy through <mod>.os.<name> that counts every caller, the same way it flags a rebound stdlib clock.

Only tests and docs/spec files change. No product code, no CI config.

Backwards compatibility

Compatible: only a test, a convention doc and a review rule change. No product behaviour moves.

Removes nothing: the diff only adds a doc rule and a review-rule clause, and narrows which calls a test spy counts.

Tests

Proof: not provable -- test-only change

Evidence on a Linux DevSpace with 8 cores, 30 runs each. "After" is the caller-frame filter on both spies (the same filter #38980 uses for the counter):

Run Before After
TestBootSpawnLatency + test_reap_skips_recycled_pid + this class, -n0 0/30 pass 30/30 pass
6 CPU spinners + -n 8 -p randomly, test_app_backend.py + test_app_backend_stale_reap.py + this file 21/30 pass (9 failed) 30/30 pass

On a second host (aarch64), with 16 CPU spinners, -n 8 -p randomly and the same files, this test failed 14/30 before the change and passed 30/30 after. The -n0 order failed 30/30 before and passed 30/30 after. TestBackendLifecycle (4 tests) fails on that host before and after, because it has no sandbox backend (SandboxUnavailableError). That failure is unrelated.

This file plus the 69 test files that read testing-conventions.md or AUTOSDE.yaml: 3059 passed, 54 skipped.

Manual verification

N/A — the repro runs above are the verification. Local gpt and opus reviews of the change: PASS, no findings.

Related Issues

N/A — found by the Linux flaky-test scan across the five PRs above.

Sibling sweep

Swept: git grep -nE 'setattr([A-Za-z_.]*os, "scandir"' -- test/
Sibling: test/scale_budget/conftest.py -- tracked in #39021
Sibling: test/scale_budget/test_listing_budget.py -- tracked in #39021
Sibling: test/test_agent_scratch.py -- tracked in #39021
Sibling: test/test_agents_dir_fingerprint_skips_aliases.py -- tracked in #39021
Sibling: test/test_agents_dir_memo.py -- tracked in #39021
Sibling: test/test_app_command_agent_guard.py -- tracked in #39021
Sibling: test/test_backend_tmp.py -- tracked in #39021
Sibling: test/test_chat_folder_steering_dirs.py -- tracked in #39021
Sibling: test/test_config_app_owned_sections.py -- tracked in #39021
Sibling: test/test_crew_log_fullscan_guard.py -- tracked in #39021
Sibling: test/test_crewmate_prune_migration.py -- tracked in #39021
Sibling: test/test_dev_fleet_disk_measure_without_du.py -- tracked in #39021
Sibling: test/test_dev_fleet_npm_preflight.py -- tracked in #39021
Sibling: test/test_display_verdicts.py -- tracked in #39021
Sibling: test/test_folder_steering.py -- tracked in #39021
Sibling: test/test_history_listing_index.py -- tracked in #39021
Sibling: test/test_ledger_sweep.py -- tracked in #39021
Sibling: test/test_managed_agent_shadow.py -- tracked in #39021
Sibling: test/test_markdown_agent_specs.py -- tracked in #39021
Sibling: test/test_mate_config_migration.py -- tracked in #39021
Sibling: test/test_mcp_apps_render_claim_recovery.py -- tracked in #39021
Sibling: test/test_mcp_command_casing_shared.py -- tracked in #39021
Sibling: test/test_mcp_gateway_command_casing.py -- tracked in #39021
Sibling: test/test_member_essential_context.py -- tracked in #39021
Sibling: test/test_member_essential_context_left_out_sources.py -- tracked in #39021
Sibling: test/test_member_essential_context_resource_admission.py -- tracked in #39021
Sibling: test/test_native_skill_projection_lease_scan.py -- tracked in #39021
Sibling: test/test_native_skill_projection_scan_bounds.py -- tracked in #39021
Sibling: test/test_owner_only_files.py -- tracked in #39021
Sibling: test/test_pre_fix_fork_carrier_repair.py -- tracked in #39021
Sibling: test/test_project_scan_walker.py -- tracked in #39021
Sibling: test/test_project_tree.py -- tracked in #39021
Sibling: test/test_registry_store_art_prewarm_rewarm.py -- tracked in #39021
Sibling: test/test_sandbox_governance_mask_model_loaders.py -- tracked in #39021
Sibling: test/test_session_storage_restore.py -- tracked in #39021
Sibling: test/test_side_readonly_spec_reap.py -- tracked in #39021
Sibling: test/test_skill_mapping_and_lazy_default.py -- tracked in #39021
Sibling: test/test_skill_pending_api.py -- tracked in #39021
Sibling: test/test_skill_trust_store_project_skill_loading.py -- tracked in #39021
Sibling: test/test_skills_cold_start.py -- tracked in #39021
Sibling: test/test_snapshot_unreadable_entries.py -- tracked in #39021
Sibling: test/test_spawn_parent_available_agents.py -- tracked in #39021
Sibling: test/test_subagent_panel_durable_replay.py -- tracked in #39021
Sibling: test/test_trash_pinned_sibling_removal.py -- tracked in #39021

Pattern harvest

Rule candidate: review-prompt
Pattern: counting or recording spy set through <mod>.os.<name> patches the stdlib for the whole worker

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 24ef86c374c185a1574dfcab8aff1c0895385d19 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 24ef86c374c185a1574dfcab8aff1c0895385d19 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 24ef86c374c185a1574dfcab8aff1c0895385d19 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This only changes tests, docs and a review rule. The spy now uses the same caller-frame filter as #38980, which removes the cross-test coupling at its source.

Notes

  • The spy relies on _recorded_app_names calling os.scandir directly (manager_records.py:135). If that call is ever moved into a helper, the spy will silently stop advancing the clock, and the settle-window test will then pass without testing anything. A guard that fails when no scan was matched would make that refactor show up as a failure instead.

[DESIGN-REVIEWED] 24ef86c

Description read

[DESCRIPTION-READ] 0152121201f97d9dd5f8a6f29c2d98cadb68b980938979f8a6d959b1ca721c38

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 24ef86c374c185a1574dfcab8aff1c0895385d19 — this comment is updated in place on each push.

Review details

No findings. The one candidate is only a difference in wording between the AUTOSDE rule text and the conventions doc. It does not change what any code does at runtime. The test change works: _recorded_app_names calls os.scandir(root) directly, so it is the frame the spy checks, and sys is already imported in the test module.

[OPUS-REVIEWED] 24ef86c

Verdict parsed from the review's SHA-scoped output markers for commit 24ef86c374c185a1574dfcab8aff1c0895385d19.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 24ef86c374c185a1574dfcab8aff1c0895385d19 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 24ef86c374c185a1574dfcab8aff1c0895385d19 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 24ef86c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 24ef86c374c185a1574dfcab8aff1c0895385d19 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/apps-walk-cache-flaky branch from 76516af to 4fd2b72 Compare November 18, 2026 06:19
@bolichen97
bolichen97 marked this pull request as ready for review November 18, 2026 06:19
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 18, 2026 06:19
@bolichen97
bolichen97 requested a review from CrysisDeu November 18, 2026 06:19
@bolichen97
bolichen97 enabled auto-merge (squash) November 18, 2026 06:19
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/apps-walk-cache-flaky branch from 4fd2b72 to 6392928 Compare November 18, 2026 07:27
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 18, 2026
The settle-window test advanced its clock on every os.scandir of the apps
folder, from any module or thread, because manager_records.os is the
shared os module. Filter it by caller frame, as the unchanged-folder test
does, and write the rule for counting and recording spies into the
testing conventions and the AUTOSDE review list.
@bolichen97
bolichen97 force-pushed the crew/LinuxFlackyGuard/apps-walk-cache-flaky branch from 6392928 to 24ef86c Compare November 18, 2026 08:24
@bolichen97 bolichen97 changed the title test(apps): count only list_apps' own scans of the apps folder test(apps): move the settle-window clock only on the listing's own walk Nov 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Main already fixed the unchanged-folder counter in #38980, using the same caller-frame filter. I rebased this PR onto it. What is left here: the same filter on the settle-window test's clock spy, plus the convention and AUTOSDE rule that keep the class from coming back.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 18, 2026
@bolichen97
bolichen97 merged commit 8773ddf into main Nov 18, 2026
100 of 105 checks passed
@bolichen97
bolichen97 deleted the crew/LinuxFlackyGuard/apps-walk-cache-flaky branch November 18, 2026 09:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 18, 2026
@bolichen97

bolichen97 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

实机验证 (real-machine verification)

  • Machine: existing DevSpace ds-linux-flaky-guard-trial-n8evipa4 (Linux, 8 cores), worktree ~/w-walkcache/repo. No new machine was created.
  • Before (the test file as it was on main before the fix, same commands as after):
    • TestBootSpawnLatency + test_reap_skips_recycled_pid + TestListingReusesTheWalk, -n0 -p no:randomly: 0/30 passed. Every run failed with assert 2 == 1.
    • 6 CPU spinners + -n 8 -p randomly, test_app_backend.py + test_app_backend_stale_reap.py + the test file: 21/30 passed (9 failed with assert 2 == 1).
  • After (caller-frame filter, branch head before the final rebase onto test(apps): count only the listing walk in the unchanged-folder test #38980, same machine and commands):
    • -n0 order: 30/30 passed.
    • spinners + -n 8 -p randomly: 30/30 passed.
  • Cleanup: nothing to delete, because the machine already existed and is shared. The CPU spinners were bounded by timeout and have exited.
  • After, as merged (current main d80be9d7b6, which contains the squash of this PR, 8773ddf334; same machine, same two commands):
    • -n0 order: 30/30 passed (19 passed per run).
    • 6 CPU spinners + -n 8 -p randomly: 30/30 passed (174 passed per run).
    • No TestListingReusesTheWalk test failed in any of the 60 runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant