Repository navigation
test: close two home-resolving leaks behind the dashboard_origin flake - #20593
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the two leakers are fixed where they leak, and the new floor names the next one. Two test-side items are larger than they need to be. Notes
What this change shipsInventory (8 items) — 6 justifiedIntent: stop an unrelated dashboard test from failing at teardown on PR branches, because two other tests resolve the real data home from background threads. FIX.
[FIRST-PRINCIPLES-REVIEWED] 05c97dc Description read[DESCRIPTION-READ] c8dd5ec3b28c50ee7043b3b5d356c6028e71351c2b87f9a03501247021a5fcf6 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: each leak is closed where it starts, the shared teardown now names the next leaker, and the only production change moves the [DESIGN-REVIEWED] 05c97dc Description read[DESCRIPTION-READ] d4cc5e4974fc2873c4474ddd56a86a693cf17743f9f6157c137afdabe1f4578b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
ef17ea5 to
4e7f54d
Compare
cruisercohen
left a comment
There was a problem hiding this comment.
Requesting changes on 4e7f54d for two small test-side items. The fix itself checks out.
What I verified locally (this head, merge base 10e53cc)
- Reproduced the flake on the merge base. A throwaway plugin sets
embeddings._EVICT_POLL_MAX_SECS = 0.01, then the run istest_custom_embedding_model.py::TestLoadTimeDimValidation::test_matching_dim_loadsfollowed bytest_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startup(-p no:randomly -n0). On the merge base, 5 of 5 runs error at teardown with the real-data-home assertion. On this head, 5 of 5 pass. - The root cause is fixed without the victim change. This head with the merge base's
test_dashboard_origin.py(the old cleared-env victim) passes 3 of 3. This head with the merge base'stest_custom_embedding_model.py(the old leaker) failstest_matching_dim_loadsby name through the new_no_leaked_embed_evictorfloor. - The home guard is not weakened.
conftest.pyonly adds code, and on this head a probe test that clearsos.environand callsconfig_dir()still errors with the same teardown assertion. test_a_debounced_write_resolves_no_path_when_it_wakesfails when onlysession_tree_projection.pyis reverted to the merge base, and passes on this head. The_schedule_checkpoint/_debounced_writechange is correct:rootis resolved under the lock besidetarget, the one caller passes it, andself._rootstill wins for a seeded projection.- The six changed test files pass (511 tests) in 3 of 3 runs at
-n 4.check_comment_history.pypasses, and every check on this head is green.
Requested changes
-
test/test_embeddings.py:1122:evictor.join(timeout=5)in_close_built_embedders. If that join gives up while the evictor is still alive, the rootdir_no_leaked_embed_evictorfloor tears down after this class fixture and fails the test. So the 5 s bound adds a failure that depends on host load. Repro: a probe test inTestLlamaCppEmbedderpins_INFER_STOP_TIMEOUT_SECSto 0 and makes oneembedding_idle_evict_secstick take 6 s. It errors at teardown with "left its idle evictor running". Withtimeout=30the same probe passes. Please bound it at 30 s, the default_INFER_STOP_TIMEOUT_SECS. -
Some added docstrings describe the old code or the incident instead of the current behaviour. Fast Gate's history pattern lets them through, but they will read as stale once this merges:
test/test_dashboard_origin.py:279: "The earlier shape, apatch.dictthat CLEAREDos.environon every test, ...", and :285 "nine PR branches in a week". Suggest present tense, for example "Clearingos.environwould also drop the floor'sKIROCREW_HOMEpin for the test's duration ...".conftest.py:2394-2399: the "MEASURED on a Linux shard" sentence and "nine PR branches in a week".test/test_crew_log_session_tree_projection.py:497-500: "Resolving_current_root()there read the next test's home ... the teardown guard blamed ..." describes the old bug. Suggest saying what resolving at wake time would do.
The measurement already lives in the PR body and in the testing-conventions entry.
Non-blocking
reset_for_testsinsession_tree_projection.py(line 2070, not changed here) still says a pending write "resolves its target directory when it WAKES". With this change the worker resolves nothing, so that sentence is now wrong. Worth fixing while you are in the file.- Overlap with #20024: it edits the same test in
test/test_dashboard_origin.py, pinning_resolve_default_homeand joining the lookup thread. This PR removes the leakers and keeps theKIROCREW_HOMEpin, so #20024's home pin is not needed for this flake, and the two PRs will conflict textually. Once this lands, #20024 is either superseded or reduces to its lookup-thread join, which belongs to #20022.
4e7f54d to
c393baa
Compare
|
Addressed on c393baa (single commit, amended). Thanks for the local verification. 1. 2. Docstrings in past tense / incident narration. All three rewritten to describe current behaviour:
Non-blocking, taken: #20024: agreed on the overlap; this PR keeps the pin and removes the leakers, so #20024's Local on this head: the four touched test files 412 passed at |
cruisercohen
left a comment
There was a problem hiding this comment.
Re-review of c393baa85c (same merge base 10e53cc119; the interdiff against 4e7f54d8b8 is 5 files, +22/-23). Both requested changes are addressed and I found nothing blocking.
1. Evictor join bound: met. test/test_embeddings.py:1124 now joins with timeout=30, which matches _INFER_STOP_TIMEOUT_SECS = 30.0 in embeddings.py:230. The fixture docstring (:1111-1116) explains why. I re-ran the probe: a subclass of TestLlamaCppEmbedder pins _INFER_STOP_TIMEOUT_SECS to 0 and makes the evictor's second embedding_idle_evict_secs tick take 7 s while the test closes. It passes on this head. With the join set back to 5 s, the same probe errors at teardown with "left its idle evictor running".
2. Docstrings: met. All three now describe current behaviour:
test/test_dashboard_origin.py_not_over_ssh: "Clearingos.environwould also drop ...". The earlier-shape sentence and the branch count are gone.conftest.py_no_leaked_embed_evictor: the MEASURED sentence is gone. It now points to the testing-conventions entry for the measurement.test/test_crew_log_session_tree_projection.pytest_a_debounced_write_resolves_no_path_when_it_wakes: says what resolving at wake time would do.
The non-blocking reset_for_tests sentence (session_tree_projection.py:2066-2074) now matches the code, since the target is pinned when the write is armed. That edit is docstring-only; the production change in that file is otherwise unchanged from the head I reviewed before. I found no other added comment or docstring that narrates history, and check_comment_history.py against the merge base passes.
The fix itself, re-checked on this head.
- With a throwaway plugin that sets
embeddings._EVICT_POLL_MAX_SECS = 0.01, I rantest_custom_embedding_model.py::TestLoadTimeDimValidation::test_matching_dim_loadsfollowed bytest_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startup(-p no:randomly -n0). It passes 3 of 3 on this head and errors at teardown 2 of 2 on the merge base. - The six changed test files: 511 passed.
CI. The only red test is test_acp_bindings.py (approve_tool shape and the frozen member inventory; Backend Tests 3.12 shard 5 and Windows shard 5), and Coverage Gate and PR Readiness fail because of it. This diff does not touch that test. It was red on main until #20902, which merged a few minutes after this head's CI run started, so the run's merge ref does not include the fix. A rerun reuses that merge ref; rebasing onto current main should clear it. Every AI and security lane on this head passes.
Non-blocking. The body says Fixes #20022 as well as Fixes #19514, so merging this closes #20022. Your reply says the lookup-thread join in #20024 still stands on its own under #20022. If that join is still wanted, either drop Fixes #20022 here or give the join its own issue, so it does not lose its tracking issue when this merges.
I am holding approval only because CI is not green on this head. I expect to approve once a rebased head is green.
c393baa to
8df7b16
Compare
cruisercohen
left a comment
There was a problem hiding this comment.
Approving on 8df7b16. This head is a rebase of c393baa onto current main with no code change: same single commit, same nine files, identical added and removed lines.
Both earlier asks are met here:
- The idle-evictor join in
_close_built_embeddersnow waits 30 seconds, matching the embedder's own stop timeout, so a slow in-flight tick does not trip the leak check at teardown. - The added docstrings in the dashboard-origin, conftest and session-tree-projection tests describe current behaviour in present tense.
Required checks are green on this head.
|
HOLD at 8df7b16. The fix looks sound. What still blocks approval: two current-head AI items have no reply.
Also, please settle the tracking for #20022. This PR's CI on this head is green. I read the code and tests but did not run them. — crew-pr-reviewer (r105) |
test_dashboard_origin.py's hanging-lookup test erred at teardown on nine PR branches in a week: the home guard found the operator's real ~/.kiro/crew resolved. The test resolves no path itself. Two background resolvers left by earlier tests on the worker did, and the victim's patch.dict(clear=True) window was where their tick landed with no KIROCREW_HOME to pin it. Leaker one: a LlamaCppEmbedder whose fake model was published and never closed keeps a kc-embed-evict daemon that re-reads the memory config through config_dir() on every tick. Twenty tests in four files built one and walked away; each now closes it. Leaker two: SessionTreeProjection's debounced checkpoint write built its payload root with a live _current_root() when it WOKE on the mc-maint pool, a debounce after the arming test had ended. The root is now pinned beside the target when the write is armed, so the worker resolves nothing. The floor catches the next one by name: _no_leaked_embed_evictor fails the test that leaves an evictor running and stops it, and the teardown hook cancels pending checkpoint writes before the pins lift. The victim keeps the pin, unsetting only the two SSH variables its subject reads.
8df7b16 to
05c97dc
Compare
|
HOLD at 05c97dc. The fix itself is sound. The items from our r105 HOLD still have no reply:
To clear: one reply per item (a fix or an evidence-backed rebuttal). I read the code and tests but did not run them. — crew-pr-reviewer (r124) |
|
@CrysisDeu iamwhatever's r124 hold on 05c97dc lists four items that still have no reply. One reply per item (a fix or an evidence-backed rebuttal) clears it:
|
Problem / Motivation
Goal:
test_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startupstops erring at teardown on PR branches that do not touch it.The Backend Tests (3.12, shard 4) job errs on unrelated PRs with:
Nine distinct PR branches in seven days (example: run 38215481814). The test resolves no path itself: it formats three URL lines with the hostname lookup patched. Something else on the worker resolved the data home while this test had
os.environcleared.Why it matters
Every PR runs this shard. A random red costs a rerun and hides real regressions in the same job. The guard that fires is correct -- the operator's real
~/.kiro/crewWAS resolved and created on the runner -- but it blames the test that happened to be running, so the two tests actually leaking passed every time and nobody could fix them from the failure alone.Not a goal
format_dashboard_urlsdoes or to the home guard's verdict.patch.dict("os.environ", {}, clear=True): each is a potential victim of the SAME class, and the guard names them when they are; the two leakers found here are closed, and the new floor names the next leaker.test_boot_smoke_default_modewitness (closed by test(integration): list atomic_write umask cache as known boot change #19168) or to thetest_tmpdir_helpersprefix ratchet (its red named a branch-onlymkdtemp(dir="/tmp")that is already gone from that branch's head); both were in the same CI mine and neither is an order leak on main.What changed (motivation → approach → change)
Finding the leakers. A throwaway plugin (not committed) wrapped
config_dir()and_resolve_default_home()to log every call made off the main thread, tagged with the test that started the thread or queued the pool job, and the test running when the call landed. One run of CI's shard 4 (22,628 tests,-n 6) found two sources, both resolving the home from under a later test:kc-embed-evictthreads born intest_custom_embedding_model.py, landing intest_outbound_files,test_telegram_parity,test_options_cap_contractandtest_subagent_stall_activity_scope.LlamaCppEmbedderstarts a daemon idle evictor when a model is published, and_evict_loopcallsembedding_idle_evict_secs()->_read_memory_config()->config_path()->config_dir()on every tick. A test that loads a fake model and neverclose()s the backend leaves that poller running for the rest of the worker. Twenty tests in four files did (test_embeddings.pyx14,test_custom_embedding_model.pyx3,test_embed_scores_buffer.pyx2,test_embedding_model_apply.pyx1).mc-maintpool jobs fromtest_crew_log_emit.pyandtest_crew_log_edge_fanout.py:SessionTreeProjection._debounced_writepinned its target path when armed but built the payload's"root"with a live_current_root()when it WOKE, a one-second debounce later, in whichever test was running then.Why the victim is this test. Under the per-test home a stray resolution lands in some other test's home and nothing notices.
TestFormatDashboardUrlsclearedos.environon every test, so for its duration there was noKIROCREW_HOMEat all; the hanging-lookup test holds that window open for 200 ms (the bounded join), the widest in its file, so a tick inside it fell through to the real home and the guard fired on it.The change, three parts, each needed on its own:
session_tree_projection._schedule_checkpointresolvesrootbesidetargetunder the lock and passes both to_debounced_write, whose payload readsself._root or root; the worker no longer resolves the data home. The twenty tests close their backend:TestLlamaCppEmbedderandTestEmbedPrioritygain a class autouse fixture that owns every backend their helper builds (and joins the evictor, since the straggler test pins the close timeout to zero and the sharedmonkeypatchunwinds after the fixture); the inline sites usetry/finally: close().conftest.pygains_no_leaked_embed_evictor, attributed by difference and stopping the thread through its owner like_no_leaked_telemetry_exporter, andpytest_runtest_teardowncancels pending checkpoint writes (_cancel_session_tree_checkpoints->session_tree_projection.reset_for_tests) after the fold worker is retired and before the pins lift.TestFormatDashboardUrlsreplacespatch.dict("os.environ", {}, clear=True)with an autouse fixture that unsets onlySSH_CONNECTIONandSSH_CLIENT, the two variablesformat_dashboard_urlsreads;TestAllowedLoopbackPortsEnvsets or unsets its one variable the same way. The testing conventions already say "keepKIROCREW_HOMEin the cleared environment".docs/system-specs/common/testing-conventions.mdgains the two floor members and a class-4 entry recording the measurement and the three-part fix.Backwards compatibility
Compatible:
_debounced_writegains a third optional parameter with a default; the only caller is_schedule_checkpoint(plus one test calling it directly, which still passes two arguments in the existing root-change test). The checkpoint file'srootis the same string as before for a seeded projection and the arm-time root for an unseeded one, which_load_checkpointalready validates against the live root.Removes nothing: the diff adds a parameter, two floor fixtures, closes in tests and a doc section.
Tests
Changed:
test/test_crew_log_session_tree_projection.pygainstest_a_debounced_write_resolves_no_path_when_it_wakes(arms a write on an unseeded projection through a recording pool, asserts the root was pinned at arm time, then fires the worker with_current_rootreplaced by a raiser; red onorigin/main--_debounced_writetakes two arguments there and the payload resolves live -- green here).test/test_dashboard_origin.py,test/test_embeddings.py,test/test_custom_embedding_model.py,test/test_embed_scores_buffer.py,test/test_embedding_model_apply.pyas above.Forced condition (throwaway
-pplugin shrinkingembeddings._EVICT_POLL_MAX_SECSto 10 ms so a leaked evictor ticks continuously; leaker then victim,-p no:randomly -n0):origin/main@ 10e53cc)test_custom_embedding_model.py::TestLoadTimeDimValidation::test_matching_dim_loadsthentest_dashboard_origin.py::...::test_a_hanging_hostname_lookup_cannot_stall_startupconftest.pyfloor and the old teststest_matching_dim_loadsfails by name (left its idle evictor running)Determinism proof on the six changed test files (511 tests):
-n0-n 4 --dist loadgroup --max-worker-restart=2, withtest_crew_log_emit.pyandtest_outbound_files.pyas co-tenants (934 tests)-n0python -m pytest -n0 test/test_flake_pattern_ratchet.pyLlamaCppEmbedderorreset_for_tests,-n 10test_native_skill_projection.py::test_launch_record_resources_match_the_alias_file_kiro_cli_loads(fails identically onorigin/mainon this host: its tmp root sits under~/.kiro/crew/scratch) andTestBundledLinuxX86CpuGate::test_operator_override_is_kept_out_of_the_bundled_shape_rule(host hasLLAMA_CPP_LIB_PATHexported; passes 3/3 when its file runs at-n 4)Whole
test/tree with the new floor,-n 14on a 32-core host: 92,388 passed, 83 failed. 74 of the 83 fail identically onorigin/mainon this host (its scratch root sits under~/.kiro/crew,setfaclis unavailable, the sandbox userns probe answers EPERM); the other 9 are load timeouts (test_installer_distro,test_issue_radar_crew_*) that pass 2/2 when rerun together. Zero occurrences of the new guard's message or of the home guard's across the run.Gates:
isort,flake8,black,mypy src/kiro_crew/(2382 files, clean),scripts/check_comment_history.pyagainstorigin/main(clean),docs-lint.sh,check_sync_io_in_async,check_loop_bound_locks,check_testpaths_coverage,check_brand_name,check_subprocess_encoding. Frontend gates not run: the diff touches no frontend file.Manual verification
N/A -- the forced-condition and tripwire runs above are the verification; no UI or external service is involved.
Related Issues
Fixes #20022
Fixes #19514
Pattern harvest
Rule candidate: review-prompt
Pattern: a background thread or pool job that resolves
config_dir()/data_home()when it RUNS rather than when it was started or queued (a poller re-reading config, a debounced writer building its payload late); and a test thatpatch.dict("os.environ", {}, clear=True)instead of unsetting the variables its subject reads.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.