Skip to content

test: close two home-resolving leaks behind the dashboard_origin flake - #20593

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/flake-shared-state-leaks
Oct 27, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/flake-shared-state-leaks

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startup stops erring at teardown on PR branches that do not touch it.

The Backend Tests (3.12, shard 4) job errs on unrelated PRs with:

ERROR at teardown of TestFormatDashboardUrls.test_a_hanging_hostname_lookup_cannot_stall_startup
AssertionError: this test resolved the operator's REAL default data home (/home/runner/.kiro/crew): it dropped the KIROCREW_HOME pin and let config_dir() / data_home() fall through to ~/.kiro/crew

Nine distinct PR branches in seven days (example: run 38215481814). The test resolves no path itself: it formats three URL lines with the hostname lookup patched. Something else on the worker resolved the data home while this test had os.environ cleared.

Why it matters

Every PR runs this shard. A random red costs a rerun and hides real regressions in the same job. The guard that fires is correct -- the operator's real ~/.kiro/crew WAS resolved and created on the runner -- but it blames the test that happened to be running, so the two tests actually leaking passed every time and nobody could fix them from the failure alone.

Not a goal

  • No change to what format_dashboard_urls does or to the home guard's verdict.
  • No sweep of the other ~50 tests that still patch.dict("os.environ", {}, clear=True): each is a potential victim of the SAME class, and the guard names them when they are; the two leakers found here are closed, and the new floor names the next leaker.
  • No change to the test_boot_smoke _default_mode witness (closed by test(integration): list atomic_write umask cache as known boot change #19168) or to the test_tmpdir_helpers prefix ratchet (its red named a branch-only mkdtemp(dir="/tmp") that is already gone from that branch's head); both were in the same CI mine and neither is an order leak on main.

What changed (motivation → approach → change)

Finding the leakers. A throwaway plugin (not committed) wrapped config_dir() and _resolve_default_home() to log every call made off the main thread, tagged with the test that started the thread or queued the pool job, and the test running when the call landed. One run of CI's shard 4 (22,628 tests, -n 6) found two sources, both resolving the home from under a later test:

  1. kc-embed-evict threads born in test_custom_embedding_model.py, landing in test_outbound_files, test_telegram_parity, test_options_cap_contract and test_subagent_stall_activity_scope. LlamaCppEmbedder starts a daemon idle evictor when a model is published, and _evict_loop calls embedding_idle_evict_secs() -> _read_memory_config() -> config_path() -> config_dir() on every tick. A test that loads a fake model and never close()s the backend leaves that poller running for the rest of the worker. Twenty tests in four files did (test_embeddings.py x14, test_custom_embedding_model.py x3, test_embed_scores_buffer.py x2, test_embedding_model_apply.py x1).
  2. mc-maint pool jobs from test_crew_log_emit.py and test_crew_log_edge_fanout.py: SessionTreeProjection._debounced_write pinned its target path when armed but built the payload's "root" with a live _current_root() when it WOKE, a one-second debounce later, in whichever test was running then.

Why the victim is this test. Under the per-test home a stray resolution lands in some other test's home and nothing notices. TestFormatDashboardUrls cleared os.environ on every test, so for its duration there was no KIROCREW_HOME at all; the hanging-lookup test holds that window open for 200 ms (the bounded join), the widest in its file, so a tick inside it fell through to the real home and the guard fired on it.

The change, three parts, each needed on its own:

  • Leakers close what they start. session_tree_projection._schedule_checkpoint resolves root beside target under the lock and passes both to _debounced_write, whose payload reads self._root or root; the worker no longer resolves the data home. The twenty tests close their backend: TestLlamaCppEmbedder and TestEmbedPriority gain a class autouse fixture that owns every backend their helper builds (and joins the evictor, since the straggler test pins the close timeout to zero and the shared monkeypatch unwinds after the fixture); the inline sites use try/finally: close().
  • The floor catches the next one by name. Rootdir conftest.py gains _no_leaked_embed_evictor, attributed by difference and stopping the thread through its owner like _no_leaked_telemetry_exporter, and pytest_runtest_teardown cancels pending checkpoint writes (_cancel_session_tree_checkpoints -> session_tree_projection.reset_for_tests) after the fold worker is retired and before the pins lift.
  • The victim keeps the pin. TestFormatDashboardUrls replaces patch.dict("os.environ", {}, clear=True) with an autouse fixture that unsets only SSH_CONNECTION and SSH_CLIENT, the two variables format_dashboard_urls reads; TestAllowedLoopbackPortsEnv sets or unsets its one variable the same way. The testing conventions already say "keep KIROCREW_HOME in the cleared environment".

docs/system-specs/common/testing-conventions.md gains the two floor members and a class-4 entry recording the measurement and the three-part fix.

Backwards compatibility

Compatible: _debounced_write gains a third optional parameter with a default; the only caller is _schedule_checkpoint (plus one test calling it directly, which still passes two arguments in the existing root-change test). The checkpoint file's root is the same string as before for a seeded projection and the arm-time root for an unseeded one, which _load_checkpoint already validates against the live root.

Removes nothing: the diff adds a parameter, two floor fixtures, closes in tests and a doc section.

Tests

Changed: test/test_crew_log_session_tree_projection.py gains test_a_debounced_write_resolves_no_path_when_it_wakes (arms a write on an unseeded projection through a recording pool, asserts the root was pinned at arm time, then fires the worker with _current_root replaced by a raiser; red on origin/main -- _debounced_write takes two arguments there and the payload resolves live -- green here). test/test_dashboard_origin.py, test/test_embeddings.py, test/test_custom_embedding_model.py, test/test_embed_scores_buffer.py, test/test_embedding_model_apply.py as above.

Forced condition (throwaway -p plugin shrinking embeddings._EVICT_POLL_MAX_SECS to 10 ms so a leaked evictor ticks continuously; leaker then victim, -p no:randomly -n0):

run parent (origin/main @ 10e53cc) this fix
test_custom_embedding_model.py::TestLoadTimeDimValidation::test_matching_dim_loads then test_dashboard_origin.py::...::test_a_hanging_hostname_lookup_cannot_stall_startup 2 passed, 1 error: the exact teardown assertion above 2 passed
same two files with the new conftest.py floor and the old tests test_matching_dim_loads fails by name (left its idle evictor running) --

Determinism proof on the six changed test files (511 tests):

run result
20 x -n0 20/20: 511 passed
10 x -n 4 --dist loadgroup --max-worker-restart=2, with test_crew_log_emit.py and test_outbound_files.py as co-tenants (934 tests) 10/10: 934 passed
shuffled node ids, seed 2839, -n0 511 passed
python -m pytest -n0 test/test_flake_pattern_ratchet.py 53 passed
255 test files mentioning the projection, the shared embedder, LlamaCppEmbedder or reset_for_tests, -n 10 25,663 passed, 2 failed: test_native_skill_projection.py::test_launch_record_resources_match_the_alias_file_kiro_cli_loads (fails identically on origin/main on this host: its tmp root sits under ~/.kiro/crew/scratch) and TestBundledLinuxX86CpuGate::test_operator_override_is_kept_out_of_the_bundled_shape_rule (host has LLAMA_CPP_LIB_PATH exported; passes 3/3 when its file runs at -n 4)

Whole test/ tree with the new floor, -n 14 on a 32-core host: 92,388 passed, 83 failed. 74 of the 83 fail identically on origin/main on this host (its scratch root sits under ~/.kiro/crew, setfacl is unavailable, the sandbox userns probe answers EPERM); the other 9 are load timeouts (test_installer_distro, test_issue_radar_crew_*) that pass 2/2 when rerun together. Zero occurrences of the new guard's message or of the home guard's across the run.

Gates: isort, flake8, black, mypy src/kiro_crew/ (2382 files, clean), scripts/check_comment_history.py against origin/main (clean), docs-lint.sh, check_sync_io_in_async, check_loop_bound_locks, check_testpaths_coverage, check_brand_name, check_subprocess_encoding. Frontend gates not run: the diff touches no frontend file.

Manual verification

N/A -- the forced-condition and tripwire runs above are the verification; no UI or external service is involved.

Related Issues

Fixes #20022
Fixes #19514

Pattern harvest

Rule candidate: review-prompt
Pattern: a background thread or pool job that resolves config_dir()/data_home() when it RUNS rather than when it was started or queued (a poller re-reading config, a debounced writer building its payload late); and a test that patch.dict("os.environ", {}, clear=True) instead of unsetting the variables its subject reads.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 15, 2026 02:27
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 05c97dcd7459239f08c20af35d685be90890e408 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 05c97dcd7459239f08c20af35d685be90890e408 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the two leakers are fixed where they leak, and the new floor names the next one. Two test-side items are larger than they need to be.

Notes

  • Item 3 — duplicate of test/embeddings_helpers.py:_live_embed_threads: the new _live_embed_evictors in test/conftest_floor_restores.py walks the same threading.enumerate() and only narrows the name from the kc-embed- prefix to the exact kc-embed-evict. That makes two spellings of one thread probe (grepped _live_embed_threads: 1 definition, 3 test-file importers). Subtraction: give the floor one probe and drop the other's definition.
  • Item 4 — oversized: once Item 1 lands, "the worker no longer resolves the data home". So the teardown cancel no longer removes the Goal's failure. It only stops a write landing in the earlier test's own home, and no measured failure is cited for that. Subtraction: remove _cancel_session_tree_checkpoints and its pytest_runtest_teardown call unless a run shows a failure without them. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (8 items) — 6 justified

Intent: stop an unrelated dashboard test from failing at teardown on PR branches, because two other tests resolve the real data home from background threads. FIX.

  1. A debounced session-tree checkpoint pins its root when it is armed, not when it wakes — justified
  2. About twenty embedding tests close the backends they load, which stops their idle evictors — justified
  3. A new rootdir floor fails any test that leaves a kc-embed-evict thread running — duplicate of test/embeddings_helpers.py:_live_embed_threads (probe only)
  4. Every test's teardown now cancels pending checkpoint writes — oversized
  5. The dashboard URL tests unset only the two SSH variables and keep KIROCREW_HOME — justified
  6. TestAllowedLoopbackPortsEnv stops clearing the whole environment — justified
  7. _debounced_write gains an optional root parameter, with one caller — justified
  8. The testing-conventions spec records the two floor members and a class-4 entry — justified

[FIRST-PRINCIPLES-REVIEWED] 05c97dc

Description read

[DESCRIPTION-READ] c8dd5ec3b28c50ee7043b3b5d356c6028e71351c2b87f9a03501247021a5fcf6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 05c97dcd7459239f08c20af35d685be90890e408 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: each leak is closed where it starts, the shared teardown now names the next leaker, and the only production change moves the root lookup from when the write fires to when it is armed.

[DESIGN-REVIEWED] 05c97dc

Description read

[DESCRIPTION-READ] d4cc5e4974fc2873c4474ddd56a86a693cf17743f9f6157c137afdabe1f4578b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 05c97dcd7459239f08c20af35d685be90890e408 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 05c97dc

Verdict parsed from the review's SHA-scoped output markers for commit 05c97dcd7459239f08c20af35d685be90890e408.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 05c97dcd7459239f08c20af35d685be90890e408 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 05c97dcd7459239f08c20af35d685be90890e408 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 05c97dc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 05c97dcd7459239f08c20af35d685be90890e408 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 15, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/flake-shared-state-leaks branch from ef17ea5 to 4e7f54d Compare October 15, 2026 03:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on 4e7f54d for two small test-side items. The fix itself checks out.

What I verified locally (this head, merge base 10e53cc)

  • Reproduced the flake on the merge base. A throwaway plugin sets embeddings._EVICT_POLL_MAX_SECS = 0.01, then the run is test_custom_embedding_model.py::TestLoadTimeDimValidation::test_matching_dim_loads followed by test_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startup (-p no:randomly -n0). On the merge base, 5 of 5 runs error at teardown with the real-data-home assertion. On this head, 5 of 5 pass.
  • The root cause is fixed without the victim change. This head with the merge base's test_dashboard_origin.py (the old cleared-env victim) passes 3 of 3. This head with the merge base's test_custom_embedding_model.py (the old leaker) fails test_matching_dim_loads by name through the new _no_leaked_embed_evictor floor.
  • The home guard is not weakened. conftest.py only adds code, and on this head a probe test that clears os.environ and calls config_dir() still errors with the same teardown assertion.
  • test_a_debounced_write_resolves_no_path_when_it_wakes fails when only session_tree_projection.py is reverted to the merge base, and passes on this head. The _schedule_checkpoint / _debounced_write change is correct: root is resolved under the lock beside target, the one caller passes it, and self._root still wins for a seeded projection.
  • The six changed test files pass (511 tests) in 3 of 3 runs at -n 4. check_comment_history.py passes, and every check on this head is green.

Requested changes

  1. test/test_embeddings.py:1122: evictor.join(timeout=5) in _close_built_embedders. If that join gives up while the evictor is still alive, the rootdir _no_leaked_embed_evictor floor tears down after this class fixture and fails the test. So the 5 s bound adds a failure that depends on host load. Repro: a probe test in TestLlamaCppEmbedder pins _INFER_STOP_TIMEOUT_SECS to 0 and makes one embedding_idle_evict_secs tick take 6 s. It errors at teardown with "left its idle evictor running". With timeout=30 the same probe passes. Please bound it at 30 s, the default _INFER_STOP_TIMEOUT_SECS.

  2. Some added docstrings describe the old code or the incident instead of the current behaviour. Fast Gate's history pattern lets them through, but they will read as stale once this merges:

    • test/test_dashboard_origin.py:279: "The earlier shape, a patch.dict that CLEARED os.environ on every test, ...", and :285 "nine PR branches in a week". Suggest present tense, for example "Clearing os.environ would also drop the floor's KIROCREW_HOME pin for the test's duration ...".
    • conftest.py:2394-2399: the "MEASURED on a Linux shard" sentence and "nine PR branches in a week".
    • test/test_crew_log_session_tree_projection.py:497-500: "Resolving _current_root() there read the next test's home ... the teardown guard blamed ..." describes the old bug. Suggest saying what resolving at wake time would do.

    The measurement already lives in the PR body and in the testing-conventions entry.

Non-blocking

  • reset_for_tests in session_tree_projection.py (line 2070, not changed here) still says a pending write "resolves its target directory when it WAKES". With this change the worker resolves nothing, so that sentence is now wrong. Worth fixing while you are in the file.
  • Overlap with #20024: it edits the same test in test/test_dashboard_origin.py, pinning _resolve_default_home and joining the lookup thread. This PR removes the leakers and keeps the KIROCREW_HOME pin, so #20024's home pin is not needed for this flake, and the two PRs will conflict textually. Once this lands, #20024 is either superseded or reduces to its lookup-thread join, which belongs to #20022.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 15, 2026
@CrysisDeu
CrysisDeu force-pushed the fix/flake-shared-state-leaks branch from 4e7f54d to c393baa Compare October 15, 2026 17:28
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Addressed on c393baa (single commit, amended). Thanks for the local verification.

1. test/test_embeddings.py join bound. _close_built_embedders now joins the evictor with timeout=30, the default _INFER_STOP_TIMEOUT_SECS, and the docstring says why that bound and not a shorter one. Reproduced your probe (a subclass pinning _INFER_STOP_TIMEOUT_SECS to 0 with one 6 s embedding_idle_evict_secs tick): at 5 s it errors at teardown with "left its idle evictor running"; at 30 s it passes.

2. Docstrings in past tense / incident narration. All three rewritten to describe current behaviour:

  • test/test_dashboard_origin.py _not_over_ssh: now "Clearing os.environ would also drop the floor's KIROCREW_HOME pin ..."; the "earlier shape" sentence and the branch count are gone.
  • conftest.py _no_leaked_embed_evictor: the MEASURED sentence and the branch count are gone; it states what a tick does under a cleared environment and points at the class-4 entry in testing-conventions.md for the measurement.
  • test/test_crew_log_session_tree_projection.py test_a_debounced_write_resolves_no_path_when_it_wakes: now says what resolving at wake time would do; the "measured: ... landing in ..." parenthetical is gone.

Non-blocking, taken: reset_for_tests in session_tree_projection.py now says the target was pinned when the write was armed, and that the cancel is about a write that would recreate a directory the test has removed.

#20024: agreed on the overlap; this PR keeps the pin and removes the leakers, so #20024's _resolve_default_home pin is not needed for this flake. I have not touched #20024; the lookup-thread join there stands on its own under #20022.

Local on this head: the four touched test files 412 passed at -n 4; check_comment_history.py against the merge base clean; flake ratchet 53 passed; flake8/isort/black/mypy clean on the touched files.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of c393baa85c (same merge base 10e53cc119; the interdiff against 4e7f54d8b8 is 5 files, +22/-23). Both requested changes are addressed and I found nothing blocking.

1. Evictor join bound: met. test/test_embeddings.py:1124 now joins with timeout=30, which matches _INFER_STOP_TIMEOUT_SECS = 30.0 in embeddings.py:230. The fixture docstring (:1111-1116) explains why. I re-ran the probe: a subclass of TestLlamaCppEmbedder pins _INFER_STOP_TIMEOUT_SECS to 0 and makes the evictor's second embedding_idle_evict_secs tick take 7 s while the test closes. It passes on this head. With the join set back to 5 s, the same probe errors at teardown with "left its idle evictor running".

2. Docstrings: met. All three now describe current behaviour:

  • test/test_dashboard_origin.py _not_over_ssh: "Clearing os.environ would also drop ...". The earlier-shape sentence and the branch count are gone.
  • conftest.py _no_leaked_embed_evictor: the MEASURED sentence is gone. It now points to the testing-conventions entry for the measurement.
  • test/test_crew_log_session_tree_projection.py test_a_debounced_write_resolves_no_path_when_it_wakes: says what resolving at wake time would do.

The non-blocking reset_for_tests sentence (session_tree_projection.py:2066-2074) now matches the code, since the target is pinned when the write is armed. That edit is docstring-only; the production change in that file is otherwise unchanged from the head I reviewed before. I found no other added comment or docstring that narrates history, and check_comment_history.py against the merge base passes.

The fix itself, re-checked on this head.

  • With a throwaway plugin that sets embeddings._EVICT_POLL_MAX_SECS = 0.01, I ran test_custom_embedding_model.py::TestLoadTimeDimValidation::test_matching_dim_loads followed by test_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startup (-p no:randomly -n0). It passes 3 of 3 on this head and errors at teardown 2 of 2 on the merge base.
  • The six changed test files: 511 passed.

CI. The only red test is test_acp_bindings.py (approve_tool shape and the frozen member inventory; Backend Tests 3.12 shard 5 and Windows shard 5), and Coverage Gate and PR Readiness fail because of it. This diff does not touch that test. It was red on main until #20902, which merged a few minutes after this head's CI run started, so the run's merge ref does not include the fix. A rerun reuses that merge ref; rebasing onto current main should clear it. Every AI and security lane on this head passes.

Non-blocking. The body says Fixes #20022 as well as Fixes #19514, so merging this closes #20022. Your reply says the lookup-thread join in #20024 still stands on its own under #20022. If that join is still wanted, either drop Fixes #20022 here or give the join its own issue, so it does not lose its tracking issue when this merges.

I am holding approval only because CI is not green on this head. I expect to approve once a rebased head is green.

@CrysisDeu
CrysisDeu force-pushed the fix/flake-shared-state-leaks branch from c393baa to 8df7b16 Compare October 15, 2026 20:54
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
cruisercohen
cruisercohen previously approved these changes Oct 15, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving on 8df7b16. This head is a rebase of c393baa onto current main with no code change: same single commit, same nine files, identical added and removed lines.

Both earlier asks are met here:

  • The idle-evictor join in _close_built_embedders now waits 30 seconds, matching the embedder's own stop timeout, so a slow in-flight tick does not trip the leak check at teardown.
  • The added docstrings in the dashboard-origin, conftest and session-tree-projection tests describe current behaviour in present tense.

Required checks are green on this head.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 8df7b16.

The fix looks sound. _schedule_checkpoint pins root beside target under the lock, so the debounced worker no longer resolves the data home when it wakes (session_tree_projection.py#L1726-L1735). The twenty embedding tests close their backends, and the new evictor floor names the next leaker. cruisercohen's change request is cleared by their approval on this head.

What still blocks approval: two current-head AI items have no reply.

  1. First Principles CONCERNS: _cancel_session_tree_checkpoints() in pytest_runtest_teardown (conftest.py#L1962-L1980) runs reset_for_tests() on every test, which also drops the process-wide projection. The lane says it is not needed once the root is pinned at arm time. Please either remove it (plus its testing-conventions bullet) or reply with the run that shows it is needed.
  2. Design punchline: please confirm that the autouse _no_leaked_embed_evictor floor adds no visible per-test time across a full shard.

Also, please settle the tracking for #20022. This PR's Fixes #20022 closes it on merge, but your reply says the lookup-thread join in #20024 still stands under #20022. Either drop Fixes #20022 or give that join its own issue and link it here.

CI on this head is green. I read the code and tests but did not run them.

— crew-pr-reviewer (r105)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 18, 2026
test_dashboard_origin.py's hanging-lookup test erred at teardown on nine PR
branches in a week: the home guard found the operator's real ~/.kiro/crew
resolved. The test resolves no path itself. Two background resolvers left
by earlier tests on the worker did, and the victim's patch.dict(clear=True)
window was where their tick landed with no KIROCREW_HOME to pin it.

Leaker one: a LlamaCppEmbedder whose fake model was published and never
closed keeps a kc-embed-evict daemon that re-reads the memory config
through config_dir() on every tick. Twenty tests in four files built one
and walked away; each now closes it. Leaker two: SessionTreeProjection's
debounced checkpoint write built its payload root with a live
_current_root() when it WOKE on the mc-maint pool, a debounce after the
arming test had ended. The root is now pinned beside the target when the
write is armed, so the worker resolves nothing.

The floor catches the next one by name: _no_leaked_embed_evictor fails the
test that leaves an evictor running and stops it, and the teardown hook
cancels pending checkpoint writes before the pins lift. The victim keeps
the pin, unsetting only the two SSH variables its subject reads.
@iamwhatever
iamwhatever force-pushed the fix/flake-shared-state-leaks branch from 8df7b16 to 05c97dc Compare October 24, 2026 01:29
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 24, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 05c97dc.

The fix itself is sound. _schedule_checkpoint now pins the payload's root beside target when the write is armed, so the debounced worker no longer resolves the data home when it wakes (session_tree_projection_checkpoint.py#L283-L295). The embedding tests close their backends, and the new evictor floor names the next leaker. This head is a rebase of 8df7b16 onto the conftest split, and CI is green.

The items from our r105 HOLD still have no reply:

  1. First Principles Item 4: _cancel_session_tree_checkpoints() runs reset_for_tests() in every test's teardown (conftest.py#L936). The lane says it is not needed once root is pinned at arm time. Remove it and its testing-conventions bullet, or reply with a run that fails without it.
  2. First Principles Item 3: _live_embed_evictors (conftest_floor_restores.py#L305) repeats test/embeddings_helpers.py:_live_embed_threads. Keep one probe, or say why both are needed.
  3. Please confirm that the autouse _no_leaked_embed_evictor floor adds no visible per-test time across a full shard.
  4. Fixes #20022 would close Flaky: test_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startup #20022 on merge, but your reply says the lookup-thread join in test(dashboard): pin default home and join resolver thread to stop isolation leak #20024 still belongs to it. Either drop that line or give the join its own issue and link it.

To clear: one reply per item (a fix or an evidence-backed rebuttal). I read the code and tests but did not run them.

— crew-pr-reviewer (r124)

@cruisercohen

Copy link
Copy Markdown
Contributor

@CrysisDeu iamwhatever's r124 hold on 05c97dc lists four items that still have no reply. One reply per item (a fix or an evidence-backed rebuttal) clears it:

  1. reset_for_tests() in every test's teardown (conftest.py L936): remove it and its testing-conventions bullet now that root is pinned at arm time, or show a run that fails without it.
  2. _live_embed_evictors repeats _live_embed_threads in test/embeddings_helpers.py: keep one probe, or say why both are needed.
  3. Confirm the autouse _no_leaked_embed_evictor floor adds no visible per-test time across a full shard.
  4. Fixes #20022 would close Flaky: test_dashboard_origin.py::TestFormatDashboardUrls::test_a_hanging_hostname_lookup_cannot_stall_startup #20022 on merge, but your earlier reply says the lookup-thread join in test(dashboard): pin default home and join resolver thread to stop isolation leak #20024 still belongs to it: drop that line, or give the join its own issue and link it.

@chenmingwei23
chenmingwei23 merged commit a3f00fd into main Oct 27, 2026
107 of 108 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/flake-shared-state-leaks branch October 27, 2026 06:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants