Repository navigation
fix: gatewayd orphan reap, knowledge close seam; tenth hygiene pass - #12820
Conversation
|
Intent: Close every test-hygiene finding a five-run Linux baseline surfaced -- the one deterministic red, the three single-round flakes, and the descriptor, spawn, kill, network and memory classes behind them -- at the seam production already reads, fixing the production defects those flakes exposed (crew-log lease pinned by a returned traceback; a private MCP backend whose reap teardown cancelled; a knowledge store with no cross-thread close) rather than the assertions that went red, and record each class in the testing spec so the next pass does not re-derive it. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the session-wide conftest flag still leaves the flag-off production connection path exercised (the two production-guard tests must build stores outside the flipped flag). Suggestions
[DESIGN-REVIEWED] 6e62dd6 |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the suite-wide test connection factory: What this change shipsInventory (10 items) — 10 justifiedIntent: FIX — make five-run-red test flakes deterministic and reap the production orphan they exposed.
[FIRST-PRINCIPLES-REVIEWED] 6e62dd6 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNothing blocks; two advisory findings. FINDING — test/test_source_corpus.py:399 — FINDING — src/kiro_crew/knowledge/store.py:26 — the flag comment, the [OPUS-REVIEWED] 6e62dd6 |
c99786a to
1eae174
Compare
|
self-added: yes
|
|
self-added: yes
|
1eae174 to
7d27b67
Compare
|
self-added: yes
|
|
self-added: yes
|
7d27b67 to
0afc43c
Compare
|
self-added: yes
|
|
self-added: yes
|
|
Human judgment recorded@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
Human judgment recorded, but the following reviewer lane(s) could not be re-run automatically: GPT 5.6. Re-run the lane's latest workflow run for |
24e3704 to
854d4e0
Compare
|
Reviewed The blocking finding is real, but it is pre-existing on
|
| Production change | Mutation | Result |
|---|---|---|
gatewayd.py pool.spawn_shutdown(orphan) |
reverted to inline await orphan.shutdown(timeout=2.0) |
caught — test_mcp_gateway_pool_integ::test_real_stubs_without_poolable_get_their_own_backend |
vector_memory.py init() closes the handle it replaces |
removed the close block | caught — test_vector_memory::TestSchemaInit::test_a_repeated_init_closes_the_handle_it_replaces |
knowledge/store.py _close_all_for_tests |
conn.close() → pass |
caught — test_knowledge::TestKnowledgeStore::test_close_all_releases_every_threads_connection_and_reopens_lazily |
browser_cli/install.py cwd=candidate.parent |
removed the cwd= argument |
caught — test_browser_cli_install::TestWindowsGatewayCommand::test_install_stages_node_inside_the_managed_root |
crew_log/emit.py with_traceback(None) (before the rebase dropped it) |
reverted to return exc |
caught — 6 teardown lease-residue ERRORS on exactly the disk-error tests |
knowledge/store.py (+129/-5), the largest change and the one the title does not name, is production-inert. With _ALLOW_CROSS_THREAD_CLOSE_FOR_TESTS false the connect kwargs are byte-identical to before, no factory is installed, no _owner_ident is set and nothing is registered in _connections; the db property's new generation check cannot fire, because getattr(self._thread_local, "gen", self._generation) defaults to the current generation and _generation never leaves 0; _close_all_for_tests raises unless the flag is set. Only test/conftest.py sets it, and there is no production caller. No query, index, dedup or delete path changed, so there is no data-loss surface. The one production-visible delta is close() rebuilding an always-empty _connections list under a lock.
The orphan reap introduces no new PID or PGID reaper. Backend.shutdown is unchanged (0 lines) and still acts on its own spawned self.process handle, escalating through platform_compat.kill_process_tree_async. The change is purely scheduling. _safe_shutdown uses timeout=2.0, identical to the timeout=2.0 it replaces, so the reap budget is unchanged. shutdown_all snapshots _shutdown_tasks under self._lock and gathers it, so the task is joined. The _spawn_shutdown → spawn_shutdown rename is complete: 0 references to the old name remain and both production call sites are updated in-diff. The new path also has no race — release_exclusive is awaited, then spawn_shutdown runs synchronously with no suspension point between them, so teardown cannot slip between the map removal and the task creation. That window was previously the whole failure: the inline await was cancelled by the same teardown while release_exclusive had already dropped the backend from the exclusive map, so nothing reaped it.
vector_memory.py: the close sits inside init() under _db_lock, immediately followed by _init_database() reopening the handle. browser_cli/install.py: cwd is keyword-only and defaults to None, preserving the old behaviour for every caller but _staged_node_runs.
No weakened assertion across the test and doc files. Net +30 asserts, no added skip/xfail/skipif, no numeric tolerance introduced. All 21 removed asserts are accounted for: verbatim moves (test_mcp_gateway_daemon_lifecycle.py's three warming asserts move out of a manual try/finally now handled by the autouse _no_git_fingerprint fixture), black re-wraps (test_context_marker_neutralization.py), fixture-parameter rewrites with identical assert bodies (test_skill_search_index.py, test_skill_mapping_and_lazy_default.py), an absolute-path hardening (test_browser_launcher.py), and one genuine strengthening — test_argv_floor_inline_and_brace_scope.py trades a flaky wall-clock ratio for a deterministic visit-count bound plus an anti-vacuous guard, keeping the CPU-time ceilings on time.thread_time().
Both shared-infrastructure files were read in full. conftest.py's two new autouse fixtures are isolation, not weakening: _knowledge_store_cross_thread_close relaxes only SQLite's native guard while _ThreadAffineTestConnection re-applies it in Python for everything but close(), and test_knowledge.py flips the flag off to keep the production path covered; _no_boot_sandbox_sweep rebinds only kiro_crew.session.cleanup_stale_sandbox_profiles while the sweep's own tests call kiro_crew.sandbox.cleanup_stale_sandbox_profiles directly, with test_session_cleanup.py::TestTheSuiteNeverRunsTheBootSandboxSweep pinning the no-op in force. source_corpus.py's streaming rewrite is semantically equivalent — same sorted(rglob("*.py")) walk, same readable filter, and _nfkc's isascii() shortcut is sound because NFKC is identity on ASCII, so homoglyph folding is preserved. No ratchet narrows; the removed _read_tree/_normalized_texts have 0 references.
.github/black-baseline.txt removes exactly the files that became black-clean, verified with black --check, and scripts/check_black_formatting.py reports no pending prune. Docs (knowledge.md, mcp-gateway-daemon-lifecycle.md, testing-conventions.md) describe the seams consistently with the code.
Two description inaccuracies
Neither is a defect, but the review lanes read the description:
- It says
.github/black-baseline.txt"drops three files that are now black-clean"; the diff removes two (test/test_context_marker_neutralization.py,test/test_context_ui_language.py). - The Tests section names
test_vector_memory.py::TestPersistence::test_a_repeated_init_closes_the_handle_it_replaces. The actual class isTestSchemaInit— it spans lines 898-1204 and the test is at 1182. The test exists and passes.
Test runs
test_crew_log_edge_exhaustion,test_mcp_gateway_daemon_lifecycle,test_mcp_gateway_pool_integ,test_vector_memory,test_browser_cli_install,test_eventlog_hooks— 441 passed, 14 skipped in 39.20s.test_knowledge,test_knowledge_store_onloop_db,test_security— 1409 passed, 1 skipped, 3 failed. All three failures areTestFileReaderPdf::*onpdfplumber import failed, an optional dependency missing from my local venv. That class is not in the diff.
Scope note and what I did not verify
The bulk of the verification above ran at 24e370457450de77f328061312621af1d28e43cf, before the rebase. I re-checked the carry-over rather than assuming it: the rebase dropped exactly four files (crew_log/emit.py, crew_log/lease.py, test_crew_log_edge_exhaustion.py, test_eventlog_hooks.py — now on main via #12812) and added none, every surviving production patch is byte-identical line-for-line, and the only conftest.py change is "ninth" → "tenth" in two comments. So the findings transfer, except the emit.py mutation row above, which is no longer in this PR's scope and is listed only as the evidence it was.
Not verified: the three TestFileReaderPdf tests did not execute locally. The Windows behaviour of both kill_pid and platform_compat.kill_process_tree_async is reasoned from source only — I have no Windows host, which is also why I am relaying the blocking finding rather than claiming to have reproduced the kill. The full suite was not run, only the targeted module suites, so a cross-file ordering interaction outside them is unverified. testing-conventions.md's added lines were spot-checked against the seams they describe rather than read line by line.
854d4e0 to
8028200
Compare
|
self-added: yes
|
8028200 to
78109bc
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
Problem / Motivation
Five full runs of the backend suite on one commit were red every time, and not with the same test.
test/test_dev_fleet_app.py::test_trusted_bin_pins_the_resolved_target_not_the_symlinkfailed 5/5 on a host whoseTMPDIRsits under$HOME. Three other tests each failed exactly once: a wall-clock linearity ratio intest/test_argv_floor_inline_and_brace_scope.py, a stale crew-log write lease intest/test_eventlog_hooks.pyleft bytest/test_crew_log_edge_exhaustion.py870 tests earlier on the same worker, and a second WARNING intest/test_taskq_runner_adapter.pyfrom a sandbox sweep another test'sSessionManagerhad armed. Underneath them the per-test probe found realgh,ssh,aws configure,systemctlandmisespawned from unit tests, 52 tests leaking descriptors, eight tree-scan ratchets holding 200+ MiB, and a teardown that SIGKILLed pid numbers it had already proven dead.Why it matters
A single CI run would have shipped each of the three flakes as green. One of them is a production bug:
mcp_gateway/gatewayd.pylet a private MCP backend outlive the daemon because its reap was cancelled by the daemon's own teardown. (The third,crew_log/emit.pypinning a unit's write lease through a returned exception's traceback, was found by this pass and landed onmainfirst through the tenth-pass PR, which returns the failure's type; this PR carries no crew-log change.) The descriptor class had no fix available to tests at all:KnowledgeStorecould not close a connection another thread opened, so 52 tests leaked three descriptors per store.What changed (motivation → approach → change)
run_gatewaydmcp_gateway/gatewayd.pyreaps a disconnected stub's backend throughpool.spawn_shutdown(made public inmcp_gateway/pool.py), whichshutdown_alljoinsknowledge/store.pygains the test-only_close_all_for_tests()(generation bump; under a module flag the test conftest flips, connections opencheck_same_thread=Falsethrough a factory that keeps the thread guard in Python for everything butclose()and are registered for the teardown close; production keeps SQLite's native guard and registers nothing);vector_memory.py::init()closes the handle it replacesbrowser_cli/install.py::_runtakescwd=;_staged_node_runspassescandidate.parenttest/source_corpus.pystreams (read, normalise, filter, parse, drop) and caches only the path listEach red was a victim, not a culprit, so every fix lands at the seam the product already reads rather than in the assertion that went red. The dev-fleet test pins
Path.hometo a sibling of its fixture and a mirror test plants inside the patched home. The linearity guard counts the characters the translator reads at 3k and 12k braces (at most 6x for 4x) instead of timing a ratio, and keeps the base's 30 s and 1 s ceilings on thread CPU._one_warningfilters by logger name, andtest/conftest.pygains_no_boot_sandbox_sweep, which pinskiro_crew.session.cleanup_stale_sandbox_profilesto a no-op (138 real sweeps onmc-maintthreads acrosstest_session.pywithout it, 0 with it).The quiet classes follow the same rule. Real binaries are stopped at their resolvers: in
src/kiro_crew/apps/builtins/auto_improvement/tests/test_dogfood_learnings.pythe conftest_public_dnsfixture instead of a class-local copy that dropped the_gh_prefers_sshpin (andtest_suite_scope.pybeside it runs its clone's git from the clone), an autouse pin on the cachedssh -Vprobe in six sandbox files,platform_compat.trusted_*_binandcode_fingerprint.trusted_git_binpinned to their no-spawn arm,monkeypatch.chdir(tmp_path)where production spawnsgit -C <scratch>. Every inlineVectorMemoryStore,SkillsLoaderandSubagentManagergoes through anopenedregister-and-close fixture; store fixtures call_close_all_for_tests()(production has no moment at which every thread is provably idle short of exit, so the seam gets no production consumer).test/test_pid_lifecycle.py::_reapsignals only whileprocess_start_timestill matches the identity pinned at spawn, andtest/test_platform_compat.pyprobes a pid abovepid_maxand, on Windows where no pid ceiling is readable, asserts the same contract through a stubbedtaskkillinstead of the real one.test/test_security.pystubs the TEST-NET datagram probe with an inert socket and asserts the peers it would have used;test/test_browser_launcher.pyuses the run-ownedshort_sock_diras an absolute root instead ofchdirplus..docs/system-specs/common/testing-conventions.mdgets the tenth-pass section,docs/system-specs/modules/knowledge.mdanddocs/system-specs/modules/mcp-gateway-daemon-lifecycle.mdrecord the two new contracts, andsrc/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.mdgains nine checklist items..github/black-baseline.txtdrops two files that are now black-clean.Backwards compatibility
Compatible:
KnowledgeStore.close()keeps its per-thread contract (pinned bytest_knowledge_cross_thread.py) and_close_all_for_tests()is an additive test-only seam that raises unless the test conftest's flag is set, and production connections keepcheck_same_thread=True;BackendPool.spawn_shutdownis the renamed_spawn_shutdownwith both callers updated;_runinbrowser_cli/install.pytakes a keyword-onlycwdthat defaults to the old behaviour;_run_jobstill returns the same exception object, only without__traceback__, and its callers onlyisinstanceit.Tests
test_knowledge.py::TestKnowledgeStore::test_close_all_releases_every_threads_connection_and_reopens_lazily— every thread's connection closed, lazy reopen after.test_knowledge.py::TestKnowledgeStore::test_production_keeps_the_thread_guard_and_the_seam_refuses_without_the_flag— with the flag off, a connection used from another thread is refused and the seam raises.test_knowledge.py::TestKnowledgeStore::test_the_test_mode_connection_still_refuses_another_threads_use— under the flag, use from another thread is still refused; onlyclose()is cross-thread.test_vector_memory.py::TestSchemaInit::test_a_repeated_init_closes_the_handle_it_replaces— no orphaned handle on re-init.test_mcp_gateway_pool_integ.py::test_real_stubs_without_poolable_get_their_own_backend— no private backend pid exists afterrun_gatewaydreturns (the inline reap leaves all three alive).test_dev_fleet_app.py::test_trusted_bin_refuses_target_under_home— the refusal the symlink test now pins around still fires.test_browser_cli_install.py— the staged launcher'scwdis the managed leaf.test_session_cleanup.py::TestTheSuiteNeverRunsTheBootSandboxSweep— the conftest pin on the boot sandbox sweep is in force and reaches the name the session module calls.test_argv_floor_inline_and_brace_scope.py— work-counted linearity and a structural alternation-group bound, with the base's two ceilings kept on thread CPU; a planted per-{suffix rescan reads 15.9x and fails, as does budget removal.Manual verification
Five-run baseline: 126,905 backend tests per run with the reds above; vitest 36,811 and electron 1,999 green in all five. After the fixes, the 57 touched test files together at
-n 4: 6,391 passed, 52 skipped.mypy --platform linux src/kiro_crew, isort, flake8, black baseline, docs lint, subprocess-encoding, sync-io-in-async and lockdown gates pass.Related Issues
no linked issue: findings from a scheduled test-hygiene pass, none tracked as an issue.
Pattern harvest
Rule candidate: review-prompt
Pattern: "a caught exception returned (not raised) with its traceback attached — the traceback → frames → local → exception cycle pins everything those frames hold until cyclic GC"
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)