Repository navigation
fix(pdf-extract): count the venv redirector in the process ceiling - #14424
Conversation
|
Intent: Make PDF extraction work from a venv-hosted gateway on Windows (the Job ceiling must count the venv redirector as a process), and close the test classes the thirteenth five-run hygiene pass measured on native Windows: the review-lane bash harnesses' PATH-shim sleep, est_r8_s33's host-HOME assumption, and the trusted-apps 409 diagnosis instrument; record each class in the testing spec and the writing-tests checklist. |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS The [DESIGN-REVIEWED] 9cb0482 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The live redirector arms ( What this change shipsInventory (9 items) — 9 justifiedIntent: FIX — PDF extraction works from a venv-hosted gateway on Windows, and three Windows-only test-red classes stop wasting contributors' runs.
[FIRST-PRINCIPLES-REVIEWED] 9cb0482 |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
9559dab to
37a6e1b
Compare
|
|
|
|
|
37a6e1b to
c18f1ad
Compare
|
c18f1ad to
de2811f
Compare
|
de2811f to
9e40457
Compare
|
…s ceiling; thirteenth test-hygiene pass PDF extraction never worked from a venv-hosted gateway on Windows: the Job object's ActiveProcessLimit of one refused the venv redirector's own spawn of the base interpreter (exit 101, "Unable to create process using ...") and every document read as `protocol`. The ceiling is now 1 + platform_compat. python_launcher_hops(), which is 1 when sys.executable is a redirector and 0 everywhere else. Three test-hygiene classes from a five-round native-Windows pass (126,965 tests a round) are closed at their root: the review-lane bash harnesses override `sleep` as a shell function instead of a PATH shim that Git Bash's launcher outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the HOME it asserts passes through the scrub; and the ten trusted-apps revoke tests that answered 409 in full-suite runs only now carry the response body in their assertions (the floor that restores hooks_integration's process globals is #14388's, which met the same leak on macOS). Spec section and writing-tests checklist items record all four.
9e40457 to
9cb0482
Compare
|
Rebased: main moved by one commit (#14338, whatsapp admission / spawn run-control scope) since 9e40457 went green; no file this PR touches changed, overlap is same-directory and import-neighbour only, and the neighbour suites plus this PR's own tests pass locally on the rebased head (1157 passed). 9e40457 -> 9cb0482, CI re-running. On 9e40457 every review lane (GPT, Opus, Design, First Principles, UX, Security Scope) passed with no open items; the one red was macOS shard 1/4, which failed on two different unrelated tests across two attempts ( est_decisions_memory_recall then est_cron_manual_run_claim_window), each passing locally on both a clean origin/main tree and this head -- flakes outside this diff. |
Problem / Motivation
Goal: A gateway started from a virtual environment on Windows reads PDFs, and a Windows contributor's full test run no longer carries four classes of red that CI never sees.
On Windows, a gateway started from a virtual environment cannot read PDFs: every document comes back
PDF extraction failed: protocol, and the log shows the extractor child dying with exit 101 andUnable to create process using '"C:\Python312\python.exe" -s -P -m kiro_crew.pdf_extract_child ...'.The thirteenth five-round test-hygiene pass (native Windows, eight workers, 126,965 tests a round) found that defect and three test classes that are red on a developer's Windows box and green on CI: two review-lane bash harnesses that sleep their whole retry budget (75 s a test, four
pytest-timeoutworker kills a round), fivetest_r8_s33_subprocess_env.pytests that assert aHOMEthe host never had, and tentest_trusted_apps_api.pyrevoke tests that answer 409 in full-suite runs only.Why it matters
PDF search and PDF knowledge ingestion silently do nothing for anyone running Kiro Crew from a venv on Windows -- the ordinary developer install. The three test classes cost every Windows contributor a red full run and about an hour of worker time per run, and the trusted-apps one is order dependence that any later test touching app teardown can inherit -- its floor lands in #14388, which found the same leak on macOS; the diagnosis instrument lands here.
Not a goal
test_ai_review_workflows.py's five Windows-skippedsleep-shim sites (needs a host withjq).sandbox.apply_windows_resource_ceiling.bashand nopwshonPATH), which fail by design.What changed (motivation → approach → change)
protocolon a Windows venvpdf_extract._windows_ceilingsetActiveProcessLimit=1; a venv'sScripts\python.exeis a redirector that spawns the real interpreter as its child, so the limit refused that spawnplatform_compat.python_launcher_hops()(1 whensys.executableis the redirector, else 0); the ceiling is1 + hops, and_killends the whole tree on Windows so a timed-out document's interpreter dies with its redirectorsleeponPATH; Git for Windows'bin\bash.exelauncher prepends/usr/bin, so/usr/bin/sleepwonsleepis a shell function in the driver script, recorded per call (test_withheld_verdict_retention.py,test_review_slot_read_error.py)HOME was scrubbedx5HOMEpasses through the scrub but never gives the parent one; CI runners export it, a server session does notHOME(test_r8_s33_subprocess_env.py)teardown_incomplete, full suite onlyinit_hooks_system()publisheshooks_integration._lifecycle_dispatcherand nothing clears it; a dashboard-server test leaves one carrying aMagicMockcron service, and every later revoke on that worker tears down through ittest_trusted_apps_api.py's status assertions carry the response body, which is what named the branchThe production change is two files:
src/kiro_crew/platform_compat.pygainspython_launcher_hops(), andsrc/kiro_crew/pdf_extract.pysizes its ceiling with it. Its owning spec docs/system-specs/common/platform-compat.md gains the matching helper-table row. The ceiling still bounds a fork bomb: two processes instead of one where the redirector is real, and over-counting on a host where it is not is harmless. Launchingsys._base_executabledirectly would avoid the hop but lose the venv's site-packages, wherekiro_crewis installed, so the child could not import its module.The spec's flake-classes chapter gains the thirteenth-pass section (
docs/system-specs/common/testing-conventions.md), and thewriting-testsskill checklist (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md) gains one item per class.test_trusted_apps_api.py's status assertions carry the response body, which is what made the 409 diagnosable.Scope: the two harness files the runs measured.
test_ai_review_workflows.pyplants the samesleepshim at five sites and already skips those tests on Windows outright; un-skipping them through the same function is a follow-up that needs a host withjq.Backwards compatibility
Compatible: the extractor's ceiling only grows by the one process the redirector already occupies; no interface, config or output changes.
Tests
test_pdf_extract.py::TestWindowsCeiling::test_a_venv_redirector_is_counted_in_the_process_ceiling--max_procs == 2when the hop is 1; the existing== 1pin keeps the hop at 0.test_windows_job_limits.py::TestPythonLauncherHops-- the derivation on both platforms with pinnedsysexecutables, and a native Windows test that spawnssys.executableunder1 + hopsand asserts it ran, then (on a venv host) under1and asserts exit 101 with the redirector's message.test_pdf_extract.py::TestWindowsCeiling::test_a_timed_out_child_is_killed_as_a_tree-- the deadline path hands the applied pid tokill_process_tree;test_windows_job_limits.py::TestPythonLauncherHops::test_a_timed_out_redirector_child_does_not_outlive_the_deadline-- live kernel, the redirector's interpreter pid is gone after the timeout (a property pin: CPython's redirector already runs its child in a kill-on-close job, so this holds with the tree kill neutralised too; the faked test is what pins the mechanism).test_withheld_verdict_retention.py::test_the_retry_backoff_is_intercepted_not_slept-- the recorded schedule is5 10 15 20 25; with the override removed it fails on "no sleep reached the override" after 151 s. Both harness files: 444 tests in 54 s at-n 4(was ~3,700 s of worker time).test_r8_s33_subprocess_env.py-- 5 passed with noHOMEin the host environment.Manual verification
Probe on the Windows venv host: the same PDF under a Job limit of 1 returns
protocol; under 2 it extractsHello PDF regression. The five previously red PDF tests (test_file_grep.py::TestDocumentPass,test_knowledge.py::TestFileReaderPdf) pass on this branch and fail on the base.Related Issues
no linked issue: found by the test-hygiene sweep, not reported.
Pattern harvest
Rule candidate: review-prompt
Pattern: a Windows Job
ActiveProcessLimitor any "exactly one child" bound sized without counting the venv redirector thatsys.executableis on Windows; and, for tests, a coreutil neutralised by aPATHshim instead of a shell function under Git Bash.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)