Skip to content

fix(pdf-extract): count the venv redirector in the process ceiling - #14424

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/hygiene-sweep-10
Sep 27, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/hygiene-sweep-10

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway started from a virtual environment on Windows reads PDFs, and a Windows contributor's full test run no longer carries four classes of red that CI never sees.

On Windows, a gateway started from a virtual environment cannot read PDFs: every document comes back PDF extraction failed: protocol, and the log shows the extractor child dying with exit 101 and Unable to create process using '"C:\Python312\python.exe" -s -P -m kiro_crew.pdf_extract_child ...'.

The thirteenth five-round test-hygiene pass (native Windows, eight workers, 126,965 tests a round) found that defect and three test classes that are red on a developer's Windows box and green on CI: two review-lane bash harnesses that sleep their whole retry budget (75 s a test, four pytest-timeout worker kills a round), five test_r8_s33_subprocess_env.py tests that assert a HOME the host never had, and ten test_trusted_apps_api.py revoke tests that answer 409 in full-suite runs only.

Why it matters

PDF search and PDF knowledge ingestion silently do nothing for anyone running Kiro Crew from a venv on Windows -- the ordinary developer install. The three test classes cost every Windows contributor a red full run and about an hour of worker time per run, and the trusted-apps one is order dependence that any later test touching app teardown can inherit -- its floor lands in #14388, which found the same leak on macOS; the diagnosis instrument lands here.

Not a goal

  • The hooks_integration process-global floor: it lands in test: Close what the eleventh hygiene sweep found #14388, which met the same leak on macOS.
  • Un-skipping test_ai_review_workflows.py's five Windows-skipped sleep-shim sites (needs a host with jq).
  • Any change to the agent-tree Job ceiling in sandbox.apply_windows_resource_ceiling.
  • The two host-provisioning reds (no bash and no pwsh on PATH), which fail by design.

What changed (motivation → approach → change)

Symptom Root cause Change
Production PDFs read as protocol on a Windows venv pdf_extract._windows_ceiling set ActiveProcessLimit=1; a venv's Scripts\python.exe is a redirector that spawns the real interpreter as its child, so the limit refused that spawn platform_compat.python_launcher_hops() (1 when sys.executable is the redirector, else 0); the ceiling is 1 + hops, and _kill ends the whole tree on Windows so a timed-out document's interpreter dies with its redirector
Tests 75 s a test, worker timeouts A fake sleep on PATH; Git for Windows' bin\bash.exe launcher prepends /usr/bin, so /usr/bin/sleep won sleep is a shell function in the driver script, recorded per call (test_withheld_verdict_retention.py, test_review_slot_read_error.py)
Tests HOME was scrubbed x5 The test asserts HOME passes through the scrub but never gives the parent one; CI runners export it, a server session does not autouse fixture plants HOME (test_r8_s33_subprocess_env.py)
Tests Revoke answers 409 teardown_incomplete, full suite only init_hooks_system() publishes hooks_integration._lifecycle_dispatcher and nothing clears it; a dashboard-server test leaves one carrying a MagicMock cron service, and every later revoke on that worker tears down through it The floor that restores the globals is #14388's (the concurrent macOS pass met the same leak); this PR only makes test_trusted_apps_api.py's status assertions carry the response body, which is what named the branch

The production change is two files: src/kiro_crew/platform_compat.py gains python_launcher_hops(), and src/kiro_crew/pdf_extract.py sizes its ceiling with it. Its owning spec docs/system-specs/common/platform-compat.md gains the matching helper-table row. The ceiling still bounds a fork bomb: two processes instead of one where the redirector is real, and over-counting on a host where it is not is harmless. Launching sys._base_executable directly would avoid the hop but lose the venv's site-packages, where kiro_crew is installed, so the child could not import its module.

The spec's flake-classes chapter gains the thirteenth-pass section (docs/system-specs/common/testing-conventions.md), and the writing-tests skill checklist (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md) gains one item per class. test_trusted_apps_api.py's status assertions carry the response body, which is what made the 409 diagnosable.

Scope: the two harness files the runs measured. test_ai_review_workflows.py plants the same sleep shim at five sites and already skips those tests on Windows outright; un-skipping them through the same function is a follow-up that needs a host with jq.

Backwards compatibility

Compatible: the extractor's ceiling only grows by the one process the redirector already occupies; no interface, config or output changes.

Tests

  • test_pdf_extract.py::TestWindowsCeiling::test_a_venv_redirector_is_counted_in_the_process_ceiling -- max_procs == 2 when the hop is 1; the existing == 1 pin keeps the hop at 0.
  • test_windows_job_limits.py::TestPythonLauncherHops -- the derivation on both platforms with pinned sys executables, and a native Windows test that spawns sys.executable under 1 + hops and asserts it ran, then (on a venv host) under 1 and asserts exit 101 with the redirector's message.
  • test_pdf_extract.py::TestWindowsCeiling::test_a_timed_out_child_is_killed_as_a_tree -- the deadline path hands the applied pid to kill_process_tree; test_windows_job_limits.py::TestPythonLauncherHops::test_a_timed_out_redirector_child_does_not_outlive_the_deadline -- live kernel, the redirector's interpreter pid is gone after the timeout (a property pin: CPython's redirector already runs its child in a kill-on-close job, so this holds with the tree kill neutralised too; the faked test is what pins the mechanism).
  • test_withheld_verdict_retention.py::test_the_retry_backoff_is_intercepted_not_slept -- the recorded schedule is 5 10 15 20 25; with the override removed it fails on "no sleep reached the override" after 151 s. Both harness files: 444 tests in 54 s at -n 4 (was ~3,700 s of worker time).
  • test_r8_s33_subprocess_env.py -- 5 passed with no HOME in the host environment.

Manual verification

Probe on the Windows venv host: the same PDF under a Job limit of 1 returns protocol; under 2 it extracts Hello PDF regression. The five previously red PDF tests (test_file_grep.py::TestDocumentPass, test_knowledge.py::TestFileReaderPdf) pass on this branch and fail on the base.

Related Issues

no linked issue: found by the test-hygiene sweep, not reported.

Pattern harvest

Rule candidate: review-prompt
Pattern: a Windows Job ActiveProcessLimit or any "exactly one child" bound sized without counting the venv redirector that sys.executable is on Windows; and, for tests, a coreutil neutralised by a PATH shim instead of a shell function under Git Bash.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner September 27, 2026 08:41
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Make PDF extraction work from a venv-hosted gateway on Windows (the Job ceiling must count the venv redirector as a process), and close the test classes the thirteenth five-run hygiene pass measured on native Windows: the review-lane bash harnesses' PATH-shim sleep, est_r8_s33's host-HOME assumption, and the trusted-apps 409 diagnosis instrument; record each class in the testing spec and the writing-tests checklist.
Not a goal: the hooks_integration process-global floor (lands in #14388); un-skipping test_ai_review_workflows' five Windows-skipped shim sites (needs a jq host); any change to the agent-tree Job ceiling in sandbox.apply_windows_resource_ceiling; the two host-provisioning reds (no bash/pwsh on PATH).

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 9cb0482b4417ebd2d8b1176badc28a16c0c593fe — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

The _WINDOWS_ONLY arms that pin the fix never run on CI — confirm the claimed native venv-host run before merge.

[DESIGN-REVIEWED] 9cb0482

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 9cb0482b4417ebd2d8b1176badc28a16c0c593fe and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9cb0482

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9cb0482b4417ebd2d8b1176badc28a16c0c593fe: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 9cb0482b4417ebd2d8b1176badc28a16c0c593fe — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The live redirector arms (if hops: and the timeout property test's venv half) run only on a venv host — CI's interpreter runner never exercises them, so the author's Windows box is their sole witness.

What this change ships

Inventory (9 items) — 9 justified

Intent: FIX — PDF extraction works from a venv-hosted gateway on Windows, and three Windows-only test-red classes stop wasting contributors' runs.

  1. PDFs extract from a Windows venv gateway; the process ceiling counts the redirector — justified
  2. New public helper platform_compat.python_launcher_hops() (1 production consumer, pdf_extract.py:226; placement mandated by the platform-compat routing invariant) — justified
  3. A timed-out extractor child is killed as a tree on Windows, not the redirector alone — justified
  4. Review-lane bash tests intercept sleep as a shell function, recorded per call (2 files; the 9 shim sites in test_ai_review_workflows.py are declared deferred) — justified
  5. test_r8_s33_subprocess_env.py plants HOME so the pass-through assertion measures the scrub — justified
  6. test_trusted_apps_api.py status assertions print the response body on failure — justified
  7. platform-compat spec gains the ceiling-sizing helper row (same-commit spec invariant) — justified
  8. testing-conventions thirteenth-pass section + three writing-tests checklist items — justified
  9. New pin asserting the recorded backoff schedule 5 10 15 20 25 — justified

[FIRST-PRINCIPLES-REVIEWED] 9cb0482

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 9cb0482b4417ebd2d8b1176badc28a16c0c593fe — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9cb0482

Verdict parsed from the review's SHA-scoped output markers for commit 9cb0482b4417ebd2d8b1176badc28a16c0c593fe.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9cb0482b4417ebd2d8b1176badc28a16c0c593fe: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Timed-out venv extractor survives its redirector span=d96f3567bbc8

fixed in 37a6e1b: pdf_extract._kill now calls platform_compat.kill_process_tree(proc.pid, SIGKILL) on Windows BEFORE proc.kill(), while the redirector is alive to name its child, then reaps as before.
Pinned twice: test_pdf_extract.py::TestWindowsCeiling::test_a_timed_out_child_is_killed_as_a_tree (faked Windows path: the deadline hands the applied pid to kill_process_tree) and test_windows_job_limits.py::TestPythonLauncherHops::test_a_timed_out_redirector_child_does_not_outlive_the_deadline (live venv host: the redirector's interpreter pid is gone after the timeout).
Measured while pinning: CPython's redirector already runs its child in its own KILL_ON_JOB_CLOSE job, so the interpreter died with the redirector even with the tree kill neutralised. The tree kill makes that guarantee the gateway's own instead of a launcher detail; the native test is a property pin and its docstring says so.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • RUN_STATS placeholder left in the thirteenth-pass record span=90dfb3da3a34

fixed in 37a6e1b: the placeholder now reads the measured figures (121,041 to 121,046 passed, 23 to 28 failed, 0 errors, 5,742 skipped per round, 59 to 80 minutes, minimum available memory 27.1 GiB) in the slot every prior pass fills.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 2 - one consumer, generalized (python_launcher_hops executable parameter) span=79f7cae25ac7

fixed in 37a6e1b: the executable parameter is removed; python_launcher_hops() reads sys.executable only, and the unit assertion that exercised the parameter is gone with it.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 5 - zero consumers (_sleep_calls copy in test_review_slot_read_error) span=a772f0a274f7

fixed in 37a6e1b: _sleep_calls is deleted from test_review_slot_read_error.py; SLEEP_LOG stays because the override function writes to it.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • python_launcher_hops is a general Windows-Job sizing fact next to a pdf-only caller span=cfd9e01641f9

rebutted as already satisfied: the helper lives in platform_compat, the module that owns �pply_job_limits, not in pdf_extract; its docstring is written as the shared contract on purpose, so a later agent-tree ceiling caller imports it rather than re-deriving. Nothing in this PR to change; the agent-tree ceiling is out of this PR's goal.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • python_launcher_hops lands without its owning spec row span=308885e71df1

fixed in c18f1ad: docs/system-specs/common/platform-compat.md gains a row in the one-helper-per-POSIX-call table, in the same commit as the helper: sizing an ActiveProcessLimit as one Python child is 1 + python_launcher_hops(), a timed-out child is killed as a tree, and the NOT column names the literal max_procs=1 and the bare proc.kill().

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • A new test spawns a real sleeper child whose only kill sits on the passing path span=04eac822e8e7

fixed in de2811f: est_a_timed_out_redirector_child_does_not_outlive_the_deadline now runs its body under ry/finally; the finally collects the recorded interpreter pid with platform_compat.kill_pid(child_pid, SIGKILL) when it still exists, so the failing arm (an orphaned interpreter) is reaped by the test rather than left on the host. The sleeper is 8 s instead of 60 s, so a missed kill self-heals inside the test's own budget. This is the
o-test-side-effects shape the sibling test in the same class already follows.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/common/testing-conventions.md and writing-tests/SKILL.md since this head went green (#14421, the twelfth pass, landed; its section and checklist items sit before this PR's thirteenth-pass section and items); c18f1ad -> de2811f, CI re-running.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 27, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • python_launcher_hops returns 0 on OSError, under-counting on the error path span=50f0a6990cd8

fixed in 9e40457: on a
ealpath error the helper falls back to comparing the unresolved spellings, so differing paths count the hop (1) and identical ones do not (0) -- failing toward the harmless over-count instead of the defect. Pinned by TestPythonLauncherHops::test_an_unresolvable_path_fails_toward_the_hop (realpath patched to raise; both arms).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 27, 2026
…s ceiling; thirteenth test-hygiene pass

PDF extraction never worked from a venv-hosted gateway on Windows: the Job
object's ActiveProcessLimit of one refused the venv redirector's own spawn of
the base interpreter (exit 101, "Unable to create process using ...") and every
document read as `protocol`. The ceiling is now 1 + platform_compat.
python_launcher_hops(), which is 1 when sys.executable is a redirector and 0
everywhere else.

Three test-hygiene classes from a five-round native-Windows pass (126,965 tests
a round) are closed at their root: the review-lane bash harnesses override
`sleep` as a shell function instead of a PATH shim that Git Bash's launcher
outranks (75 s a test, four worker timeouts a round); test_r8_s33 plants the
HOME it asserts passes through the scrub; and the ten trusted-apps revoke
tests that answered 409 in full-suite runs only now carry the response body in
their assertions (the floor that restores hooks_integration's process globals
is #14388's, which met the same leak on macOS). Spec section and writing-tests
checklist items record all four.
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved by one commit (#14338, whatsapp admission / spawn run-control scope) since 9e40457 went green; no file this PR touches changed, overlap is same-directory and import-neighbour only, and the neighbour suites plus this PR's own tests pass locally on the rebased head (1157 passed). 9e40457 -> 9cb0482, CI re-running. On 9e40457 every review lane (GPT, Opus, Design, First Principles, UX, Security Scope) passed with no open items; the one red was macOS shard 1/4, which failed on two different unrelated tests across two attempts ( est_decisions_memory_recall then est_cron_manual_run_claim_window), each passing locally on both a clean origin/main tree and this head -- flakes outside this diff.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 27, 2026
@bolichen97
bolichen97 enabled auto-merge (rebase) September 27, 2026 15:56
@chenmingwei23
chenmingwei23 merged commit a41323c into main Sep 27, 2026
97 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/hygiene-sweep-10 branch September 27, 2026 16:00
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants