Skip to content

test(context): bound the import-order subprocess by the lost-run bound - #20795

Merged
cruisercohen merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/20782-import-order-subprocess-bound
Oct 16, 2026
Merged

cruisercohen merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/20782-import-order-subprocess-bound

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the context import-order test waits as long as the suite's other bounded waits for its fresh interpreter, and a child that never finishes fails with a message naming the import order.

test/test_agent_sdk_context_bridge.py::test_context_import_orders_do_not_form_a_cycle starts a fresh interpreter per import order and gave it timeout=30. On a loaded Windows runner a cold import of those modules took longer than that, so two parametrizations failed with a bare subprocess.TimeoutExpired on a PR that touched nothing near them. The test checks import order, not import speed.

Why it matters

Every PR's Windows shard can go red on this test for reasons unrelated to its diff, and the bare TimeoutExpired does not say which import order was slow, so each one costs a log read and a rerun.

Not a goal

  • Making the cold import faster.
  • Changing subprocess bounds in tests that already wait 120 s or more.

What changed (motivation → approach → change)

Symptom: TimeoutExpired after 30 s on Windows (5). Cause: a fixed 30 s literal, shorter than the bound the suite gives every other wait, on a step whose cost is a cold interpreter start. Change:

Test Bound before Bound now (Linux shards and local, --timeout=120) Bound now (Windows shards, --timeout=180)
test_context_import_orders_do_not_form_a_cycle 30 s 60 s 90 s
test_context_composition_contract.py::test_every_owner_is_loaded_by_a_fresh_import_of_the_facade 60 s 60 s 90 s
  • Both tests pass default_timeout() (from kiro_crew.testing.wait: half the running test's timeout, which the rootdir conftest publishes at each test's setup) to subprocess.run. TimeoutExpired becomes an AssertionError that names what hung, e.g. importing kiro_crew.essential_delivery -> kiro_crew.providers.base -> kiro_crew.context in a fresh interpreter did not finish within 60 s.
  • A hang inside an import writes nothing to stderr, so the child arms faulthandler.dump_traceback_later a few seconds before the bound (55 s of 60 s, 85 s of 90 s). A hang then puts its import stack in the assertion message. The partial stderr is decoded first, because on POSIX the re-raised TimeoutExpired carries bytes even with text=True.

A child that never finishes fails at half the test timeout, so pytest-timeout never kills the worker first. Measured here: a cold run of one order takes about 4.5 s.

Checked and left alone: the other fresh-interpreter import tests (test_agent_lifecycle_cycle.py, test_agent_import_hoist.py, test_perf_boot_path.py and the composition-contract probes) already wait 120 s or more.

Why not import every order in one interpreter: the test needs a clean sys.modules per order to prove there is no cycle, so each order needs its own interpreter.

Backwards compatibility

Compatible: test-only. No production code changes.

Removes nothing: both waits take the lost-run bound in place of a literal, and a timeout now names the import that hung.

Tests

Run on Linux (aarch64, Python 3.12) on this head, basetemp outside the repo:

  • Bound probe (a proof-only -p plugin printing default_timeout() inside each of the 4 cases): 90.0 s under --timeout=180 (the Windows shard's flag), 60.0 s under --timeout=120. The value comes from the published runner timeout, not the 60 s fallback.
  • Hang inside the import (a proof-only edit to kiro_crew/context.py, child only, writes partial-é to stderr and then sleeps in a named function), --timeout=20 so the bound is 10 s: all 4 cases fail by name in 42 s, and each message shows the decoded partial-é, the faulthandler dump with the hung function's frame in context.py, and no b'...'.
  • Forced condition (a proof-only -p plugin prepends a 35 s sleep to the child's -c code, as slow as the reported Windows runner), --timeout=120: the parent commit's import-order test fails 3/3 with a bare subprocess.TimeoutExpired ... timed out after 30 seconds; this head passes 3/3.
  • Determinism (testing-conventions D12) for the 4 changed cases: 20/20 green at -n0; 10/10 green at -n 4 --dist loadgroup --max-worker-restart=2 with both files as co-tenants (129 tests); both files in a shuffled order, seed 27909: 129 passed.
  • Both files plus test_dashboard_server_composition_contract.py::test_the_lazy_imports_stay_inside_the_functions_that_need_them: 130 passed.
  • Every test/test_*ratchet*.py plus test_agent_refactor_facade.py, test_spawn_audit.py and test_file_line_cap.py: 394 passed. test_flake_pattern_ratchet.py at -n0: 58 passed.
  • Black gate, isort, flake8, subprocess-encoding gate, comment-history gate (zero markers), Semgrep 1.78.0 with the repository rules on both files (0 findings): clean.

Manual verification

N/A — test-only change; the bound probe and the hang runs above are the verification.

Related Issues

Fixes #20782

Pattern harvest

Rule candidate: semgrep
Pattern: a test passes a fixed timeout= literal shorter than default_timeout() to subprocess.run on a cold interpreter start, and lets TimeoutExpired escape without naming what hung.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 15, 2026 11:02
@jeeshofone
jeeshofone requested a review from buluoray October 15, 2026 11:02
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 15, 2026
@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 1026ca3b46f23ba1d9aa104eade2afbe9b8b10b9 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 1026ca3b46f23ba1d9aa104eade2afbe9b8b10b9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 1026ca3b46f23ba1d9aa104eade2afbe9b8b10b9 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

Suggestions

  • Both files now carry the same timeout handling: arm faulthandler, catch TimeoutExpired, decode stderr, raise AssertionError. Move it into one helper in kiro_crew.testing (next to default_timeout) so the next fresh-interpreter test reuses it instead of copying a third time.

[DESIGN-REVIEWED] 1026ca3

Description read

[DESCRIPTION-READ] f142ee5d5aad931e49d200d9758890b5dcd37acfddcab64ad33bd43f4da259e7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 1026ca3b46f23ba1d9aa104eade2afbe9b8b10b9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1026ca3

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 1026ca3b46f23ba1d9aa104eade2afbe9b8b10b9 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1026ca3

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 15, 2026
@jeeshofone
jeeshofone force-pushed the fix/20782-import-order-subprocess-bound branch from 48849b6 to 61829bc Compare October 15, 2026 13:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
@jeeshofone
jeeshofone force-pushed the fix/20782-import-order-subprocess-bound branch from 61829bc to c592f01 Compare October 15, 2026 18:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
@jeeshofone
jeeshofone force-pushed the fix/20782-import-order-subprocess-bound branch from c592f01 to c6cd5b1 Compare October 15, 2026 19:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 15, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head c6cd5b1.

What I checked.

  • The change is test-only: test_agent_sdk_context_bridge.py and test_context_composition_contract.py. Both cold-subprocess import probes now take their bound from default_timeout() (half the running test's timeout) instead of fixed 30 s / 60 s values, so a slow cold interpreter on a Windows shard fails inside the runner's budget rather than at a hard-coded wall.
  • The faulthandler.dump_traceback_later call fires a few seconds before that bound with exit=False, so a real hang leaves a stack in stderr and the assertion names where the import stopped.
  • The TimeoutExpired handlers decode bytes stderr (POSIX re-raises it undecoded even with text=True) and raise an AssertionError that names the import order and the bound, with from None so the message is not buried under the subprocess traceback.
  • Locally on this head: the 9 tests in the two files pass. A probe that shrinks the bound below the cold-import time confirms the timeout path names the order, carries decoded stderr (no b'...') and, with a slightly longer bound, includes the faulthandler stack.
  • The new head is a rebase of c592f01: same patch, same added and removed lines.

CI on this head.
The run has finished red, and none of the failures is in the two files this PR touches:

  • Backend Tests 3.12 shards 1-8 and Windows shard 6 fail only on test_the_lazy_imports_stay_inside_the_functions_that_need_them (assert 84 == 83). That count is wrong on main too; #20930 fixed it on main at 13:35 PDT (20:35 UTC), after this run's merge ref was taken, so a re-run will not pick it up. Updating the branch onto current main should clear it.
  • Windows shard 2 fails on test_spawn_delete_queued.py::test_an_internal_parent_deletes_its_own_queued_spawn, which this diff does not touch and which passes locally. Worth watching on the next run.

Looks right to me. I'll approve once the required checks are green on a head that has the main fix for the lazy-import count.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 15, 2026
The context import-order test gave its fresh interpreter a fixed 30 s, which a
cold import on a loaded Windows runner exceeded. Pass default_timeout() and
turn TimeoutExpired into an assertion naming the import order. The facade
fresh-import test in the context composition contract gets the same bound and
a named failure.

A hang inside an import writes nothing to stderr, so each child arms
faulthandler.dump_traceback_later shortly before the bound; the assertion then
shows the stack the import stopped in. The partial stderr is decoded first,
because on POSIX the re-raised TimeoutExpired carries bytes even with text=True.

Fixes kirodotdev#20782
@jeeshofone
jeeshofone force-pushed the fix/20782-import-order-subprocess-bound branch from c6cd5b1 to 1026ca3 Compare October 16, 2026 13:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 16, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Design Review on c6cd5b1c4, answered on 1026ca3b4, a rebase of c6cd5b1c4 onto current main (git range-diff shows the patch unchanged).

  • Windows shard bound: checked, it is the published 180 s timeout, not the fallback

The Windows shard step in .github/workflows/ci.yml runs pytest with --timeout=180, so the Windows bound is 90 s, not 60 s. The Linux shards and the setup.cfg default run --timeout=120, which gives 60 s.
The rootdir conftest.py pytest_runtest_setup writes _runner_timeout_secs(item) (the timeout marker, then --timeout, then PYTEST_TIMEOUT, then the ini value) into kiro_crew.testing.wait.runner_timeout_secs before each test body runs. Both test files import kiro_crew.testing.wait at module level, so the module is loaded when the value is published.
Probe on a1269bd85, the same patch one rebase before 1026ca3b4 (a proof-only -p plugin printing the value inside the 4 cases): default_timeout() is 90.0 with runner_timeout_secs=180.0 under --timeout=180, and 60.0 with runner_timeout_secs=120.0 under --timeout=120. The 60 s fallback applies only when runner_timeout_secs is unset or not a positive finite number.
The PR body's table now gives both bounds, and the faulthandler dump point for each (55 s of 60 s, 85 s of 90 s).

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Re the review on c6cd5b1c4: rebased onto main at 12835a82c, so 1026ca3b4 carries #20930's lazy-import count fix. git range-diff shows the same single commit, patch unchanged.

CI on c6cd5b1c4, by cause

Timeout path, checked
The proofs below ran on a1269bd85, the same patch one rebase earlier; main's 4 commits since then touch neither test file, kiro_crew/testing/wait.py, conftest.py nor setup.cfg.

  • Decoded stderr on timeout: test/test_agent_sdk_context_bridge.py:85 and test/test_context_composition_contract.py:1779.
  • faulthandler.dump_traceback_later in the child: test/test_agent_sdk_context_bridge.py:72 and test/test_context_composition_contract.py:1758.
  • Hang mutation (a proof-only edit making kiro_crew.context's import hang in the child after writing partial-é to stderr), --timeout=20: 4/4 cases fail by name at the 10 s bound, each with the decoded partial-é, the faulthandler frame in context.py, and no b'...'.
  • Forced 35 s cold import: the parent commit's import-order test fails 3/3 with a bare TimeoutExpired after 30 s; this patch passes 3/3. D12: 20/20 at -n0, 10/10 at -n 4 with both files as co-tenants, shuffled order (seed 27909) 129 passed.

First-pass check: R1-R8 OK

  • AUTOSDE rules matching test/**/*.py:
    • no-test-side-effects: both children run with cwd=tmp_path and inherit the rootdir floor's per-test KIROCREW_HOME and KIROCREW_WORKSPACE through dict(os.environ). The import-order child's bytecode goes under tmp_path.
    • tests-are-deterministic: the literal bound is replaced by the suite's lost-run bound, default_timeout(), whose expiry fails by name. Nothing asserts an elapsed time.
    • a-ratchet-may-only-tighten: no ratchet or baseline is touched.
    • a-stopwatch-is-not-a-barrier: nothing is timed; the child's exit is the barrier.
    • an-async-test-respects-the-on-loop-lock: no async test in the diff.
    • an-invariant-line-needs-a-mutation-pin: the timeout path is pinned by the hang mutation above.
  • Gates on 1026ca3b4: black gate, isort, flake8, subprocess-encoding gate, comment-history gate (0 markers), Semgrep 1.78.0 with the repository rules on both files (0 findings), the flake-pattern ratchet (58 passed). Every test/test_*ratchet*.py with the facade, spawn-audit and file-line-cap guards passed on a1269bd85 (394 passed). mypy is not rerun: CI's step checks src/kiro_crew/ and this diff touches no file there.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 16, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 16, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving 1026ca3, a rebase of c6cd5b1 onto main: git range-diff shows the same single commit, with the same patch-id and the same changed lines in both test files.

On this head, both PR test files pass locally (129 tests), and the lazy-import count test that was red on c6cd5b1 passes now that #20930 is in the base. The two subprocess probes take their bound from default_timeout(), dump a stack through faulthandler before it, and fail by name with decoded stderr.

The Backend Tests (3.12, 4) red on this head was test_instances.py::TestOrphanForwarderReclaim::test_recorded_argv_fingerprint_survives_command_line_drift[False], a file this PR does not touch. It passed on rerun and is tracked in #21261.

All required checks are green on this head.

@cruisercohen
cruisercohen merged commit 9dff1e4 into kirodotdev:main Oct 16, 2026
151 of 153 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 16, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the two import-order subprocess tests use the default lost-run bound and dump a stack trace before it, so a hang names the import it stopped in; test-only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky on Windows: context import-order cycle test times out its cold subprocess

3 participants