Repository navigation
fix(e2e): pre-compile kiro_crew before the boot matrix so first boot is warm - #19668
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The warm-up compiles only Watch
Suggestions
[DESIGN-REVIEWED] bd3b19e Description read[DESCRIPTION-READ] 78353a78d5beac1673d8ad47be137d6163e55566fd4e0a69a38ac9e77fd2b548 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The compileall step fixes the cause. Two riders add little: the cold hint repeats the existing timeout advice, and the deque cap only bounds a buffer this PR adds. Not justified as shipped
What this change shipsInventory (5 items) — 2 justifiedIntent: stop the first e2e gateway boot on a cold runner from timing out. FIX.
[FIRST-PRINCIPLES-REVIEWED] bd3b19e Description read[DESCRIPTION-READ] 19a45c87fe1aa79b03d7ba5d5aa4972c09b21d2e7aad4c6299fca68529f5ca86 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings block this merge; one advisory diagnostic gap. FINDING — src/kiro_crew/testing/harness.py:492 — on child exit, the bytes from [OPUS-REVIEWED] bd3b19e Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
040ac6c to
fb48620
Compare
fb48620 to
18efb50
Compare
18efb50 to
71702a8
Compare
71702a8 to
bf5da3f
Compare
d3d0be6 to
e2f2c1d
Compare
|
/ai-review disposition gpt bf5da3f src/kiro_crew/testing/harness.py:532: fixed — stdout_received changed from unbounded list to deque(maxlen=_StdoutPump.TAIL_CHUNKS); dropped-chunk count added to diagnostic message; test added with _MockPump feeding TAIL_CHUNKS+1 entries to verify eviction and tail retention |
e2f2c1d to
7ddd1bb
Compare
7ddd1bb to
c4a6332
Compare
|
/ai-review disposition fable 7ddd1bb test/test_harness.py:326: fixed — replaced wall-clock timeouts with _DeadlineAfterEof mock clock; _MockPump.eof_served flag fires the deadline only after the EOF sentinel is returned, making both tests deterministic (D7/D8). _make_deadline_after_eof_clock() patches kiro_crew.testing.harness.time via unittest.mock.patch.object. |
c4a6332 to
b315ce5
Compare
|
/ai-review disposition fable 7ddd1bb test/test_harness.py:326: fixed — tests are now signal-based, not wall-clock-based. Product change: EOF on stdout without READY now raises immediately (not 'keep waiting until deadline'). Tests use _MockPump([]) for 'timeout fires' cases (no EOF, queue.Empty forever) and _MockPump([..., None]) + timeout=30 for diagnostic tests (EOF fires the error, timeout=30 is safety net). Wall-clock timing removed from both tests. |
b315ce5 to
b5354c7
Compare
|
/ai-review disposition first-principles b315ce5 src/kiro_crew/testing/harness.py: fixed — (3) restored 'continue' in the 'chunk is None' branch with explicit #9585 pin reference; (4) renamed eof_served→exhausted and added _make_deadline_after_eof_clock() consumer, removing dead code. Diagnostic tests (stdout/state, bounded deque) use fake clock: time.monotonic() returns 0 until pump.exhausted, then timeout+1 — deterministic, no wall-clock dependence. FakePopen([]) restored for the silent-alive timeout test. |
b5354c7 to
654e367
Compare
|
/ai-review disposition first-principles b315ce5: FP item 1 (compileall step, justified) — no action. FP item 2 (diagnostics, rides along) — declared in 'What changed' as incidental improvements separate from the Goal. FP item 3 (EOF pin reversed) — fixed: restored 'continue' with explicit #9585 cite; no GatewaySpawnError on stdout EOF. FP item 4 (eof_served dead code) — fixed: removed exhausted/eof_served entirely; tests now use _MockPump(proc=fake, exit_code=1) which sets proc.returncode after delivering all chunks, triggering the harness's existing exit-check branch (no fake clock, no patched time module). |
654e367 to
40d430e
Compare
…inate cold-start flake Root cause: the first gateway spawn on a freshly provisioned ubuntu-latest runner has no .pyc files and spends ~43 s compiling src/kiro_crew from disk before the gateway's own logging starts (stderr is empty throughout). The 60 s ready-line timeout fires on the first test; every subsequent test reuses the warm page cache and boots in ~20 s. Evidence (run 38339527809, job 115116247097): boot1 (cold): 62.3 s → timeout (FAILED) boot2 (warm): 19.5 s (PASSED) delta = 42.8 s — pure .pyc compilation overhead on first subprocess import Fix: add `python -m compileall -q src/kiro_crew` to the e2e-boot-matrix job after dependency installation. This pre-generates all bytecode files for the editable install so the first gateway spawn starts warm. The 60 s KIROCREW_HARNESS_READY_TIMEOUT is unchanged. Diagnostics (kept): the harness timeout error now includes the stdout tail, process-alive state, and — when both stdout and stderr are empty — a hint pointing at KIROCREW_HARNESS_READY_TIMEOUT. Tests: - test_ready_line_no_output_timeout_includes_cold_import_hint: empty-output timeout carries the cold-import hint and names KIROCREW_HARNESS_READY_TIMEOUT - test_ready_line_timeout_message_includes_stdout_and_process_state: timeout error includes stdout tail and process state; pump is pre-started and joined before the wait to avoid a thread-scheduling race (Opus finding) Closes #19667
40d430e to
bd3b19e
Compare
buluoray
left a comment
There was a problem hiding this comment.
Verdict: APPROVE at bd3b19efd7
What it fixes
test_gateway_boots_and_answers_health timed out at 60 s on the first boot of a cold Ubuntu runner. Later boots on the same runner were fine. The PR byte-compiles src/kiro_crew into the same PYTHONPYCACHEPREFIX tree that conftest._redirect_bytecode_cache uses (conftest.py:442-449) before the boot matrix starts. It also adds stdout and process state to the harness timeout and early-exit errors. No readiness timeout changes.
Does it hide a slow first boot that users would hit?
No. The problem it skips past already exists on main, and users don't hit it on a timed path:
- Desktop builds ship checked-hash pycs for the whole tree, and a coverage gate enforces it (
packaging/build-desktop.sh:636-669). Desktop users never compile on first boot. install.shusers dopip install -e(install.sh:593), so their first launch still compiles. That cost is the same before and after this PR. I found no installer step that boots the gateway against a deadline.- The boot matrix exists to prove the gateway boots on each OS (
ci.yml:3464-3480). It was never a cold-start performance gate, so a warm first boot doesn't remove a check anyone relied on.
What I verified
- The cache prefix the step computes (
ci.yml:3573-3579) matches the conftest one, including theXDG_CACHE_HOMEfallback. - CI at this head: E2E Gateway Boot is green on ubuntu-latest and windows-latest. The ubuntu first boot took 10.92 s.
- The new harness tests pass locally (
test/test_harness.py: 82 passed, 1 skipped). - Mutations, each run on its own:
- Forcing
cold_hintoff failstest_ready_line_no_output_timeout_includes_cold_import_hint. - Dropping stdout from the exit error fails
test_ready_line_timeout_message_includes_stdout_and_process_stateandtest_ready_line_stdout_received_is_bounded.
- Forcing
- Error paths are diagnostics only. The step's exit code comes from compileall, so a compile error turns the step red (fail-closed).
Non-blocking follow-ups
- The root-cause claim doesn't hold up against CI's own timing. The comment at
ci.yml:3567-3571(and the PR body) calls the 42.8 s delta "pure .pyc compilation". This step compiles the entire tree (83 MB, 2,319 files) in 8 s on the Linux runner and 9 s locally, so compilation can explain at most a fraction of the delta. The step probably also helps as a side effect, for example by warming the page cache. Please reword the comment so it doesn't record an unproven cause. - The size bound on stdout isn't tested. Replacing
deque(maxlen=_StdoutPump.TAIL_CHUNKS)with an unboundeddeque()leaves every test green, because the truncation note still fires once at length 256. Have the test assert the dropped-chunk count, or the retained length. - Possible user-side follow-up: run
compileallafter the editable install ininstall.sh, so source installs also get a warm first boot.
Not verified
- Comparing cold and warm import times of the gateway modules on this host: not run.
- The macOS leg: it doesn't run on pull requests.
- What actually makes up the cold-boot delta beyond compilation.
iamwhatever
left a comment
There was a problem hiding this comment.
This PR fixes the cold-runner timeout in test_gateway_boots_and_answers_health, which hit 62 s on boot 1 against a 19.5 s boot 2. It adds a compileall -q src/kiro_crew step to e2e-boot-matrix that writes into the same PYTHONPYCACHEPREFIX tree conftest._redirect_bytecode_cache sets for the pytest session. The harness also gets a bounded pre-READY stdout tail in its spawn errors. I checked that the fix targets the right cache. The step runs after run-as-runner under the job's ci-shell default on Linux, so it sees the same user and HOME as pytest. The gateway env is layered over os.environ (harness.py ~L1030), so the child inherits the prefix. cache_from_source makes the relative src/kiro_crew path absolute under a prefix, so the mirror paths match the editable import. The bash-quoted python -c holds no $ or backtick, so it is safe on Windows and macOS too. The Design lane's "dependencies stay cold" watch is answered by this head's own run (job 115220943068): the compile step took about 8 s and the first boot test passed in about 12 s, well under the 60 s window. The harness change only adds diagnostics. The deque is capped at _StdoutPump.TAIL_CHUNKS, and the tests end on process exit, not wall-clock time. One advisory for later: the bytes pump.drain_until_eof returns at exit go only to _scan and never into stdout_received, so a crash line printed just before exit can be missing from the new stdout section (Opus finding, harness.py:492). The PR is mergeable/clean, approved on this head, and checks are green. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: Stop
test_gateway_boots_and_answers_healthfrom timing out on the first boot of a cold CI runner.test_gateway_boots_and_answers_healthfails on Ubuntu with:Observed on 6 independent branches and on
main(run 38339527809, job 115116247097).Why it matters
Every cold-runner failure triggers a manual rerun. Six branches hit it in one day. It erodes CI signal and wastes queue time.
Not a goal
What changed (motivation → approach → change)
Root cause (with evidence).
From job 115116247097 (run 38339527809, cold fail):
The delta between boot 1 and boot 2 is 42.8 s of pure
.pyccompilation. Python is still reading and compilingsrc/kiro_crewoff disk (slack/gateway.py~840 KB,dashboard/chat_runner.py~1.1 MB, etc.) before the gateway's own logging starts — that is why stderr is empty for the whole window.Fix. Add
python -m compileall -q src/kiro_crewine2e-boot-matrixafterrun-as-runner, passingPYTHONPYCACHEPREFIXmatchingconftest._redirect_bytecode_cache's path. This pre-generates all.pycfiles for the editable install so the first gateway spawn starts warm. The 60 sKIROCREW_HARNESS_READY_TIMEOUTis unchanged.Incidental harness improvements (rides along; do not change the Goal):
stdout_receivedin_wait_for_ready_linechanged from an unboundedlisttodeque(maxlen=_StdoutPump.TAIL_CHUNKS)with a dropped-chunk counter. A gateway that emits megabytes of stdout before failing would previously retain the whole stream before displaying only 2 000 characters. (Anchored to thea-bound-bounds-every-field-it-retainsrule; GPT finding upheld by adjudication.)KIROCREW_HARNESS_READY_TIMEOUT.exited before emitting) also includes the stdout tail.Backwards compatibility
Compatible: no public API changed. The compileall step is additive; the harness error messages are richer.
Removes nothing: no path, field or key removed.
Tests
test_ready_line_no_output_timeout_includes_cold_import_hint— empty-output timeout carries the cold-import hint and namesKIROCREW_HARNESS_READY_TIMEOUT.test_ready_line_stdout_received_is_bounded—_MockPumpfeeds TAIL_CHUNKS+1 entries then signals process exit; the exit-error message carries the truncation note and the last line.test_ready_line_timeout_message_includes_stdout_and_process_state— process-exit error carries stdout content.Both new tests use process-exit termination:
_MockPumpsetsproc.returncodeafter all items are delivered, andtimeout=30is only a safety net. This is deterministic (D7/D8) without wall-clock dependence or fake clocks.Manual verification
N/A — the fix is a CI step and diagnostic message change. Evidence is the boot-time numbers above from job 115116247097.
Screenshots / video
N/A — no UI change.
Related Issues
Closes #19667
Pattern harvest
Not generalizable: the flake is specific to the first subprocess spawn on a cold Ubuntu runner with no
.pyccache for an editable install.Checklist
Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.