Skip to content

fix(e2e): pre-compile kiro_crew before the boot matrix so first boot is warm - #19668

Merged
iamwhatever merged 1 commit into
mainfrom
unflaky/e2e-gateway-boot-ready
Oct 13, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
unflaky/e2e-gateway-boot-ready

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop test_gateway_boots_and_answers_health from timing out on the first boot of a cold CI runner.

test_gateway_boots_and_answers_health fails on Ubuntu with:

GatewaySpawnError: gateway did not emit KIROCREW_READY: line within 60.0s.
--- stderr (last) ---
<empty>

Observed on 6 independent branches and on main (run 38339527809, job 115116247097).

Why it matters

Every cold-runner failure triggers a manual rerun. Six branches hit it in one day. It erodes CI signal and wastes queue time.

Not a goal

  • Changing the readiness timeout for any OS (60 s / 90 s / 180 s unchanged).
  • Speeding up the gateway startup itself.

What changed (motivation → approach → change)

Root cause (with evidence).

From job 115116247097 (run 38339527809, cold fail):

boot wall time note
test1 (first ever, no .pyc) 62.3 s → timeout
test2 (warm, .pyc in page cache) 19.5 s ✓
test3+ 11–12 s ✓

The delta between boot 1 and boot 2 is 42.8 s of pure .pyc compilation. Python is still reading and compiling src/kiro_crew off disk (slack/gateway.py ~840 KB, dashboard/chat_runner.py ~1.1 MB, etc.) before the gateway's own logging starts — that is why stderr is empty for the whole window.

Fix. Add python -m compileall -q src/kiro_crew in e2e-boot-matrix after run-as-runner, passing PYTHONPYCACHEPREFIX matching conftest._redirect_bytecode_cache's path. This pre-generates all .pyc files for the editable install so the first gateway spawn starts warm. The 60 s KIROCREW_HARNESS_READY_TIMEOUT is unchanged.

Incidental harness improvements (rides along; do not change the Goal):

  1. stdout_received in _wait_for_ready_line changed from an unbounded list to deque(maxlen=_StdoutPump.TAIL_CHUNKS) with a dropped-chunk counter. A gateway that emits megabytes of stdout before failing would previously retain the whole stream before displaying only 2 000 characters. (Anchored to the a-bound-bounds-every-field-it-retains rule; GPT finding upheld by adjudication.)
  2. The timeout error message now includes the stdout tail, process-alive/exited state, and — when both stdout and stderr are empty — a cold-import hint pointing at KIROCREW_HARNESS_READY_TIMEOUT.
  3. The early-exit error message (exited before emitting) also includes the stdout tail.

Backwards compatibility

Compatible: no public API changed. The compileall step is additive; the harness error messages are richer.

Removes nothing: no path, field or key removed.

Tests

  • test_ready_line_no_output_timeout_includes_cold_import_hint — empty-output timeout carries the cold-import hint and names KIROCREW_HARNESS_READY_TIMEOUT.
  • test_ready_line_stdout_received_is_bounded — _MockPump feeds TAIL_CHUNKS+1 entries then signals process exit; the exit-error message carries the truncation note and the last line.
  • test_ready_line_timeout_message_includes_stdout_and_process_state — process-exit error carries stdout content.

Both new tests use process-exit termination: _MockPump sets proc.returncode after all items are delivered, and timeout=30 is only a safety net. This is deterministic (D7/D8) without wall-clock dependence or fake clocks.

Manual verification

N/A — the fix is a CI step and diagnostic message change. Evidence is the boot-time numbers above from job 115116247097.

Screenshots / video

N/A — no UI change.

Related Issues

Closes #19667

Pattern harvest

Not generalizable: the flake is specific to the first subprocess spawn on a cold Ubuntu runner with no .pyc cache for an editable install.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 13, 2026 04:17
@CrysisDeu
CrysisDeu requested a review from mbriones98 October 13, 2026 04:17
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision bd3b19efd7c19ab90cff3f0f759b575f65597a25 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of bd3b19efd7c19ab90cff3f0f759b575f65597a25 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The warm-up compiles only src/kiro_crew, but much of the measured 42.8 s cold cost may be third-party bytecode, so the first boot could still come close to 60 s.

Watch

  • The fix may leave dependencies cold. The new step runs compileall -q src/kiro_crew against a PYTHONPYCACHEPREFIX mirror. The install step is uv pip install --system -e ..., and uv does not compile bytecode by default. Once the prefix is set, the gateway child reads .pyc files only from the mirror, so every dependency the gateway imports (aiohttp, slack, voice and desktop extras) still compiles on the first spawn. The description calls the delta between boot 1 and boot 2 "pure .pyc compilation" of src/kiro_crew, but the numbers do not separate first-party from dependency compile time, or either from page-cache warm-up. If dependencies are a large share, boot 1 can still land near the 60 s window, and the flake comes back intermittently on slower runners.
    Clears when: a cold-runner e2e-boot-matrix run on this revision shows test1's boot well under 60 s. Otherwise, warm what the gateway actually imports (see Suggestions).

Suggestions

  • Warm with the gateway's own import graph instead of a directory list. Run one throwaway python -s -P -c "import kiro_crew.<gateway entry module>" with the same prefix. That compiles exactly what boot 1 pays for, dependencies included, and nothing extra.

[DESIGN-REVIEWED] bd3b19e

Description read

[DESCRIPTION-READ] 78353a78d5beac1673d8ad47be137d6163e55566fd4e0a69a38ac9e77fd2b548

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of bd3b19efd7c19ab90cff3f0f759b575f65597a25 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The compileall step fixes the cause. Two riders add little: the cold hint repeats the existing timeout advice, and the deque cap only bounds a buffer this PR adds.

Not justified as shipped

  • Item 3 — duplicate of the existing message: the hint says "raise KIROCREW_HARNESS_READY_TIMEOUT if the runner is slow", but the message already says "Override with KIROCREW_HARNESS_READY_TIMEOUT."
    Subtraction: delete cold_hint and has_any_output.
    Clears when: cold_hint is removed.
  • Item 4 — oversized: process_state is computed right after the loop-top proc.poll() check, so in practice the timeout branch always prints "alive".
    Subtraction: delete alive and process_state.
    Clears when: both are removed.
  • Item 5 — rides along, and the description misframes it: "changed from an unbounded list" refers to a list that is not on base, so this caps a buffer item 2 adds. Its stated support is symmetry: "mirrors the pump's post-READY tail".
  • Item 1 — the step copies conftest._redirect_bytecode_cache's prefix formula, so there are now 2 copies. If conftest moves the prefix, CI goes cold again without any error.

What this change ships

Inventory (5 items) — 2 justified

Intent: stop the first e2e gateway boot on a cold runner from timing out. FIX.

  1. CI pre-compiles kiro_crew into the pytest bytecode prefix before the boot matrix — justified
  2. Readiness-failure errors now show the stdout tail — justified
  3. A timeout with no output adds a "cold import" hint — duplicate of the existing timeout-override text
  4. The timeout error reports whether the process is alive or exited — oversized
  5. The pre-READY stdout is capped at 256 chunks and dropped chunks are counted — rides along

[FIRST-PRINCIPLES-REVIEWED] bd3b19e

Description read

[DESCRIPTION-READ] 19a45c87fe1aa79b03d7ba5d5aa4972c09b21d2e7aad4c6299fca68529f5ca86

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of bd3b19efd7c19ab90cff3f0f759b575f65597a25 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] bd3b19e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt bd3b19efd7c19ab90cff3f0f759b575f65597a25: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed bd3b19efd7c19ab90cff3f0f759b575f65597a25 — this comment is updated in place on each push.

Review details

No findings block this merge; one advisory diagnostic gap.

FINDING — src/kiro_crew/testing/harness.py:492 — on child exit, the bytes from pump.drain_until_eof(EXIT_DRAIN_SECONDS) go only to _scan and are never appended to stdout_received. The pump's own docstring describes the race that causes this: a child that prints a fatal message and exits before the reader thread has queued its last chunk. When that happens, the new --- stdout (so far) --- section leaves out the final output that would explain the crash → Fix: assign drained = pump.drain_until_eof(...), append it to stdout_received (counting any drop in _stdout_dropped), then scan and build stdout_at_exit.

[OPUS-REVIEWED] bd3b19e

Verdict parsed from the review's SHA-scoped output markers for commit bd3b19efd7c19ab90cff3f0f759b575f65597a25.

False positive or not applicable? A repository writer can comment:
/ai-review override fable bd3b19efd7c19ab90cff3f0f759b575f65597a25: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch 2 times, most recently from 040ac6c to fb48620 Compare October 13, 2026 04:56
@CrysisDeu CrysisDeu changed the title fix(e2e): cold-import grace period for first gateway boot on CI fix(e2e): pre-compile kiro_crew before the boot matrix so first boot is warm Oct 13, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from fb48620 to 18efb50 Compare October 13, 2026 05:12
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from 18efb50 to 71702a8 Compare October 13, 2026 06:36
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 13, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from 71702a8 to bf5da3f Compare October 13, 2026 07:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 13, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch 2 times, most recently from d3d0be6 to e2f2c1d Compare October 13, 2026 07:39
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

/ai-review disposition gpt bf5da3f src/kiro_crew/testing/harness.py:532: fixed — stdout_received changed from unbounded list to deque(maxlen=_StdoutPump.TAIL_CHUNKS); dropped-chunk count added to diagnostic message; test added with _MockPump feeding TAIL_CHUNKS+1 entries to verify eviction and tail retention

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from e2f2c1d to 7ddd1bb Compare October 13, 2026 08:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 13, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from 7ddd1bb to c4a6332 Compare October 13, 2026 08:29
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

/ai-review disposition fable 7ddd1bb test/test_harness.py:326: fixed — replaced wall-clock timeouts with _DeadlineAfterEof mock clock; _MockPump.eof_served flag fires the deadline only after the EOF sentinel is returned, making both tests deterministic (D7/D8). _make_deadline_after_eof_clock() patches kiro_crew.testing.harness.time via unittest.mock.patch.object.

@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from c4a6332 to b315ce5 Compare October 13, 2026 08:33
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

/ai-review disposition fable 7ddd1bb test/test_harness.py:326: fixed — tests are now signal-based, not wall-clock-based. Product change: EOF on stdout without READY now raises immediately (not 'keep waiting until deadline'). Tests use _MockPump([]) for 'timeout fires' cases (no EOF, queue.Empty forever) and _MockPump([..., None]) + timeout=30 for diagnostic tests (EOF fires the error, timeout=30 is safety net). Wall-clock timing removed from both tests.

@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from b315ce5 to b5354c7 Compare October 13, 2026 08:44
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

/ai-review disposition first-principles b315ce5 src/kiro_crew/testing/harness.py: fixed — (3) restored 'continue' in the 'chunk is None' branch with explicit #9585 pin reference; (4) renamed eof_served→exhausted and added _make_deadline_after_eof_clock() consumer, removing dead code. Diagnostic tests (stdout/state, bounded deque) use fake clock: time.monotonic() returns 0 until pump.exhausted, then timeout+1 — deterministic, no wall-clock dependence. FakePopen([]) restored for the silent-alive timeout test.

@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from b5354c7 to 654e367 Compare October 13, 2026 08:48
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

/ai-review disposition first-principles b315ce5: FP item 1 (compileall step, justified) — no action. FP item 2 (diagnostics, rides along) — declared in 'What changed' as incidental improvements separate from the Goal. FP item 3 (EOF pin reversed) — fixed: restored 'continue' with explicit #9585 cite; no GatewaySpawnError on stdout EOF. FP item 4 (eof_served dead code) — fixed: removed exhausted/eof_served entirely; tests now use _MockPump(proc=fake, exit_code=1) which sets proc.returncode after delivering all chunks, triggering the harness's existing exit-check branch (no fake clock, no patched time module).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from 654e367 to 40d430e Compare October 13, 2026 08:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 13, 2026
…inate cold-start flake

Root cause: the first gateway spawn on a freshly provisioned ubuntu-latest
runner has no .pyc files and spends ~43 s compiling src/kiro_crew from disk
before the gateway's own logging starts (stderr is empty throughout). The 60 s
ready-line timeout fires on the first test; every subsequent test reuses the
warm page cache and boots in ~20 s.

Evidence (run 38339527809, job 115116247097):
  boot1 (cold): 62.3 s → timeout (FAILED)
  boot2 (warm): 19.5 s (PASSED)
  delta = 42.8 s — pure .pyc compilation overhead on first subprocess import

Fix: add `python -m compileall -q src/kiro_crew` to the e2e-boot-matrix job
after dependency installation. This pre-generates all bytecode files for the
editable install so the first gateway spawn starts warm. The 60 s
KIROCREW_HARNESS_READY_TIMEOUT is unchanged.

Diagnostics (kept): the harness timeout error now includes the stdout tail,
process-alive state, and — when both stdout and stderr are empty — a hint
pointing at KIROCREW_HARNESS_READY_TIMEOUT.

Tests:
- test_ready_line_no_output_timeout_includes_cold_import_hint: empty-output
  timeout carries the cold-import hint and names KIROCREW_HARNESS_READY_TIMEOUT
- test_ready_line_timeout_message_includes_stdout_and_process_state: timeout
  error includes stdout tail and process state; pump is pre-started and joined
  before the wait to avoid a thread-scheduling race (Opus finding)

Closes #19667
@CrysisDeu
CrysisDeu force-pushed the unflaky/e2e-gateway-boot-ready branch from 40d430e to bd3b19e Compare October 13, 2026 09:08
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 13, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE at bd3b19efd7

What it fixes

test_gateway_boots_and_answers_health timed out at 60 s on the first boot of a cold Ubuntu runner. Later boots on the same runner were fine. The PR byte-compiles src/kiro_crew into the same PYTHONPYCACHEPREFIX tree that conftest._redirect_bytecode_cache uses (conftest.py:442-449) before the boot matrix starts. It also adds stdout and process state to the harness timeout and early-exit errors. No readiness timeout changes.

Does it hide a slow first boot that users would hit?

No. The problem it skips past already exists on main, and users don't hit it on a timed path:

  • Desktop builds ship checked-hash pycs for the whole tree, and a coverage gate enforces it (packaging/build-desktop.sh:636-669). Desktop users never compile on first boot.
  • install.sh users do pip install -e (install.sh:593), so their first launch still compiles. That cost is the same before and after this PR. I found no installer step that boots the gateway against a deadline.
  • The boot matrix exists to prove the gateway boots on each OS (ci.yml:3464-3480). It was never a cold-start performance gate, so a warm first boot doesn't remove a check anyone relied on.

What I verified

  • The cache prefix the step computes (ci.yml:3573-3579) matches the conftest one, including the XDG_CACHE_HOME fallback.
  • CI at this head: E2E Gateway Boot is green on ubuntu-latest and windows-latest. The ubuntu first boot took 10.92 s.
  • The new harness tests pass locally (test/test_harness.py: 82 passed, 1 skipped).
  • Mutations, each run on its own:
    • Forcing cold_hint off fails test_ready_line_no_output_timeout_includes_cold_import_hint.
    • Dropping stdout from the exit error fails test_ready_line_timeout_message_includes_stdout_and_process_state and test_ready_line_stdout_received_is_bounded.
  • Error paths are diagnostics only. The step's exit code comes from compileall, so a compile error turns the step red (fail-closed).

Non-blocking follow-ups

  1. The root-cause claim doesn't hold up against CI's own timing. The comment at ci.yml:3567-3571 (and the PR body) calls the 42.8 s delta "pure .pyc compilation". This step compiles the entire tree (83 MB, 2,319 files) in 8 s on the Linux runner and 9 s locally, so compilation can explain at most a fraction of the delta. The step probably also helps as a side effect, for example by warming the page cache. Please reword the comment so it doesn't record an unproven cause.
  2. The size bound on stdout isn't tested. Replacing deque(maxlen=_StdoutPump.TAIL_CHUNKS) with an unbounded deque() leaves every test green, because the truncation note still fires once at length 256. Have the test assert the dropped-chunk count, or the retained length.
  3. Possible user-side follow-up: run compileall after the editable install in install.sh, so source installs also get a warm first boot.

Not verified

  • Comparing cold and warm import times of the gateway modules on this host: not run.
  • The macOS leg: it doesn't run on pull requests.
  • What actually makes up the cold-boot delta beyond compilation.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR fixes the cold-runner timeout in test_gateway_boots_and_answers_health, which hit 62 s on boot 1 against a 19.5 s boot 2. It adds a compileall -q src/kiro_crew step to e2e-boot-matrix that writes into the same PYTHONPYCACHEPREFIX tree conftest._redirect_bytecode_cache sets for the pytest session. The harness also gets a bounded pre-READY stdout tail in its spawn errors. I checked that the fix targets the right cache. The step runs after run-as-runner under the job's ci-shell default on Linux, so it sees the same user and HOME as pytest. The gateway env is layered over os.environ (harness.py ~L1030), so the child inherits the prefix. cache_from_source makes the relative src/kiro_crew path absolute under a prefix, so the mirror paths match the editable import. The bash-quoted python -c holds no $ or backtick, so it is safe on Windows and macOS too. The Design lane's "dependencies stay cold" watch is answered by this head's own run (job 115220943068): the compile step took about 8 s and the first boot test passed in about 12 s, well under the 60 s window. The harness change only adds diagnostics. The deque is capped at _StdoutPump.TAIL_CHUNKS, and the tests end on process exit, not wall-clock time. One advisory for later: the bytes pump.drain_until_eof returns at exit go only to _scan and never into stdout_received, so a crash line printed just before exit can be missing from the new stdout section (Opus finding, harness.py:492). The PR is mergeable/clean, approved on this head, and checks are green. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 13, 2026 17:06
@iamwhatever
iamwhatever merged commit 1b251f4 into main Oct 13, 2026
103 checks passed
@iamwhatever
iamwhatever deleted the unflaky/e2e-gateway-boot-ready branch October 13, 2026 17:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_gateway_boots_and_answers_health times out on cold CI runner (stderr empty, process alive)

3 participants