Repository navigation
feat(ci): cross-os end-to-end testing framework - #9585
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS A measured gap (#8117 plus 13 post-break fixes), closed at the right layer — real boots per OS, with pins that can go red and residuals stated, not claimed closed. Suggestions
[DESIGN-REVIEWED] d22ce97 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of I've now read the contract, the intent file, and the patch (workflows, scripts, conftest, product code, new pod backend, gap lists, and the modified test pins — the large test diffs in First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: make CI run the product for real on all three OSes — boot gateways, install artifacts, run pods — so platform-only startup breaks like #8117 (provenance: revert 56f67aa plus 13 counted follow-up fixes) cannot land green. ADDITION. More than 10 observable differences; the 10 most visible kept.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 28130e6 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
a83fa88 to
c0f7e02
Compare
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- scripts/smoke-windows-install.ps1:355 -- False positive or not applicable? A repository writer can comment: |
c0f7e02 to
756400f
Compare
756400f to
0d76f11
Compare
0d76f11 to
787649f
Compare
UX Review (Fable 5) — ✅ PASSUX-level review of The PR is entirely backend, CI, and test infrastructure: a Windows Task Scheduler pod backend, an E2E boot matrix, harness changes, and bounded hostname resolution. No UX-Verdict: PASS No user-visible surface changes — CI/backend only; the new CLI refusal copy states cause and next step, and the terminal panel loses noise, not affordance. [UX-REVIEWED] d22ce97 |
|
Re GPT 5.6 finding
"Store it in sandbox-hidden host state" presumes a sandbox that Windows does not have; the doc in |
|
Reasoning kept for the record. The finding this addressed is absent from the lane's output on the current head, so this is commentary rather than a disposition record. Disposition — GPT 5.6 F1
Leg 2 (successor-record failure) — fixed, and it was mine. The adjudicator confirmed this one end to end, including that the code's own remedy was the harm trigger: on an Leg 1 (the record is deleted by something with write access) — no safe fix exists at this layer. The adjudicator did not confirm it ("would require confirming the sandbox does not fence The suggested remedy ("refuse teardown when task liveness is unknown") was implemented and then withdrawn, because every available liveness signal is worse than the gap:
Storing the record outside agent-writable state is the real fix and it is a platform-posture change (where pod state lives, on every platform), not something to land inside a CI-framework PR. Recorded as a residual rather than silently closed. |
GPT 5.6 Review — 🔴 changes requested (blocking)GPT 5.6 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/pod/windows.py:1049 -- Restart tracking remains anchored to the original process |
|
Reasoning kept for the record. The finding this addressed is absent from the lane's output on the current head, so this is commentary rather than a disposition record. Disposition — GPT 5.6, unit-prefix path escape
Every claim below was checked in code, not taken from the review. It is pre-existing, and it is not Windows-specific. The escape is self-directed and crosses no boundary. It requires the operator running What the finding did catch is a documentation gap, and that is fixed. If the repo does want that surface constrained, the shape is a validator at config load covering |
|
Reasoning kept for the record. The finding this addressed is absent from the lane's output on the current head, so this is commentary rather than a disposition record. Disposition — GPT 5.6, handoff published after liveness disappears
The finding is correct, and it is a refinement of a fix from two pushes ago. What changed in response. The marker is now the FIRST statement after the reap: measured, there are zero executable statements between Why the remainder is not closable at this layer. Publishing the marker BEFORE the wait would make it present for the pod's entire lifetime, which defeats the freshness bound — and that bound is not decoration: a supervisor killed mid-handoff cannot retract its own marker, so without it a dead supervisor would make Proportion, for the record. Two pushes ago this window was the FULL The three findings before this one in the same area were fixed outright (the successor that could not be recorded, the anchor that did not advance, the handoff window itself), so this is not a pattern of deferral — it is the point where the next increment costs a contract change. |
|
GPT 5.6 Review —
|
`kirocrew pod` had two backends (systemd --user on Linux, launchd on macOS) and refused on Windows. This adds the third so a Windows developer can pod a worktree, and leaves the Linux and macOS paths byte-identical: every dispatch site gained an `elif IS_WINDOWS:` beside the existing `if IS_MACOS:`, and no shared branch changed. Task Scheduler, not a service. A pod is per-user and disposable, so `sc.exe create` is wrong twice over: it needs an administrator right and it installs a machine-wide LocalSystem service, which would stop the pod running as the user whose data it is isolating. `schtasks.exe` creates a task in the caller's own namespace with no elevation, matching `systemd --user` and launchd's `gui/<uid>`. A detached supervised process was the documented fallback and is not needed. Five consequences, each stated rather than papered over: - A task carries no environment block, so the pod plane the CLI resolved would be lost. The task's action is a generated `.cmd` that sets the plane from the same `config.environment_vars` selection both other backends serialise, then re-enters `kirocrew pod _run <name>`. Boot logic stays in Python. A plane path holding a double quote or a newline is refused at `pod up`, because cmd.exe cannot express it. - There is no restart policy, so a crashed pod stays down. That removes launchd's restart-loop hazard entirely (a terminal refusal keeps its honest exit code, no exit-0 translation) but means the crash signal `pod up` waits on has to be recorded: the wrapper writes the boot's exit code beside the task and `unit_state` reads it. - `schtasks /Query` reports no pid, and Windows has no `exec` (CPython's `os.execve` there spawns and terminates the caller). So `boot` supervises the gateway as the wrapper's child and records its pid plus its creation-time identity, which restores the invariant `port_owner` rests on: the recorded pid IS the process that bound the port. It stays independent of the gateway's own PID sidecar. - `schtasks` output is localized in both its CSV headers and its status words, so this backend never parses it. Keying on `Status == "Running"` would report every pod down on a non-English host, and that is the fail-OPEN direction, where teardown deletes a live pod's HOME. `schtasks` is used only where its exit code is the answer. - There are no cgroups, so the memory and CPU ceiling is NOT enforced. Same gap as macOS, and a weaker knob that read as the 4G cap is deliberately not emitted. `port_owner` no longer refuses non-POSIX hosts. That gate was unsatisfiable rather than strict: `pod up` mints a token and `mint_token` requires positive proof, so a healthy Windows pod could never earn its own credential. Both facts the proof needs answer on win32 now, and the freshness rule alone decides, on every platform. `pod api` still refuses on Windows, and that is fail-closed rather than a gap: its authenticated request travels over the pod's private AF_UNIX dashboard socket with no TCP fallback, and CPython there has no `AF_UNIX`. Adding a fallback is exactly what would hand a token to whatever answered the pod's released port. Docs updated in the same commit: the pod README's Platform section (now a three-backend capability table), the pod package docstring, the `cli.md` spec row that claimed Linux-only, the Windows install guide's per-feature table, and two `platform-compat.md` rows (no `exec` on Windows, and never parsing a Windows system tool's localized output). Tests: `test/test_pod_windows.py` covers rendering, the pid record's start-time binding, the `unit_state` mapping, the pinned `schtasks` argv, the teardown contract and the runtime dispatch on every platform, plus four real-`schtasks` tests behind `skipif(not IS_WINDOWS)`. The file is collected on the win32 CI leg with no mocks of `sys.platform` there. Follow-up, deliberately not in this commit: a real-pod win32 boot test (`pod up --seed minimal` against a scratch KIROCREW_HOME) plus a CI canary step, and applying a Job object ceiling to the pod's tree. (cherry picked from commit f0e54e18889b2eced45b30ec60a7a50d8a80327e)
Nothing in CI booted a gateway on Windows or macOS, which is how #8117 shipped a Windows startup break with every unit shard green. This adds the layers that run the product, not a mock of it, on all three OSes. Gateway boot matrix. `test/e2e/test_gateway_boot_matrix.py` boots a real gateway per test against the packaged fake ACP backend on ubuntu-latest, macos-15 and windows-latest (`e2e-boot-matrix` job), asserts health, one completed prompt turn, both sandbox tiers and a clean teardown, and greps the pass count so a skip cannot go green. `testing/harness.py` reads the READY line through a reader thread and tears down via a process-tree kill, so it runs on Windows and `test_harness.py` is collected there. Windows fail-closed contract. `test/test_windows_fail_closed_optin.py` pins, on a real win32 host with real filesystem state, that the official kiro backend delegates with no settings file present and that an unclassified spawn fails closed until `sandbox_allow_unsandboxed_exec` is set. `backend-test-windows-fail-closed` runs it by node id. Windows installer smoke. `scripts/smoke-windows-install.ps1` installs the built NSIS artifact on a clean runner, boots the installed gateway, and uninstalls; wired into build-windows.yml after the artifact upload. macOS full suite. `backend-test-macos` moves from the deprecated macos-14 label and a 51-file glob to macos-15 running the whole suite in 3 duration-balanced shards, with `test/macos-expected-failures.txt` and `test/macos-collect-ignore.txt` applied by the same conftest mechanism as the Windows lists. Pod scenario suite. `test/e2e/scenarios/` drives settings persistence, cron fire, a tool turn, service-unit rendering and a wheel install against a real `kirocrew pod`, nightly on ubuntu-latest and macos-15 with `systemd --user` via loginctl linger. Pods can now be handed the fake backend through `KIROCREW_POD_KIRO_BIN`. Specs and CI docs are updated alongside each layer.
Summary
CI had no job that booted a gateway on Windows or macOS. That is how PR #8117 shipped a Windows startup break with every unit shard green, reverted in 56f67aa43. Since 2026-07-01, 13 more Windows-only fixes landed after the break reached main, 5 of them the same lock-file truncation class and 4 the same junction class. Unit tests were modelling the platform instead of running on it.
This PR adds the end-to-end layers that run the product on all three OSes. Two commits.
Commit 1:
feat(pod): add a windows task-scheduler backendkirocrew podgains a Windows backend (src/kiro_crew/pod/windows.py) overschtasks.exe, the per-user, no-elevation equivalent ofsystemd --userand launchd. Linux and macOS dispatch is unchanged. Limits are documented insrc/kiro_crew/pod/README.md: no env block on a task (a generated.cmdre-enterskirocrew pod _run), no restart policy, no cgroup ceiling, andpod apirefuses on Windows because CPython there has noAF_UNIX.test/test_pod_windows_boot.pyboots a real pod on the Windows leg (pod-boot-windowsjob, pass count grepped), and the gateway child runs under the same Job object resource ceiling the cgroup path applies on Linux, installed on aCREATE_SUSPENDEDprocess before its first instruction.Commit 2:
feat(ci): add a cross-os end-to-end testing frameworktest/e2e/test_gateway_boot_matrix.py: real gateway per test against the packaged fake ACP backend; health, one completed prompt turn, both sandbox tiers, clean teardown; pass count greppede2e-boot-matrixjob on ubuntu-latest, macos-15, windows-latesttesting/harness.pyreads READY via a reader thread and tears down via a process-tree kill, so it runs on Windows;test_harness.pyis collected there againtest/test_windows_fail_closed_optin.py: on a real win32 host, the official kiro backend delegates with no settings file present, and an unclassified spawn fails closed untilsandbox_allow_unsandboxed_execis set (the exact branch #8117 broke)backend-test-windows-fail-closedjobscripts/smoke-windows-install.ps1: silent install of the built NSIS artifact on a clean runner, boot the installed gateway, health, uninstallbuild-windows.ymlafter artifact upload (nightly and dispatch);build.yml's PR-time installer job now installs the real wheel into a python-build-standalone runtime and boots the installed gateway against the fake backend, so-SkipGatewayValidationis gonebackend-test-macosmoves from the deprecatedmacos-14label and a 51-file glob tomacos-15running the whole suite in 3 shards, withtest/macos-expected-failures.txtandtest/macos-collect-ignore.txtthrough the same conftest mechanism as the Windows listsci.ymltest/e2e/scenarios/: settings persistence, cron fire, a tool turn, service-unit rendering, wheel install, against a realkirocrew pod; pods accept the fake backend viaKIROCREW_POD_KIRO_BINpod-scenariosjob innightly.ymlon ubuntu-latest and macos-15,systemd --uservia loginctl lingerThe OS gap lists changed from
skipto strictxfailDeclaring this because it is a semantic change to an existing mechanism, not just a
second copy of it.
test/windows-expected-failures.txtand the newtest/macos-expected-failures.txtboth call themselves burn-down backlogs whoseentries should be deleted as gaps close — but a
skipnever executes, so nothingcould ever report that a gap HAD closed and the lists could only grow. Under
xfail(strict=True)a listed test that still fails is green (the gap is known) andone that starts PASSING fails the job with XPASS, which is the signal to delete the
line, delivered by the run that earned it. The first run under it found 23 entries
whose gaps had already closed.
Two consequences worth stating plainly. Listed tests now RUN, so the Windows shards
do strictly more work than before. And a listed test that HANGS must never be added
to either list, because executing it would hang the shard rather than skip it.
Plain pytest throughout. No BDD or infra-assertion framework: the repo's isolation, timeout and sharding story is pytest-shaped, and a second framework would need a second isolation story.
Tested
Linux, in the consolidated worktree:
mypy src/kiro_crewclean; black, isort, flake8,check_black_formatting.py,check_subprocess_encoding.py,check_comment_history.py,docs-lint.shand the brand gate clean; every changed test file plus the CI-surface pins pass; the 7 boot-matrix tests pass (37 s).On this PR's own CI: the gateway boot matrix passes on ubuntu-latest, macos-15 and windows-latest; the Windows fail-closed contract job passes; the Windows installer builds a real wheel into python-build-standalone and boots it; the Pod Boot Canary boots a real pod under Task Scheduler on windows-latest and reclaims it; all three macOS full-suite shards finish inside the cap with the terminal handler suite green in a dedicated step; the four Windows shards now also collect
test_harness.pyand every pod Windows suite.What booting on macOS and Windows caught
None of these were visible to the unit shards, and every one is fixed in this PR rather than deferred.
Product defects:
KIROCREW_READY:format_dashboard_urlscalledsocket.gethostbynameon the event loop, the mDNS timeout tripped the loop watchdog. The lookup is bounded in a thread and the boot announcement runs off the loop._kill_sessionclosed the controller fd before ending the child; on darwin a parked PTY read never wakes, so the handler hung and left an orphan shell. It now kills the group, waits bounded, joins the reader, then closes.proc.wait()sat ahead of the close. Every wait in the kill is bounded now._pty_child_envsetsBASH_SILENCE_DEPRECATION_WARNING=1(setdefault, a user value wins).apple_speechrefused the macOS runner's Xcode because the versioned bundle sits behind a symlink; the trust walk resolves the bundle prefix.dev_fleet/repository.pyhad no arm for the EPERM a type-change produces on macOS.pod_name_mutexwas a no-op whereverfcntlwas absent, so a concurrentpod up/pod downon Windows was unguarded; it locks throughplatform_compat.file_lockon both planes.stop()delete a live gateway's task and home while reporting success; it is a boot failure now, with the child tree killed.port_ownercalled the pod's own gateway foreign on Windows: pip's.exelauncher stub is the recorded pid while its child binds the port. The Windows leg accepts a listener among the recorded pid's descendants and nothing else.cli_server._ensure_nodespawnedbashfrom inside a pod on Windows, which is WSL's launcher; win32 no longer spawns a POSIX shell there.O_DIRECTORY/dir_fd; each has a win32 branch built onplatform_compat.pin_directorywith the residual stated in the docstring, the pod README and the security spec.AF_UNIXpath limit under a deep plane root and the gateway silently fell back to TCP-only.KIROCREW_POD_KIRO_BINflows through the one env selection both backends serialise.codex-review.ymlused a BSD-incompatiblesed.Test-suite assumptions that only held on Linux:
test/e2e/scenarios/conftest.pyshadowed the rootdirconftestfor the 93 files that import it (about 120 errors per shard on every OS); both e2e directories are packages now.KIROCREW_KIRO_BINpointed at a.py, whichCreateProcessrefuses; the sharedharness.fake_acp_backend_launcherwrites a.cmdshim on Windows.subprocess.Popenon the Windows pod backend must also pinprocess_start_time, which shells out topson darwin.O_BINARYsimulation picked a bit that isO_DIRECTORYon darwin; the recorder strips the simulated bit before the real open.try, and the boot canary wrote a stub SPA into the checkout; both are inside teardown now, the stub is placed by the workflow.Research behind the design is archived at
workspace/cross-os-e2e-research-2026-09-08/(audit, framework landscape, harness designs).Adversarial Windows review — findings fixed in
ca09a05aAn adversarial review of this PR's Windows surface (12 units, each required to
measure rather than reason) found that the backend answers "is this pod alive"
and "is this process mine" from data that cannot answer them, and that several of
the new gates could not go red on the thing they exist to catch. GPT 5.6's
blocking F1 turned out to be a third path into the same root cause and is fixed,
not overridden.
Three claims in the section above were contradicted by measurement. Correcting
them here rather than silently editing the list, so the record shows what changed:
AF_UNIXpath limit under a deepplane root and the gateway silently fell back to TCP-only." The PRODUCT is
unchanged:
dashboard/server.py's_start_unix_sitestill wraps the bind inone
except Exception, logs one warning and degrades to TCP-only, with nolength pre-check and no refusal. What this PR added is a short plane root in
the E2E fixture. An operator with a deep
KIROCREW_POD_ROOTstill gets a podthat boots, reports healthy, and whose
pod apirefuses forever with text thatnever mentions path length. Left as a stated gap rather than quietly claimed.
security.md. It said a same-UID componentswap "the re-witness detects but cannot undo". Both halves were false: a swap
after the pin is refused by the kernel, and the swap that works is undetected,
because a witness taken before the pin can only be compared against itself.
Corrected, and the seed branches now pin the anchor before creating anything so
the window is as small as the platform allows.
(
feat/pod-windows-backend)". That backend lands in this PR's own firstcommit, so the comment and the doc were stale on arrival. Nothing pins the
matrix, so it will not self-correct.
pod/README.md's "the ceiling is a Job object — and it IS enforced" is likewisesoftened to enforced-when-attachable-and-loud-when-not, which is what
apply_job_limitsactually contracts (it returns False on any Win32 failure andthe caller treats that as no ceiling).
Revert granularity (Design Review's Watch)
This PR carries runtime product fixes inside the
feat(ci)commit, and the reviewis right that the coupling is real. Recording the consequence explicitly, which is
what that Watch asks for: reverting the
feat(ci)commit alone reintroduces themacOS gateway dying right after
KIROCREW_READY, the embedded-terminal orphanshell hang, and the Windows liveness/reclaim defects fixed in
ca09a05a. Arevert must cherry-pick those runtime fixes back first; they are enumerated in the
"What booting on macOS and Windows caught" section above and in
ca09a05a'smessage.
CI cost of the expected-failures conversion
The gap lists applied
pytest.mark.skip, so a listed test never ran and nothingcould report that a gap had CLOSED — on files whose own headers say "fix the test
and DELETE the line". They are strict xfails now. The first run burned 23 stale
entries off the Windows list (203 → 180 entries: those gaps were already fixed).
The cost is that 180 previously-skipped tests now EXECUTE on the Windows shards:
115s single-process locally, which rides the shards' own
-n auto. One entry hadto be excluded and skipped at the test instead — a hypothesis property test with
deadline=Nonewhose example budget outruns the per-test--timeout, and onWindows pytest-timeout kills the process rather than the test, so a hanging entry
in an executed list is a shard-killer.
Pattern harvest
Rule candidate: on Windows, a process's creation-time token answers IDENTITY and
never LIVENESS — the process object outlives the process while any handle to it is
open, so
process_start_time(pid) == tokenkeeps matching a corpse. Askpid_existsfirst and let the token only narrow a live pid to the right process.This PR contained three independent instances of the inversion (
supervised_pid,stop's survivor re-check, and the boot canary's own assertion), which is whatmakes it a rule rather than a bug.
Rule candidate: a gate whose pass condition is an unanchored substring match on a
count (
grep -q "3 passed") pins nothing, because it also matches every largernumber. Anchor it (
grep -qE '(^|[^0-9])3 passed'), and where the count legitimatelymoves, compare it numerically against a floor instead of matching a range that a
collapse to one test satisfies.