Skip to content

ci: move e2e, real-adapter and windows pod boot to the fleet - #12133

Merged
bolichen97 merged 1 commit into
mainfrom
ci/decouple-hosted-e2e-namespace
Sep 20, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
ci/decouple-hosted-e2e-namespace

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

After #11880 the 16 backend shards, lint and the frontend lanes run on the CodeBuild fleet, but the browser gate E2E (stub ACP backend, offline) still ran on a shared hosted runner. It is 15 of the 27 hosted minutes a CI run has left. Only one of its steps needed the hosted kernel: the ~3.5-minute private-workflow MCP run that needs unshare --map-root-user. The other 11 minutes (SPA build, Playwright specs, memory UI evidence, i18n render gate) sat on the shared pool for no reason. Real Adapter Contract Tests was hosted with no stated fleet blocker.

Why it matters

Hosted-runner minutes are the shared queue every PR waits in. Moving these two jobs takes the per-run hosted footprint from ~27 to ~18 minutes and keeps the shared pool for the jobs that genuinely need it.

What changed (motivation → approach → change)

Job Before After
e2e hosted ubuntu-latest fleet linux_runner_large, behind run-as-runner
e2e-private-namespace (was a step inside e2e) new hosted ubuntu-latest job, one test file
real-adapter-contract hosted ubuntu-24.04 fleet linux_runner, behind run-as-runner

The E2E suite's own gateway takes unsandboxed consent for its disposable config. The fleet container refuses unshare(CLONE_NEWUSER) at the runtime policy level, which no sysctl lifts, and the agent binary this lane runs is fake_acp_backend.py, a stdlib echo stub -- so OS isolation guards nothing the browser specs assert. Without the consent every agent-dependent spec gets a sandbox-refusal card instead of a turn. A sandboxed spawn doing real work stays covered by e2e-private-namespace (hosted, real namespace) and by e2e-boot-matrix, which asserts both sandbox tiers.

The E2E job is split along the one hard boundary. The fleet container refuses user namespaces, so test/e2e/test_private_workflow_memory.py keeps its exact command, env and the AppArmor sysctl in its own small hosted job. Everything else in e2e moves to the fleet.

On the fleet the gateway refuses provider execution as root, so e2e and real-adapter-contract use the same run-as-runner boundary as the backend shards: setup and every step that writes GITHUB_OUTPUT/GITHUB_ENV run first as the runner identity, then the tests run through ci-shell as runner. The Playwright browser is pinned to $RUNNER_TEMP/ms-playwright (written to GITHUB_ENV) so the root cache restore and the runner-user specs find the same directory.

Pod Boot Canary (Windows) stays hosted and its comment now records why instead of leaving it unexplained. The Windows fleet is a container, where schtasks /Create exits 1 on its own /ST is earlier than current time warning, so the runner user cannot create the scheduled task this canary exists to prove. Measured on the fleet: all 3 tests fail at that probe before any pod boots.

The docs/ci/ci-and-reviews.md job table names the new lane and the runner each job uses.

Backwards compatibility

Compatible: no product code changes. Fork PRs and non-allowlisted actors still resolve to the hosted labels through the existing changes outputs.

Tests

  • test_ci_nonroot_boundary.py: e2e and real-adapter-contract route to the fleet with the boundary after setup and every file-command write before it; e2e-private-namespace keeps the exact private-workflow command, env and sysctl and stays hosted; e2e no longer mentions the namespace or that file.
  • test/test_playwright_e2e.py: the suite's gateway seeds sandbox_allow_unsandboxed_exec for its own throwaway home only, so smoke and the boot matrix keep their existing sandbox expectations.
  • test_ci_fleet_routing_expression_parity.py: the two migrated jobs are added to the pinned resolver-consumer table; test_ci_nonroot_boundary.py keeps pod-boot-windows in the stays-hosted set.
  • test_e2e_ci_scheduling.py: the private gate is asserted on the new job.

Manual verification

This PR runs under the modified workflow, so the CI on this PR is the proof: e2e and real-adapter-contract must show codebuild-kirocrew-gha-* runner labels and pass, and e2e-private-namespace must pass on ubuntu-latest. Measured: Real Adapter Contract Tests passed on the Linux fleet in 4 min, e2e-private-namespace passed hosted in 4 min. actionlint passes on ci.yml.

Related Issues

no linked issue: follow-up to #11880

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and cixuuz September 20, 2026 00:51
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Take the browser E2E gate, the real-adapter contract lane and the Windows pod-boot canary off the shared hosted runners and onto the team CodeBuild fleet, leaving hosted only the one E2E step that genuinely needs an unprivileged user namespace.
Not a goal: Changing any product behaviour, weakening or dropping any test, or moving the jobs that need real IPv6, a real namespace sandbox, kernel lock ownership, Docker-in-Docker or macOS.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Title claims the Windows pod boot canary moves to the fleet; the diff pins it hosted — the merge commit will record a false fact permanently.

Watch

  • The PR title ("move e2e, real-adapter and windows pod boot to the fleet") contradicts its own body and the diff, which only adds a comment explaining why pod-boot-windows cannot move ("Hosted, and it cannot move"). Once merged, the commit title is immutable and anyone reading history will believe the canary runs on the fleet.
    Clears when: the PR title/commit message is reworded to say the canary stays hosted (e.g. "…; document why windows pod boot stays hosted").
  • The description measures real-adapter-contract and e2e-private-namespace passing but not the migrated e2e job itself on the fleet — the largest and riskiest of the three moves (browser cache across the root/runner HOME split).
    Clears when: this PR's own e2e check shows a codebuild-* runner label and green, per the PR's stated proof.

Suggestions

  • e2e-private-namespace needs no SPA, Node, or browser, yet runs on frontend-only diffs; gate it on changes.outputs.only_frontend != 'true' like real-adapter-contract to save the exact hosted minutes this PR exists to reclaim.

[DESIGN-REVIEWED] 02f7b9b

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 02f7b9b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9 — this comment is updated in place on each push.

Review details

Both facts check out: real-adapter-contract declares changes in needs, and the e2e/real-adapter-contract runs-on expressions match their parity constants exactly. The sandbox_allow_unsandboxed_exec = True addition is confined to a disposable test gateway driven by a stdlib echo stub, with real sandboxed execution retained in e2e-private-namespace and e2e-boot-matrix. No production code path is touched, and the workflow/test changes are internally consistent.

No findings.

[OPUS-REVIEWED] 02f7b9b

Verdict parsed from the review's SHA-scoped output markers for commit 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 02f7b9bdb45b9890ff07a8a93a1a247ccdd3b0f9 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new hosted job re-pays checkout/install/sysctl the PR exists to save: backend-test-sandbox already provides an identical hosted user namespace.

Not justified as shipped

  • Item 2 — duplicate of .github/workflows/ci.yml backend-test-sandbox: 5 of the new job's 6 setup steps are byte-identical to that existing hosted namespace job (setup-python 3.12, setup-uv, same uv pip install line, same sysctl, same unshare --mount --map-root-user true); grepped both jobs' steps. "Keeping it separate limits hosted runner use to its ~3.5-minute run" argues for merging harder: a second job pays checkout+install again on the shared pool.
    Subtraction: delete the e2e-private-namespace job and append its one pytest step (env intact) to backend-test-sandbox, which already cleared and verified the namespace.
    Clears when: the step is merged into backend-test-sandbox, or a measured critical-path or verdict-isolation cost of doing so is named.
  • Item 6 — undeclared: real-adapter-contract's hosted fallback silently changes ubuntu-24.04 → ubuntu-latest; forced by the canonical resolver expression test_ci_fleet_routing_expression_parity.py pins, and the images are identical today.
  • Item 7 — undeclared: description says the private-workflow step "keeps its exact command", but the diff prepends echo "::remove-matcher owner=python::"; repo convention (19 existing occurrences).

What this change ships

Inventory (8 items) — 5 justified

Intent: stop spending ~15 of 27 shared hosted-runner minutes on jobs that don't need the hosted kernel — a FIX of a measured cost.

  1. e2e browser gate now runs on the CodeBuild fleet behind run-as-runner — justified
  2. Private-workflow MCP test split into a new hosted job e2e-private-namespace — duplicate of backend-test-sandbox setup (5 identical steps)
  3. real-adapter-contract moves to the fleet behind the same boundary — justified
  4. The E2E suite's disposable gateway now consents to unsandboxed exec everywhere, delegating sandbox coverage to e2e-private-namespace and e2e-boot-matrix (verified: that matrix pins both tiers) — justified
  5. Playwright browser cache moves to $RUNNER_TEMP/ms-playwright so root restore and runner-user specs agree — justified
  6. real-adapter-contract hosted fallback label changed ubuntu-24.04 → ubuntu-latest — undeclared
  7. ::remove-matcher echo added to the private test command — undeclared
  8. pod-boot-windows comment records the measured schtasks reason it stays hosted — justified

[FIRST-PRINCIPLES-REVIEWED] 02f7b9b

@bolichen97
bolichen97 force-pushed the ci/decouple-hosted-e2e-namespace branch from e79a3e4 to d763eaf Compare September 20, 2026 01:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@bolichen97
bolichen97 force-pushed the ci/decouple-hosted-e2e-namespace branch from d763eaf to 02f7b9b Compare September 20, 2026 02:52
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved to e8b94aa since this head went green-adjacent; patch content verified byte-identical across the rebase. Also: the E2E suite's own gateway now takes sandbox_allow_unsandboxed_exec for its throwaway config. The CodeBuild container refuses unshare(CLONE_NEWUSER) at the runtime policy level (no sysctl lifts it) and this lane's agent binary is fake_acp_backend.py, a stdlib echo stub, so OS isolation guards nothing the browser specs assert. Measured without it: 263/269 specs pass and all 6 failures are sandbox-refusal cards on agent-dependent turns. A sandboxed spawn doing real work stays proven by e2e-private-namespace (hosted, real namespace) and e2e-boot-matrix (both tiers). pod-boot-windows returns to windows-latest: the Windows fleet container cannot create a scheduled task (schtasks /Create rc=1), which is the one fact that canary exists to establish.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@bolichen97
bolichen97 merged commit f31f25a into main Sep 20, 2026
137 of 144 checks passed
@bolichen97
bolichen97 deleted the ci/decouple-hosted-e2e-namespace branch September 20, 2026 03:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 20, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants