Skip to content

ci: migrate eligible linux and windows jobs to codebuild - #11880

Merged
KiroCrewCaptain merged 2 commits into
mainfrom
ci/windows-shard1-canary-6738240b
Sep 19, 2026
Merged

KiroCrewCaptain merged 2 commits into
mainfrom
ci/windows-shard1-canary-6738240b

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Eligible Linux and Windows CI jobs still use hosted runners despite existing CodeBuild projects. Fleet execution exposes differences in native shells, filesystem identity, inherited signals, Python grammar and formatter memory. Runner assignment alone does not prove test or coverage parity.

Why it matters

Move compatible work to the existing fleet without losing tests, coverage or safety checks. Sharding and migration must not introduce flaky tests; retries, added skips and weaker assertions are not fixes.

What changed (motivation → approach → change)

Route all eight Linux and eight Windows backend shards, backend lint, bundle size and eligible lightweight jobs through existing fleet resolvers. Preserve repository/actor/event admission and hosted fallbacks; PR metadata events stay hosted. Linux keeps its non-root boundary and protected file-command paths. Windows setup selects native Git Bash, supplies pinned Node 24, and verifies SID ownership, ACLs, symlinks, rename and cleanup.

Preserve capability-specific coverage with hosted Linux/Windows IPv6 legs and a strict hosted kernel-lock-owner lane. Ordinary fleet shards retain both lock cases. Only independent kernel evidence permits unknown-owner behavior; strict execution requires the exact dead acquirer. Both hosted coverage artifacts remain mandatory inputs to fail-closed coverage checks.

Pin Fast Gate to Python 3.12. Fleet/local Black checks retain native discovery/configuration/caches/AST checks while recycling workers after each file, limiting concurrency and Linux address space, and rejecting incomplete results. Explicit GitHub-hosted runners retain the original native CLI and default parallelism. Test fixtures establish native-shell, clock, filesystem and process prerequisites without changing product behavior or weakening assertions.

Linux reuses the existing payload-free pytest progress recorder on full, reduced and leaf runs. A crashed worker is not restarted: the shard stays failed, with best-effort progress artifacts instead of an unreported replacement hang. Healthy selection, per-test/job timeouts and coverage are unchanged. This makes a crash diagnosable, not fixed.

Area Files
Routing and bootstrap .github/workflows/ci.yml, fast-gate.yml, sibling workflows, .github/actions/run-as-runner/action.yml, .github/actions/setup-windows-tests/action.yml and probe.py
Formatter scripts/bounded_black.py, scripts/check_black_formatting.py, scripts/ci_black_diagnostics.py
Test and coverage parity setup.cfg, test/, design-critique test_backend_routes.py, notebook test_md_notebook_windows.py
Local gates and contract Repository-owned prepare-pr SKILL.md and profiles/kirocrew.json, docs/ci/ci-and-reviews.md

Remove only obsolete expected-failure entries demonstrated by strict-XPASS and graduated encoding exceptions. Preserve hosted exceptions, permissions, matrices and Fast Gate independence. No AWS infrastructure, IAM, network, image or reserved-capacity changes.

Backwards compatibility

Compatible for application users. No production backup/context code, backup product tests/specification, frontend or translations are changed relative to main. Merged backup fix #12052, macOS fix #11953, revert #12091 and revert #12088 are consumed from main. The redundant adapter-repair commit is dropped: test/test_codex_session_mcp.py and test/test_real_adapter_gate.py have zero PR diff. Forks, absent actor membership and missing resolver outputs retain hosted fallback.

Tests

  • Latest candidate 68d2af68947adda7f2a082a9e31c9df8b6a9ae96 is two commits on 179c318748cdd6e42a8ad17272c290eab1b77bc3. Parent ran 534 channel-close, resume, continuation, capability-set, reasoning-effort, adapter-gate, memory-golden, routing, member-publication and log-append tests after the final rebase: all passed, with seven existing platform skips. Backend build passed. Main's three newer fixes are consumed unchanged; no production API shim is added.
  • The follow-up changes only test/test_channel_slots.py: explicitly establish real-file activity before, equal to and after the supplied close instant. The real writer still runs; exact mtime equality replaces approximate equality, the supplied close stamp must survive, and newer activity must still reopen the conversation. The original test fails under controlled filesystem-clock lead even when the writer preserves mtime exactly. Its Windows log does not record pre-close mtime, so clock lead versus float round-trip is not established as the precise native cause. Before rebasing, 104 related tests passed; eight negative controls caught lost mtime restoration, always-closed behavior, strict-less-than equality and lost click stamps. No sleep, retry, skip or product tolerance is added. The edited class passes Black; three unrelated existing full-file formatting differences remain in the unchanged baseline.
  • The full 65-file CI patch is byte-identical across the latest rebase (SHA-256 cc294b31c64dc7f18c3d377343f19d5aeb36622987880a1abf37bc5b8a02acd4). Apart from the one-file close-fixture follow-up, the prior 64-file diagnostic patch is unchanged from 72b02dbc26b51a3d1e0529c7c4621c27b81de99f, which passed 858 focused tests, actionlint, Bash syntax, backend build and applicable static gates. These are local/historical evidence, not current-head native CI certification.
  • Earlier fixture validation covered 973 routing/profile/backup tests and 447 clock/persistence cases. Negative controls preserve future-time refusal, retention, progress ordering, provenance, invalid-shell rejection and exact lock evidence. Black tests exercise native hosted and bounded Fleet/local paths without changing offender/graduate/fatal-error semantics.
  • Forced worker-exit tests retain a nonzero exit and flushed progress. Linux wiring pins all four pytest paths, original coverage selectors/timeouts and no inherited diagnostic PYTEST_ADDOPTS.
  • Prior differential SAST had zero findings, with 16 disclosed workflow shell-rule parsing/matching warnings. Current-head remote SAST remains required; no exhaustive-scanner claim is made.

Manual verification

Both stale-base merges are now reverted upstream. The former context import/signature errors and continuation/adapter fixture failures are not repaired inside this PR. The completed previous-head c2137d8dff39a4102c60c8800a7d46ebe1ec507b CI run passed Linux Fleet 8/8, Windows Fleet 7/8, four actual adapter contracts with zero skips, hosted IPv6/strict kernel-owner, E2E, coverage and backend lint. Its only failing shard was Windows 3, with the single close-time assertion addressed by the follow-up. All three native macOS shards passed; the last completed at 22:14:46 UTC. Those runs were allowed to finish before publication. The new candidate still requires its own native CI; this is not a current-head all-green claim.

Current previous-head Fleet lint evidence: Black took about 1,021 seconds, cgroup peak rose from 5,566,615,552 to 5,777,403,904 bytes under a 14 GiB limit (including the base environment), OOM kills stayed 0 → 0, and child max RSS was 228,708 KiB. One full green Linux matrix does not establish the older worker's cause or satisfy a consecutive-current-head-matrices criterion.

Run 35460744834 on 99d8aedda618a77a53f4ae9bed6592815ca73ade passed Windows 8/8, Linux 7/8 and all three native macOS shards. Linux 8 lost a worker, replaced it, reached 99%, then ended at the 60-minute job limit without a victim name. Later diagnostic runs produced complete summaries instead, but the earlier worker's exit cause was not established; it is not described as fixed.

Real fleet formatter evidence includes the env-only two-worker OOM run, followed by successful recycling runs. On 99d8, backend lint passed: Black took about 964 seconds, cgroup peak was 5,756,223,488 bytes under a 14 GiB limit, OOM kills stayed zero. Hosted formatting now keeps the original command/default parallelism instead of extending that cap to forks.

Prior read-only AWS inspection found matching actor lists, privileged mode disabled and concurrency limits of 250 per project. This is historical, not a fresh audit or reserved capacity. The current fresh webhook/phase audit is blocked by credential vending failure. Remaining operational evidence is explicit: distinguish dependency wait, provisioning and pure queue time against the documented pilot median-under-60s/max-versus-155s criteria, and confirm paired-edit ownership or fresh equality of CODEBUILD_ACTOR_IDS and webhook actors. These are not silently waived. Push-only/release paths are not proved by PR execution.

Rollback routes subsequent jobs to hosted by clearing CODEBUILD_ACTOR_IDS or restoring runner expressions; already queued jobs do not move. Actor routing is not an adversarial credential sandbox. Finite tests do not prove zero future flakes. No merge or review override is requested.

Related Issues

no linked issue: maintainer-requested pure CI migration.

Reuses Linux pilot #11699, fleet routing #11673, and file-level sharding #11718.

Pattern harvest

Rule candidate: review-prompt
Pattern: CI grammar, process lifetime and kernel capabilities require executable contracts; runner labels and retries do not establish parity or stability.

Checklist

  • Two Conventional Commits commits; pure CI scope verified
  • Related tests, backend build and applicable static gates verified
  • Rebase preserves the validated patch; owning CI documentation updated
  • All required current-head native CI and remaining operational evidence complete
  • No backup product behavior, UI, translations, secrets or internal references added

@bolichen97
bolichen97 requested review from a team and patrigao September 18, 2026 20:00
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Move the compatible Linux and Windows CI jobs, especially both eight-shard backend matrices and bounded-worker lint, onto the existing CodeBuild projects together. Preserve hosted fallback, every test and coverage gate, and use this PR run to verify actual runner execution.
Not a goal: No automatic merge, privilege/IAM/image changes, migration of namespace-enforcement jobs without a validated environment, or movement of signing, fork-review, Task Scheduler, interactive-desktop and Docker-dependent lanes without their required evidence.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 68d2af68947adda7f2a082a9e31c9df8b6a9ae96 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Merge-gating evidence is still open by the PR's own checklist: current-head native CI, actor-list/webhook parity, and the undiagnosed fleet worker exit.

Watch

  • The routing expression fails toward hosted only when CODEBUILD_ACTOR_IDS is absent; if the var and the AWS webhook allowlist drift apart, an admitted-by-var actor's jobs request fleet labels the webhook rejects and hang to the job timeout. The description names this ("confirm paired-edit ownership or fresh equality") but the fresh audit is blocked by credential vending, and the Linux-8 worker exit cause "was not established."
    Clears when: current-head native CI is green and the CODEBUILD_ACTOR_IDS/webhook equality and queue-time criteria are recorded (or the checklist box is checked with that evidence linked).
  • The channel-slots follow-up replaces real wall-clock/mtime sampling with synthetic os.utime after one observed Windows fleet failure whose "precise native cause is not established" per the description — if the cause was product-side (close write racing the click instant under real clocks), that race is now unexercised by any test.
    Clears when: a diagnostic run records pre-close mtime on Windows fleet and pins clock-lead vs float round-trip as the cause.

Suggestions

  • Bounded Black measured ~17 min on fleet inside the unchanged 25-minute lint job (vs ~7–8 min hosted); two green runs is thin headroom — record total-job duration or raise timeout-minutes for the routed path.

[DESIGN-REVIEWED] 68d2af6

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 68d2af68947adda7f2a082a9e31c9df8b6a9ae96 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 68d2af6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 68d2af68947adda7f2a082a9e31c9df8b6a9ae96: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 68d2af68947adda7f2a082a9e31c9df8b6a9ae96 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Every developer machine inherits the fleet's 2-worker recycled Black (~17 min measured there); the only OOM ever measured is the fleet's 15 GB cgroup.

Not justified as shipped

  • Item 6 — inherited (symmetry): check_black_formatting.py:97 gives the native path only to RUNNER_ENVIRONMENT == "github-hosted", so every local run (AGENTS.md gate, prepare-pr profile, golden-paths — counted, 3 local consumers) takes bounded_black.py's two spawn-recycled workers; the doc says "Fleet and local checks use scripts/bounded_black.py", but the measured failure (cgroup 15,032,385,536 B, oom_kill 0→1) is fleet-only and no local failure is named.
    Subtraction: shrink the bounded branch to == "self-hosted" and drop BLACK_NUM_WORKERS=2 … ci_black_diagnostics.py from profiles/kirocrew.json, returning local runs to native -m black.
    Clears when: a Black memory failure is measured on a developer machine, or the local gate returns to the native path.

What this change ships

Inventory (10 items) — 9 justified

Intent: move eligible Linux and Windows CI jobs onto the existing CodeBuild fleet without losing tests, coverage or safety checks — an ADDITION (infrastructure migration).

  1. All eight Linux backend shards run on CodeBuild large behind the non-root boundary — justified
  2. All eight Windows shards, the fail-closed lane and the boot-matrix legs run on CodeBuild Windows with a new setup/probe action — justified
  3. Every fleet route now also requires an admitted actor (CODEBUILD_ACTOR_IDS) and push/opened/synchronize; anything else falls back hosted — justified
  4. Five native-IPv6 tests move to a required hosted lane whose coverage fails closed — justified
  5. New required hosted strict kernel-lock-owner lane (KIROCREW_LOCK_OWNER_STRICT) with the orphan-lock test rewritten two-phase — justified
  6. Black runs bounded (2 recycled workers) on fleet AND every local machine; native only on hosted — inherited: the local half rests on the fleet's OOM alone
  7. Linux shards adopt the progress recorder with --max-worker-restart=0; a crashed worker fails the shard — justified
  8. Fast Gate pins Python 3.12 on every gate; prepare-pr gains a Python-floor preflight — justified
  9. Merge-label, reusable wheel and dependency-audit jobs use the fleet on push only — justified
  10. Test determinism fixes (explicit clocks, native Git Bash, device-name/mtime checks) and removed stale expected-failure entries — rides along

More exist (docs rewrite, lock/SIGINT failure-evidence collectors); capped at the 10 most visible.

[FIRST-PRINCIPLES-REVIEWED] 68d2af6

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 68d2af68947adda7f2a082a9e31c9df8b6a9ae96 — this comment is updated in place on each push.

Review details

This PR is entirely CI workflow routing, build tooling scripts, setup.cfg/profile floor bumps, and test-determinism refactors — no src/kiro_crew runtime logic paths. The candidate list is empty. The only non-test executable code is the three Black tooling scripts, which are diagnostic/build helpers; nothing in them yields a defect I can ground to a concrete input, call path, and observable wrong outcome at the required confidence. Step 2 extension is not warranted here.

No findings.

[OPUS-REVIEWED] 68d2af6

Verdict parsed from the review's SHA-scoped output markers for commit 68d2af68947adda7f2a082a9e31c9df8b6a9ae96.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 68d2af68947adda7f2a082a9e31c9df8b6a9ae96: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the ci/windows-shard1-canary-6738240b branch from 16797c8 to 6c6af10 Compare September 18, 2026 22:10
@bolichen97

Copy link
Copy Markdown
Contributor Author

span=1acf088662a4

fixed: Every matching PR route now accepts only opened/synchronize activities. Edits, reopens and labels stay hosted even for an admitted actor; repository, fork and actor checks remain in place.
Fix: 6c6af10
Verified with 1379 parent-run targeted tests, metadata-event negative cases, successful backend build and independent focused review. Native fleet execution is still being checked.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

span=d42f1b63484b

fixed: Restored the universal per-line chmod ratchet, with only the exact sticky-temp command permitted. Five negative controls cover plain, recursive, setgid, wrong-target and multi-target world-writable variants.
Fix: 6c6af10191.
Verified by 1379 parent-run targeted tests, backend build and independent focused review. No permission check or SAST rule was suppressed.

@bolichen97
bolichen97 force-pushed the ci/windows-shard1-canary-6738240b branch from 6c6af10 to 7ba4552 Compare September 19, 2026 02:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 19, 2026
@bolichen97
bolichen97 force-pushed the ci/windows-shard1-canary-6738240b branch from 7ba4552 to cd1e357 Compare September 19, 2026 03:04
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 19, 2026
@bolichen97
bolichen97 force-pushed the ci/windows-shard1-canary-6738240b branch from 99d8aed to 72b02db Compare September 19, 2026 19:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Current-head baseline regression, separate from Fleet routing: commit1c7f963706545fa4e9e6d431dc8217861665d5b2 (merged PR11947) removed execution_context from ContextBuilder.build_message while dashboard/chat_runner.py still passes it, and reintroduced an import of require_private_memory_execution which member_memory_auth.py no longer defines. Parent-side AST comparison confirms both interfaces were consistent immediately before that commit and broken after it; all three production files have zero diff in this CI PR.

Observed at current head72b02dbc26b51a3d1e0529c7c4621c27b81de99f:

I will not mask these by weakening E2E or fold a production memory/identity repair into the pure-CI migration. The independent real-adapter marker/required-gate regression from merged PR11960 is being repaired as test-only wiring; its minimum of four contracts remains unchanged.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 19, 2026
@bolichen97
bolichen97 force-pushed the ci/windows-shard1-canary-6738240b branch from c399fb3 to 105555c Compare September 19, 2026 21:07
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 19, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

span=b4735bd012b4
Disposition: needs-a-decision

  • Linux worker-exit evidence

Current-head run 35471042013 now has all eight Linux Fleet shards successful, including shard 8 job 105972168647. This is ONE full green Linux matrix, not consecutive current-head runs and not an explanation of the earlier 99d8 worker death. The new no-restart setting preserves a red result rather than retrying a crash to green.
Maintainer: before readiness, do you require the additional consecutive-matrix criterion after the pending Windows test-fixture correction, or an investigation that identifies the old worker exit cause? Neither criterion is marked complete.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=ad12857f22d6
Disposition: needs-a-decision

  • Actor admission parity remains unverified

Fresh read-only AWS vending failed (Merlon 401 / provider dispatch failure); no AWS operations or credential-file reads followed, and historical equality is not current evidence. No variable, webhook, IAM or network setting has been changed.
Maintainer: please confirm the owner responsible for paired CODEBUILD_ACTOR_IDS/webhook edits, or restore host authentication so the requested read-only equality audit can run. This acceptance condition remains open.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=2aeab904be04
Disposition: needs-a-decision

  • Push-only and release execution is not certified by this PR

I read the five named workflows: release uses tag pushes; build-wheel and dependency-vulnerability inherit admitted push callers; label-sync schedules/dispatches stay hosted but admitted main pushes migrate; Pages permits admitted push/manual events. A PR run does not prove those publish paths or fresh actor equality. I will not create a release/tag, deploy Pages, change access controls, or silently subtract requested eligible lanes to manufacture evidence.
Maintainer: should these five workflow routing changes be split out and kept hosted until separately validated, or should they remain pending explicit push/tag operational validation? The current requested migration scope is preserved; no readiness waiver is implied.

@pepmach

pepmach commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Hardening ask from two main reds on Sep 21: a foreign PPA index failing apt-get update kills the lsof install.

Two consecutive attempts on main c8f0f1be42 (docs plus a bash script, no code cause) failed Backend Tests (3.12, 7) and E2E Gateway Boot before any test ran:

Job log: apt-get update exited non-zero because the mozilla-team PPA index baked into the runner image answered 503 Service Unavailable from Launchpad (12:30-13:27Z); the Ubuntu indices were fine. .github/actions/run-as-runner/action.yml:59 chains apt-get update -qq && apt-get install -y -qq --no-install-recommends lsof, so the && never reached the install and the step exited 1 with "could not install lsof from apt". Attempt 3 passed once the PPA recovered, with no change.

Ask: do not let a foreign index gate the install. Run apt-get update best-effort (or scope it to the Ubuntu sources) and let apt-get install lsof be the arbiter; the existing ::error:: still fires on a genuine install failure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants