Repository navigation
ci: migrate eligible linux and windows jobs to codebuild - #11880
Conversation
|
Intent: Move the compatible Linux and Windows CI jobs, especially both eight-shard backend matrices and bounded-worker lint, onto the existing CodeBuild projects together. Preserve hosted fallback, every test and coverage gate, and use this PR run to verify actual runner execution. |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Merge-gating evidence is still open by the PR's own checklist: current-head native CI, actor-list/webhook parity, and the undiagnosed fleet worker exit. Watch
Suggestions
[DESIGN-REVIEWED] 68d2af6 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Every developer machine inherits the fleet's 2-worker recycled Black (~17 min measured there); the only OOM ever measured is the fleet's 15 GB cgroup. Not justified as shipped
What this change shipsInventory (10 items) — 9 justifiedIntent: move eligible Linux and Windows CI jobs onto the existing CodeBuild fleet without losing tests, coverage or safety checks — an ADDITION (infrastructure migration).
More exist (docs rewrite, lock/SIGINT failure-evidence collectors); capped at the 10 most visible. [FIRST-PRINCIPLES-REVIEWED] 68d2af6 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsThis PR is entirely CI workflow routing, build tooling scripts, No findings. [OPUS-REVIEWED] 68d2af6 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
16797c8 to
6c6af10
Compare
|
span=1acf088662a4
|
|
span=d42f1b63484b
|
6c6af10 to
7ba4552
Compare
7ba4552 to
cd1e357
Compare
99d8aed to
72b02db
Compare
|
Current-head baseline regression, separate from Fleet routing: commit1c7f963706545fa4e9e6d431dc8217861665d5b2 (merged PR11947) removed execution_context from ContextBuilder.build_message while dashboard/chat_runner.py still passes it, and reintroduced an import of require_private_memory_execution which member_memory_auth.py no longer defines. Parent-side AST comparison confirms both interfaces were consistent immediately before that commit and broken after it; all three production files have zero diff in this CI PR. Observed at current head72b02dbc26b51a3d1e0529c7c4621c27b81de99f:
I will not mask these by weakening E2E or fold a production memory/identity repair into the pure-CI migration. The independent real-adapter marker/required-gate regression from merged PR11960 is being repaired as test-only wiring; its minimum of four contracts remains unchanged. |
c399fb3 to
105555c
Compare
|
span=b4735bd012b4
|
|
span=ad12857f22d6
|
|
span=2aeab904be04
|
|
Hardening ask from two main reds on Sep 21: a foreign PPA index failing Two consecutive attempts on main
Job log: Ask: do not let a foreign index gate the install. Run |
Problem / Motivation
Eligible Linux and Windows CI jobs still use hosted runners despite existing CodeBuild projects. Fleet execution exposes differences in native shells, filesystem identity, inherited signals, Python grammar and formatter memory. Runner assignment alone does not prove test or coverage parity.
Why it matters
Move compatible work to the existing fleet without losing tests, coverage or safety checks. Sharding and migration must not introduce flaky tests; retries, added skips and weaker assertions are not fixes.
What changed (motivation → approach → change)
Route all eight Linux and eight Windows backend shards, backend lint, bundle size and eligible lightweight jobs through existing fleet resolvers. Preserve repository/actor/event admission and hosted fallbacks; PR metadata events stay hosted. Linux keeps its non-root boundary and protected file-command paths. Windows setup selects native Git Bash, supplies pinned Node 24, and verifies SID ownership, ACLs, symlinks, rename and cleanup.
Preserve capability-specific coverage with hosted Linux/Windows IPv6 legs and a strict hosted kernel-lock-owner lane. Ordinary fleet shards retain both lock cases. Only independent kernel evidence permits unknown-owner behavior; strict execution requires the exact dead acquirer. Both hosted coverage artifacts remain mandatory inputs to fail-closed coverage checks.
Pin Fast Gate to Python 3.12. Fleet/local Black checks retain native discovery/configuration/caches/AST checks while recycling workers after each file, limiting concurrency and Linux address space, and rejecting incomplete results. Explicit GitHub-hosted runners retain the original native CLI and default parallelism. Test fixtures establish native-shell, clock, filesystem and process prerequisites without changing product behavior or weakening assertions.
Linux reuses the existing payload-free pytest progress recorder on full, reduced and leaf runs. A crashed worker is not restarted: the shard stays failed, with best-effort progress artifacts instead of an unreported replacement hang. Healthy selection, per-test/job timeouts and coverage are unchanged. This makes a crash diagnosable, not fixed.
.github/workflows/ci.yml,fast-gate.yml, sibling workflows,.github/actions/run-as-runner/action.yml,.github/actions/setup-windows-tests/action.ymlandprobe.pyscripts/bounded_black.py,scripts/check_black_formatting.py,scripts/ci_black_diagnostics.pysetup.cfg,test/, design-critiquetest_backend_routes.py, notebooktest_md_notebook_windows.pySKILL.mdandprofiles/kirocrew.json,docs/ci/ci-and-reviews.mdRemove only obsolete expected-failure entries demonstrated by strict-XPASS and graduated encoding exceptions. Preserve hosted exceptions, permissions, matrices and Fast Gate independence. No AWS infrastructure, IAM, network, image or reserved-capacity changes.
Backwards compatibility
Compatible for application users. No production backup/context code, backup product tests/specification, frontend or translations are changed relative to main. Merged backup fix #12052, macOS fix #11953, revert #12091 and revert #12088 are consumed from main. The redundant adapter-repair commit is dropped:
test/test_codex_session_mcp.pyandtest/test_real_adapter_gate.pyhave zero PR diff. Forks, absent actor membership and missing resolver outputs retain hosted fallback.Tests
68d2af68947adda7f2a082a9e31c9df8b6a9ae96is two commits on179c318748cdd6e42a8ad17272c290eab1b77bc3. Parent ran 534 channel-close, resume, continuation, capability-set, reasoning-effort, adapter-gate, memory-golden, routing, member-publication and log-append tests after the final rebase: all passed, with seven existing platform skips. Backend build passed. Main's three newer fixes are consumed unchanged; no production API shim is added.test/test_channel_slots.py: explicitly establish real-file activity before, equal to and after the supplied close instant. The real writer still runs; exact mtime equality replaces approximate equality, the supplied close stamp must survive, and newer activity must still reopen the conversation. The original test fails under controlled filesystem-clock lead even when the writer preserves mtime exactly. Its Windows log does not record pre-close mtime, so clock lead versus float round-trip is not established as the precise native cause. Before rebasing, 104 related tests passed; eight negative controls caught lost mtime restoration, always-closed behavior, strict-less-than equality and lost click stamps. No sleep, retry, skip or product tolerance is added. The edited class passes Black; three unrelated existing full-file formatting differences remain in the unchanged baseline.cc294b31c64dc7f18c3d377343f19d5aeb36622987880a1abf37bc5b8a02acd4). Apart from the one-file close-fixture follow-up, the prior 64-file diagnostic patch is unchanged from72b02dbc26b51a3d1e0529c7c4621c27b81de99f, which passed 858 focused tests, actionlint, Bash syntax, backend build and applicable static gates. These are local/historical evidence, not current-head native CI certification.PYTEST_ADDOPTS.Manual verification
Both stale-base merges are now reverted upstream. The former context import/signature errors and continuation/adapter fixture failures are not repaired inside this PR. The completed previous-head
c2137d8dff39a4102c60c8800a7d46ebe1ec507bCI run passed Linux Fleet 8/8, Windows Fleet 7/8, four actual adapter contracts with zero skips, hosted IPv6/strict kernel-owner, E2E, coverage and backend lint. Its only failing shard was Windows 3, with the single close-time assertion addressed by the follow-up. All three native macOS shards passed; the last completed at 22:14:46 UTC. Those runs were allowed to finish before publication. The new candidate still requires its own native CI; this is not a current-head all-green claim.Current previous-head Fleet lint evidence: Black took about 1,021 seconds, cgroup peak rose from 5,566,615,552 to 5,777,403,904 bytes under a 14 GiB limit (including the base environment), OOM kills stayed 0 → 0, and child max RSS was 228,708 KiB. One full green Linux matrix does not establish the older worker's cause or satisfy a consecutive-current-head-matrices criterion.
Run 35460744834 on
99d8aedda618a77a53f4ae9bed6592815ca73adepassed Windows 8/8, Linux 7/8 and all three native macOS shards. Linux 8 lost a worker, replaced it, reached 99%, then ended at the 60-minute job limit without a victim name. Later diagnostic runs produced complete summaries instead, but the earlier worker's exit cause was not established; it is not described as fixed.Real fleet formatter evidence includes the env-only two-worker OOM run, followed by successful recycling runs. On
99d8, backend lint passed: Black took about 964 seconds, cgroup peak was 5,756,223,488 bytes under a 14 GiB limit, OOM kills stayed zero. Hosted formatting now keeps the original command/default parallelism instead of extending that cap to forks.Prior read-only AWS inspection found matching actor lists, privileged mode disabled and concurrency limits of 250 per project. This is historical, not a fresh audit or reserved capacity. The current fresh webhook/phase audit is blocked by credential vending failure. Remaining operational evidence is explicit: distinguish dependency wait, provisioning and pure queue time against the documented pilot median-under-60s/max-versus-155s criteria, and confirm paired-edit ownership or fresh equality of
CODEBUILD_ACTOR_IDSand webhook actors. These are not silently waived. Push-only/release paths are not proved by PR execution.Rollback routes subsequent jobs to hosted by clearing
CODEBUILD_ACTOR_IDSor restoring runner expressions; already queued jobs do not move. Actor routing is not an adversarial credential sandbox. Finite tests do not prove zero future flakes. No merge or review override is requested.Related Issues
no linked issue: maintainer-requested pure CI migration.
Reuses Linux pilot #11699, fleet routing #11673, and file-level sharding #11718.
Pattern harvest
Rule candidate: review-prompt
Pattern: CI grammar, process lifetime and kernel capabilities require executable contracts; runner labels and retries do not establish parity or stability.
Checklist