Skip to content

ci: route fast-gate.yml and 39 more Linux jobs to the CodeBuild fleet - #11673

Merged
bolichen97 merged 1 commit into
mainfrom
fix/fast-gate-fleet-runner
Sep 18, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/fast-gate-fleet-runner

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Fast Gate's twelve jobs, plus 36 more stateless Linux jobs across 22 other
workflow files (48 total, listed in "What changed" below), all ran on
GitHub-hosted ubuntu-latest, even though ci.yml already routes its own
Linux jobs to the self-hosted CodeBuild fleet (kirocrew-gha-linux) for
same-repo runs.

Why it matters

These jobs are the majority of this repository's routine CI cost: cheap,
frequent, pure Python/Node/gh-CLI/Bedrock-OIDC work with no Docker
dependency, run on every push and PR to main. Leaving them on
ubuntu-latest pays hosted-runner queue time and cost for work the fleet
already absorbs.

What changed (motivation → approach → change)

ci.yml resolves its Linux runner label once, in its changes job: a
same-repo run gets a per-run CodeBuild label, a fork run — or any run
outside kirodotdev/KiroCrew — stays on ubuntu-latest. The real trust
boundary is on the AWS side: the CodeBuild webhook allowlists actors who
can push to this repository (docs/ci/ci-and-reviews.md:392-411); this
expression only decides where a job's label points.

Fast Gate's twelve gate jobs can't reuse ci.yml's needs-based pattern —
a test pins every gate job to carry no needs:/if:, since a needs:
edge would let a failed sibling skip a gate that must never be skippable —
so the routing is expressed two ways:

  • Inside ci.yml (changes, await-fast-gate, frontend-lint,
    lockfile-engines-floor, backend-test-crew-container,
    electron-test) reuse changes's needs.changes.outputs.linux_runner
    output. backend-lint stays on ubuntu-latest: it OOM'd on the fleet's
    default tier, then failed twice more ("runner lost communication with
    the server") on the _large (16 GB) variant too, reproduced
    deterministically via two gh run rerun --job retries — a real
    incompatibility, not a flake.
  • Everywhere else — Fast Gate's twelve, plus 36 more jobs across 22
    other files — carries one self-contained runs-on: expression computed
    from github.* context, no job dependency needed.

Three AI-review findings caught real gaps before merge. GPT 5.6: generate
runs an untrusted-diff Claude step whose Write tool is fenced by a
hardcoded Write(//home/runner/work/_actions/**) deny — a GitHub-hosted
path CodeBuild relocates, failing the fence open next to a live Bedrock
credential; stays on ubuntu-latest. Self-identified once validate
started failing: a diff limited to this file's own runs-on: lines gives
generate's model no shell/flow/cron surface to propose, and
scope_candidates.py treats an honestly empty candidates.json as
untrustworthy by design — merging it with "proposed but already covered"
would be a false green — so validate/publish stay on ubuntu-latest
too. GPT 5.6 again: issue-summary.yml/issue-triage.yml fire on
issues: opened, triggerable by any GitHub user — no PR head repo to
compare, so the fork check always requests the fleet, rejected by the AWS
actor allowlist, queuing forever. Reverted both, plus
ai-review-human-override.yml/disposition-deferral-check.yml
proactively. Design Review: the same mechanism hits nightly.yml's
schedule/workflow_dispatch trigger — reverted that too. Separately,
Design Review: ~50 hand-copies of the expression had nothing pinning them
identical, and the first parity test was vacuous. The rewrite asserts
byte-identity across every copy, that the routed set equals a fixed list,
and that every ci.yml resolver consumer reads the exact expected text.

Untouched, deliberately: memory-benchmark.yml's measure
(ubuntu-24.04-pinned), security-scope-review.yml's adjudicate (OS
matrix), backend-lint (fleet incompatibility, above), the eight permanent
exceptions (generate, validate, publish, summarize, triage,
record, validate-deferral, nightly.yml's version), and everything
Windows/macOS-only, Docker-in-Docker, root-permission-sensitive, or
holding production signing credentials. release.yml stayed routed: its
push-to-tag actor already needs write access, so this allowlist gap
doesn't apply. Jobs this PR cannot trigger (pages.yml, etc.) get their first live run
post-merge — issue #11705 tracks that, accepted-and-deferred per Design
Review span eb2ca73d8905.

No job's steps, env, triggers, permissions, or timeout changed in any
touched file — only runs-on and, where applicable, needs.

flowchart LR
  classDef changed fill:#FEF3C7,stroke:#D97706
  classDef ctx fill:#E0F2FE,stroke:#0284C7

  PR["push / pull_request"]:::ctx --> J1["48 jobs across<br/>29 workflow files"]:::changed
  J1 --> R1["ci.yml jobs:<br/>needs.changes.outputs.linux_runner"]:::changed
  J1 --> R2["everywhere else:<br/>inline github.* expression"]:::changed

  linkStyle 0 stroke:#D97706,stroke-width:2px
  linkStyle 1 stroke:#D97706,stroke-width:2px
Loading

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

Tests

A new test, test_ci_fleet_routing_expression_parity.py, adds three checks:
every copy of the inline routing expression is byte-identical to one
canonical copy, every job expected to carry either that expression or a
named permanent exception is enumerated exactly (a silent add/drop is
visible in a future diff), and every ci.yml job reading the resolver's
output does so with the exact expected text.
Beyond that, the change is routing metadata on GitHub-hosted workflow jobs,
not application code, so correctness is otherwise verified by the live run
of this workflow set on this PR: every routed job completing on a
codebuild-kirocrew-gha-linux-* label is the proof, visible in this PR's
own check run.

Not provable from this PR's own check run: jobs whose trigger this PR
cannot fire — pages.yml, add-contributor.yml, and others gated on
workflow_dispatch/schedule/push to main — get their first live
CodeBuild run only after this PR merges. Filed
#11705 to track that;
accepted-and-deferred per the Design Review finding at span
eb2ca73d8905.

Manual verification

Parsed all 63 workflow YAML files and confirmed every one still parses.
Re-executed the exact assertions from tests that pin job shape —
test_fast_gate_barrier.py's test_every_gate_is_unconditional,
test_builtin_skill_scope.py's and test_vendor_manifest.py's equivalent
checks, test_bench_round9.py's test_the_measuring_job_pins_its_runner
— against the final squashed YAML. (Full pytest needs a dependency closure
unavailable in this sandbox; each assertion was replicated verbatim from
its test's source against the same parsed YAML the real test would use.)
Diffed every touched file against its pre-change version: only runs-on:
and, where applicable, needs: changed.

Related Issues

no linked issue: internal follow-up from the CI/CodeBuild fleet migration
audit, not tracked by a GitHub issue.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and iamwhatever September 18, 2026 03:00
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Route Fast Gate's twelve cheap gate jobs to the self-hosted CodeBuild fleet (kirocrew-gha-linux), the same way ci.yml already routes its own Linux jobs, so these frequent/cheap checks stop paying hosted-runner queue time and cost.
Not a goal: Migrating any other workflow (fork-*.yml, backend-test, Windows/macOS jobs) — those are separate, deliberately-scoped follow-ups per the earlier CI-migration audit.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 6cfa0e84f92ec6431db333e198cfd73e9461c1a7 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6cfa0e8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6cfa0e84f92ec6431db333e198cfd73e9461c1a7: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 6cfa0e84f92ec6431db333e198cfd73e9461c1a7 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Same trigger class gets opposite treatments: workflow_dispatch is the stated reason nightly.yml stays hosted, yet pages.yml and main-ratchet-audit.yml route it to the fleet.

Watch

nightly.yml's revert comment says a workflow_dispatch actor "need not be a collaborator with fleet access… queues forever," but routed pages.yml and main-ratchet-audit.yml also fire on workflow_dispatch with the identical fork-term-always-false mechanism. If nightly's reasoning holds, a manual dispatch of the Pages deploy or the ratchet audit queues forever with no hosted fallback; if it doesn't hold, the exception list carries a false rationale a future maintainer will act on. Neither run is provable from this PR (both are in the #11705 post-merge class).
Clears when: a live post-merge workflow_dispatch of pages.yml/main-ratchet-audit.yml completes on a CodeBuild label (attach the run to #11705), or the exception rationale is rewritten to one consistent rule for the trigger class.

Suggestions

Invert the canonical expression into a trigger allowlist — fleet only when github.event_name is push or same-repo pull_request, hosted otherwise. One rule replaces the fork-shape denylist plus ~13 annotated schedule/dispatch/issues exceptions, and an unforeseen trigger type fails safe to ubuntu-latest instead of failing to queue-forever.

[DESIGN-REVIEWED] 6cfa0e8

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 6cfa0e84f92ec6431db333e198cfd73e9461c1a7 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6cfa0e8

Verdict parsed from the review's SHA-scoped output markers for commit 6cfa0e84f92ec6431db333e198cfd73e9461c1a7.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6cfa0e84f92ec6431db333e198cfd73e9461c1a7: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 6cfa0e84f92ec6431db333e198cfd73e9461c1a7 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Five routed jobs keep workflow_dispatch — the trigger this PR's own exception comments declare queue-forever unroutable; the rule and the routed set cannot both be right.

Not justified as shipped

  • Item 3 — unverified premise: main-ratchet-audit.yml:63 and pages.yml:12 carry workflow_dispatch yet their five jobs are routed, while this PR's own nightly.yml comment says a dispatch actor "need not be a collaborator with fleet access" and an allowlist rejection "queues forever" — the routing and the stated rule contradict each other, and neither the description nor Design Review's nightly finding addresses these two files.
    Subtraction: if the dispatch-actor claim holds, revert these five runs-on: lines to ubuntu-latest (and their _EXPECTED_ROUTED_JOBS entries).
    Clears when: the author confirms against the AWS webhook allowlist that a workflow_dispatch actor (GitHub requires write access to dispatch) is admitted — or unroutes the five jobs, or corrects the exception comments' workflow_dispatch claim.
  • Item 2 — overstated: "Fast Gate's twelve, plus 36 more jobs across 22 other files — carries one self-contained runs-on: expression"; I count 31 copies across 11 files (19 non-fast-gate across 10), 35 jobs routed in total, not 48 (grep: codebuild-kirocrew-gha-linux-{0}-{1}).
    Clears when: the description's counts match the shipped diff.

What this change ships

Inventory (8 items) — 6 justified

Intent: route stateless Linux CI jobs from GitHub-hosted runners onto the existing self-hosted CodeBuild fleet for same-repo runs — ADDITION (extends the documented ci.yml pilot).

  1. Fast Gate's twelve gates now run on the CodeBuild fleet for same-repo runs — justified
  2. Fourteen more push/PR-triggered jobs across eight files carry the same inline routing expression — overstated: description claims 36 across 22 files
  3. main-ratchet-audit's three jobs and pages' two routed while keeping workflow_dispatch — unverified premise: contradicts the PR's own exception rule
  4. Four ci.yml jobs switch to reading the existing changes resolver output, two gaining a needs: changes edge — justified
  5. Twenty-two unrouted jobs gain comments recording why each stays on ubuntu-latest — justified
  6. backend-lint pinned to ubuntu-latest, recording a reproduced fleet memory incompatibility — justified
  7. New test pins all 31 expression copies byte-identical, the routed set exact, and nine resolver consumers' text — justified
  8. First live fleet run of jobs this PR cannot trigger deferred post-merge, tracked in Observe first post-merge CodeBuild run for schedule/dispatch-only workflows routed by #11673 #11705 — justified

[FIRST-PRINCIPLES-REVIEWED] 6cfa0e8

@bolichen97
bolichen97 force-pushed the fix/fast-gate-fleet-runner branch from 4e031b9 to c450bfd Compare September 18, 2026 03:53
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97
bolichen97 force-pushed the fix/fast-gate-fleet-runner branch from c450bfd to 2d19a9c Compare September 18, 2026 05:17
@bolichen97 bolichen97 changed the title ci: route fast-gate.yml jobs to the CodeBuild fleet ci: route fast-gate.yml and 45 more Linux jobs to the CodeBuild fleet Sep 18, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ human override accepted

@bolichen97 overrode this lane for 6cfa0e84f92ec6431db333e198cfd73e9461c1a7 via /ai-review override (this commit only).

[SCOPE-REVIEWED] 6cfa0e8

@bolichen97
bolichen97 force-pushed the fix/fast-gate-fleet-runner branch from 2d19a9c to 12e15a5 Compare September 18, 2026 05:30
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97
bolichen97 force-pushed the fix/fast-gate-fleet-runner branch from 12e15a5 to 34e8481 Compare September 18, 2026 05:51
@bolichen97 bolichen97 changed the title ci: route fast-gate.yml and 45 more Linux jobs to the CodeBuild fleet ci: route fast-gate.yml and 44 more Linux jobs to the CodeBuild fleet Sep 18, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

span=367312660991 -- fixed

Legitimate and unbounded-harm: routing security-scope-review.yml's generate job to CodeBuild would have made its Write fence (hardcoded Write(//home/runner/work/_actions/**), the GitHub-hosted path) fail open, since CodeBuild relocates the actions directory. The job runs an untrusted-diff-driven Claude step with a live Bedrock credential -- exactly the escalation path GPT named.
Fix in 34e8481: reverted generate to runs-on: ubuntu-latest with a comment naming the reason and the condition for revisiting (fence made CodeBuild-portable). validate/publish (no model step, no Write tool) are unaffected and remain routed.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=33263c34ae0d -- fixed

Legitimate: ~50 hand-copied inline runs-on: expressions with nothing pinning them identical is a real drift hazard -- a future fix to the fork rule applied to one copy and missed elsewhere reopens the hole silently, and this repo pins every other load-bearing workflow shape with a test.
Fix in 34e8481: added test/test_ci_fleet_routing_expression_parity.py -- one test asserts every copy of the expression is byte-identical to a canonical one, a second enumerates every job expected to carry either the canonical expression or a named permanent exception, so a missed copy or a silently added/dropped job fails a test instead of shipping unnoticed. Verified the test catches drift via a negative control (mutated one copy, confirmed the assertion fails) before committing.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=ca21d3b4ab9a -- fixed

Legitimate wording defect, not a code defect: the PR body's original "no fork-trust boundary" framing was imprecise -- the actual boundary is the AWS-side CodeBuild webhook's actor allowlist (docs/ci/ci-and-reviews.md:392-411), not the fork check this diff's runs-on: expression computes. The expression decides where a job's label points; the webhook decides who may reach the fleet at all, independent of that label.
Fix in 34e8481: rewrote the PR body's Problem/Motivation and What-changed sections to name the actor-allowlist boundary explicitly and describe the expression's actual role (label routing, not admission control). No code changed for this finding.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=f6ff3388fcbc -- fixed

Legitimate: the PR body said these jobs "reuse the existing changes job's output" without disclosing that backend-lint specifically reads the pricier linux_runner_large (8 vCPU/15 GB) variant, which contradicts the cost-motivated framing of the change without the OOM justification stated up front.
Fix in 34e8481: rewrote the relevant body paragraph to name linux_runner_large explicitly and state the OOM (exit 137) reason inline, matching the comment already present at that job's runs-on: line.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=eb2ca73d8905 -- accepted-and-deferred

Legitimate and correctly out of scope for this PR: jobs gated on workflow_dispatch/schedule/push-to-main (release.yml, nightly.yml, pages.yml, add-contributor.yml, issue-triage.yml, and others) genuinely cannot be exercised by this PR's own pull_request-triggered check run -- their first real CodeBuild run happens post-merge. Forcing every one to fire in this PR (e.g. via manual workflow_dispatch against this branch) would be a different, riskier verification path than the routing change itself and isn't proportional to a runs-on: label swap.
Filed #11705 (deferred-finding label, assignee bolichen97, Due: 2026-09-25) listing every affected workflow, to be closed once each has been observed completing on a codebuild-kirocrew-gha-linux-* label at least once post-merge.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97
bolichen97 force-pushed the fix/fast-gate-fleet-runner branch from e498380 to 4d61b65 Compare September 18, 2026 09:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97
bolichen97 force-pushed the fix/fast-gate-fleet-runner branch from 4d61b65 to 477e1e7 Compare September 18, 2026 09:35
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97
bolichen97 force-pushed the fix/fast-gate-fleet-runner branch from 477e1e7 to 6cfa0e8 Compare September 18, 2026 09:53
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

/ai-review override target=security-scope head=6cfa0e84f92ec6431db333e198cfd73e9461c1a7

Reason: this diff (CI fleet-routing YAML + one test file) has zero shell/flow/cron surface for the security-scope lane to adjudicate. The generate model correctly returned an empty candidate set. scope_candidates.py validate() deliberately classifies an empty raw_rows as untrustworthy (a test-pinned guard against a silently-failed model call, distinct from the exit-3 "all proposed candidates already covered" path). This is a real design gap for scope-less diffs touching security-scope-review.yml's own routing metadata, not a defect in this PR's content.

Confirmed: no shell/flow/cron content in this diff. Requesting override for this head only; the underlying script gap should be tracked separately.

@github-actions

Copy link
Copy Markdown
Contributor

AI-review override not recorded. Use /ai-review override <fable|gpt|design|ux|first-principles|scope|all> <current-sha>: <one-sentence reason>.

@bolichen97

Copy link
Copy Markdown
Contributor Author

/ai-review override scope 6cfa0e8: Diff has zero shell/flow/cron surface for this lane to adjudicate; generate correctly returned an empty candidate set, which validate() test-pinned as untrustworthy by design.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the scope AI finding as false positive, not applicable, or explicitly accepted for 6cfa0e84f92ec6431db333e198cfd73e9461c1a7.

Diff has zero shell/flow/cron surface for this lane to adjudicate; generate correctly returned an empty candidate set, which validate() test-pinned as untrustworthy by design.

This decision applies only to this commit. A new push requires a new judgment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants