Repository navigation
ci: run one backend shard non-root on codebuild - #11699
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Undocumented 95-line hunk: WatchThe diff adds [DESIGN-REVIEWED] 737dee0 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Undeclared rider: new macOS responsible-process-walk tests ship in a CI-routing PR the description never mentions; otherwise verify the shard-1 canary went green on CodeBuild. Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: run the backend test shards on CodeBuild compute without weakening the suite's non-root permission assertions, starting with one canary shard. ADDITION (CI infrastructure).
[FIRST-PRINCIPLES-REVIEWED] 737dee0 |
GPT 5.6 Review — ✅ human override acceptedHuman judgment by @bolichen97 overrides the GPT 5.6 finding for This comment is updated in place on each push. The model was not re-run because an authorized human decision supersedes it. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
9bc239a to
e76be8c
Compare
Disposition — GPT 5.6 F1 (security, blocking): implemented in
|
e76be8c to
76e1e56
Compare
Disposition — GPT 5.6 F1 round 2 (security, blocking): implemented in
|
Disposition — GPT 5.6 F1 round 3 (
|
|
/ai-review override gpt 76e1e56: pinned jq hash verified byte-identical to jqlang's published sha256sum.txt for jq-1.7.1 jq-linux-amd64, and the canary's run-as-runner step passed its sha256sum check on CodeBuild |
Human judgment recorded@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
76e1e56 to
0e8e385
Compare
…census #11699 routed backend-test shard 1 to the CodeBuild fleet via a canary-conditional runs-on that reads needs.changes.outputs.linux_runner_large, but did not add the job to _EXPECTED_RESOLVER_CONSUMER_JOBS in test_ci_fleet_routing_expression_parity.py. The census test's added-consumer assertion now fails on main for every PR (and main itself). Add the entry with its exact canary expression so the byte-match check also pins the canary form against silent drift.
PR kirodotdev#11699 made ci.yml's backend-test job read needs.changes.outputs.linux_runner_large (shard-1 CodeBuild canary, conditional on matrix.group) without adding the job to _EXPECTED_RESOLVER_CONSUMER_JOBS, so test_every_ci_yml_resolver_consumer_reads_the_resolver_not_a_literal fails on main tip and reds Backend Tests shard 2 for every open PR. Add the job with its exact conditional expression so the census locks the canary's byte-form the same way it locks the canonical reads.
…census (#11876) #11699 routed backend-test shard 1 to the CodeBuild fleet via a canary-conditional runs-on that reads needs.changes.outputs.linux_runner_large, but did not add the job to _EXPECTED_RESOLVER_CONSUMER_JOBS in test_ci_fleet_routing_expression_parity.py. The census test's added-consumer assertion now fails on main for every PR (and main itself). Add the entry with its exact canary expression so the byte-match check also pins the canary form against silent drift. Co-authored-by: Helena Stafford <helenars@amazon.com>
The pilot's first attempt to route the backend shards to the CodeBuild runner failed 37-42 tests per shard: that runner executes the job as root, and this suite asserts permission semantics root does not have (read-only refusals, root-owned-ancestor checks, PermissionError) while the code refuses to run providers as root at all. AWS documents no non-root mode for its GitHub Actions runner, so the shards were parked behind "once a custom image exists". A custom image is not the prerequisite. .github/actions/run-as-runner draws the boundary inside the job instead. The runner process and the setup-* actions stay root; the action then creates an unprivileged `runner` account, hands it only the workspace and the job temp, registers the setup-python toolcache lib dirs with ldconfig, closes the two measured image gaps (lsof, jq 1.7), and installs /usr/local/bin/ci-shell -- a `shell:` program the steps whose correctness depends on not being root declare as their shell. On ubuntu-latest ci-shell is a bash passthrough and the action changes nothing else, so one expression serves both runners. Measured on the spike (run 34717454028, one shard): uid 1002(runner), HOME=/home/runner, workspace and RUNNER_TEMP owned by runner, 25,224 passed / 77 skipped / 4 failed, with not one root-semantic failure among the four -- a missing lsof (installed here), a 1.6013s reading against a 1.6000s latency bound, and two eager-session scheduling assertions. Only shard 1 is routed. Shards 2-8 stay hosted as controls and exercise the passthrough; they move after three consecutive green canaries. A CodeBuild-only step asserts the boundary before any test runs, and test_ci_nonroot_boundary.py pins the workflow half so a step that silently loses its shell fails as a routing bug instead of reappearing as dozens of permission failures. The namespace jobs stay hosted on purpose: `unshare --mount --map-root-user` is EPERM on this project, so dropping privilege buys backend-test-sandbox, e2e, release-candidate-tests and gui-user-test nothing and routing them would turn real enforcement into a skip.
…census (kirodotdev#11876) kirodotdev#11699 routed backend-test shard 1 to the CodeBuild fleet via a canary-conditional runs-on that reads needs.changes.outputs.linux_runner_large, but did not add the job to _EXPECTED_RESOLVER_CONSUMER_JOBS in test_ci_fleet_routing_expression_parity.py. The census test's added-consumer assertion now fails on main for every PR (and main itself). Add the entry with its exact canary expression so the byte-match check also pins the canary form against silent drift. Co-authored-by: Helena Stafford <helenars@amazon.com>
What changed
backend-testshard 1 now runs on the CodeBuild-hosted runner, behind a newnon-root boundary. Shards 2-8 stay on
ubuntu-latest..github/actions/run-as-runner(new composite action). On a root runner itcreates a
runneraccount with a real home, hands it only$GITHUB_WORKSPACEand
$RUNNER_TEMPviachown, registers the setup-python toolcache lib dirswith
ldconfig, installs the two measured image gaps, and installs/usr/local/bin/ci-shell. On a non-root runner it installs a bash passthroughand changes nothing else, so one expression serves both runners.
ci.yml:runs-on: ${{ matrix.group == 1 && needs.changes.outputs.linux_runner_large || 'ubuntu-latest' }},the action invoked after the
setup-*steps, a CodeBuild-only step thatasserts the boundary before any test runs, and
shell: /usr/local/bin/ci-shell {0}on the pytest step and the coverage staging step.
test/test_ci_nonroot_boundary.py(new) pins the workflow half.docs/ci/ci-and-reviews.mdrewritten for the pilot section this replaces(it still said the shards move "once a non-root custom image exists").
Why
AWS documents no non-root mode for the CodeBuild GitHub Actions runner, and the
pilot's first attempt to route these shards failed 37-42 tests per shard on
exactly that: the suite asserts permission semantics root does not have
(read-only refusals, root-owned-ancestor checks,
PermissionError) and the coderefuses to run providers as root by design. That was read as "blocked until a
custom image exists". It is not — the runner process can stay root as long as
one controlled boundary drops privilege below it.
Evidence
Prior experiment: commit
1746f115bf1ed2b126098996471f98259e18a09aonci/codebuild-backend-spike, CI run34717454028, job103617927018.uid=0(root)uid=1002(runner) gid=1002(runner)HOME/root/home/runner$RUNNER_TEMPownerResidual-failure disposition (the four):
test_cli_server_more_coverage.py::TestStopViaService::test_explicit_port_bypasses_the_servicelsofis absent fromstandard:7.0, present onubuntu-latest. Fixed here (distro package, installed by the action).Scope boundary
unshare --mount --map-root-user truereturnsOperation not permittedon thisCodeBuild project, so dropping privilege buys the namespace-dependent jobs
nothing:
backend-test-sandbox,e2e,release-candidate-testsandgui-user-teststay hosted, andtest_ci_nonroot_boundary.pyasserts they arenot routed. They verify a real user namespace; routing them would turn an
enforcement check into a skip. Also out of scope: Windows, macOS, production
signing/publishing, privileged mode, Docker-in-Docker, and any change to the
CodeBuild project, fleet, IAM or secrets.
Follow-up (separate PR, not this one): evaluate compute that can give those
four a real unprivileged user namespace — a deliberately reviewed privileged
CodeBuild configuration, an ephemeral EC2 runner, or staying hosted. Option one
is not selectable without first proving nested namespace/mount behaviour and
reviewing the wider privilege boundary.
Security review
persist-credentials: falseon the checkout is unchanged; the job carries nosecrets.*in its environment.runuser). No sudoers rule, nochmod 777,/rootuntouched, no credential file copied into/home/runner.Asserted by the new test, not just reviewed.
changes, which alreadyroutes forks to
ubuntu-latest, and the AWS-side webhook additionally filterson the triggering account.
ghis deliberately not installed — no backend test was measured to need it.
runuser -mpreserves the step environment because the Actions toolchain isreachable no other way; this job's environment holds no secret, and the
CodeBuild service role is reachable from any process in the build regardless.
Verification
Local: YAML + composite-action schema valid;
docs-lint.shpasses; black /isort / flake8 clean on the new file; 2,095 tests across every suite that parses
these workflows pass (
test_ci_nonroot_boundary,test_ci_failure_annotations,test_ci_pytest_progress,test_security,test_fast_gate_barrier,test_builtin_skill_scope,test_vendor_manifest,test_ai_review_workflows,test_local_gate,test_coverage_omit_contract,test_e2e_ci_scheduling,test_macos_platform_tests_gate,test_main_ratchet_audit,test_workflow_cache_setup_uniqueness,test_fork_stage2_checkout,test_pr_readiness_dispositions,test_pod_scenario_matrix,test_issue_summary_workflow,test_prepare_pr_diff_signals).The canary itself is the real verification, and it runs on this PR. The
remaining seven shards are not migrated here: they move in a follow-up only
after three consecutive green canaries including one rerun, with coverage
uploaded by the unprivileged process each time.
Rollback
One line: replace shard 1's
runs-onexpression withubuntu-latest(or setboth outputs in the
changesjob's runner step toubuntu-latest). Removing therun-as-runnerstep as well restores the previous job exactly; a CodeBuildproject that receives webhooks for jobs it does not match starts nothing.
Note on overlap
Open PR #11673 also edits
ci.yml(runs-onrouting for other jobs). It does not touchbackend-test, sothe overlap is textual at most.