Repository navigation
🤖 tests: resolve the sandbox image by stable tag and attestation, else build locally - #7672
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🟢 e2e stories (Haiku): 14 passed22 agent steps · 7 replayed from cache · 15 model calls · 46.3k tokens (28% cached) All 14 tests in 9 files
e2e 0.20.0 · 1m 11s · web · run artifacts 🟢 e2e Electron stories (Haiku): 3 passed3 agent steps · 3 model calls · 7.9k tokens (32% cached) All 3 tests in 3 files
e2e 0.20.0 · 27.4s · electron · run artifacts 🟢 e2e agent repros (Haiku): no tests selectedSpend: $0.00 of the $1.00 cap (proxy ledger) e2e 0.20.0 · 5.3s · 2 targets (web, phone) · run artifacts |
…build locally Every sandbox job (all job kinds, in CI and on developer machines) now looks up ghcr.io/coder/xum-bugbash-sandbox:inputs-<key> anonymously, verifies its attestation on the host with `gh attestation verify` (coder/xum, refs/heads/main, the bugbash-sandbox-image workflow), and pulls name@digest. Any other outcome (not published, lookup error, no gh, no token, failed verification) logs the reason and builds the image from the checkout with build.sh, with the LOCAL IMAGE warning. The inputs label check stays for both. Removes image.json, readImageLock, the stale-key refusal, the localBuild option and the BUGBASH_SANDBOX_BUILD_STALE opt-in. CI passes GH_TOKEN to the launcher steps only (host side; the container env drops it). gh also gets GH_CONFIG_DIR when set, for a gh login in a custom config folder. The publish workflow writes its record to $RUNNER_TEMP/publish.json. --- _Generated with `xum` • Model: `anthropic:claude-opus-5-5` • Thinking: `high` • Cost: `$4.06`_ <!-- mux-attribution: model=anthropic:claude-opus-5-5 thinking=high costs=4.06 -->
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
56adeb5 to
692161f
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 692161f046
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The first push publish (run 40480740229) warned that creating the storage record needs artifact-metadata: write. Runners verify the attestation bundle in GHCR and never read a storage record, so the step creates none.
The verified path printed nothing, so a job log could not show which image ran. It now logs the name@digest, as the fallback logs its reason.
The verify step gave gh only PATH, HOME, GH_TOKEN and GH_CONFIG_DIR. A login under a custom XDG_CONFIG_HOME, or a token in the desktop keyring (gh auth login's default, reached over the session bus), then looked like no login, and the launcher built the image locally. gh now also gets XDG_CONFIG_HOME, DBUS_SESSION_BUS_ADDRESS and XDG_RUNTIME_DIR. Docker and the job container still get none of them.
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@codex review |
🛡️ Codex Security Review · Automatically triggeredSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
Codex Review: Didn't find any major issues. Delightful! ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Readiness record
Generated with |
StatusThe readiness record is consistent with the supplied PR state: the local checkout is clean and at the stated final commit, The merge could not be started because this environment does not have the official Checks
|
|
Post-merge check on
So a job whose inputs key main has published now pulls the attested image, and a job with a new key builds the image locally and warns. |
Summary
Every sandbox job now gets its image without a pin. In CI and on a developer machine, the runner looks up the stable tag
ghcr.io/coder/xum-bugbash-sandbox:inputs-<key>for its checkout's inputs key. It checks the tag's build attestation withgh, and it uses the image only if main's publish workflow built it. In every other case it builds the image from its own checkout, about 40 s in CI, and warns.tests/bugbash/sandbox/image.json, theBUGBASH_SANDBOX_BUILD_STALEopt-in and the job-kind gate go away. With this PR, an image change takes one normal PR: no manual publish, no pin PR and no stale-key refusal. This is PR 2 of 2. It is stacked on #7671, which makes main publish the stable tag.Background
The user approved the plan "Sandbox Image Upsert in CI". Today the runner trusts only the digest in
image.jsonand refuses a checkout whose key differs from it. Only the self-check and the exact-step repros can build locally, and only in CI. So a Dockerfile change broke the story lanes until a person published the image and pinned it. After #7362, that kept the merge queue red for 3.5 hours.Implementation
Session.ensureImage()(tests/bugbash/sandbox/runner.ts) resolves the image in this order:docker buildx imagetools inspect.gh attestation verify oci://IMAGE@digest --repo coder/xum --bundle-from-oci --source-ref refs/heads/main --signer-workflow coder/xum/.github/workflows/bugbash-sandbox-image.ymlon the host.ghgets onlyPATH,HOME,GH_TOKENandGH_CONFIG_DIR. None of them reaches Docker or the job container.IMAGE@digestand pulls it when Docker does not have it.build.sh: a missing tag, a lookup error, a bad digest, nogh, no gh token, a failed check or a timeout.org.xum.bugbash.inputsmust equal the key) still runs for both kinds of image. It still refuses on a mismatch.The launcher prints the LOCAL IMAGE warning with the reason, as a
::warning::in Actions, for every job kind.A local build gives a job no new power. Each job already stages its checkout's git-listed files and mounts its
dist/andnode_modules/in the container. So a job runs either an image that main's workflow built and attested, or one built from its own checkout. A registry tag that someone else pushed is never used.CI gives the launcher steps
GH_TOKEN: ${{ github.token }}: the story lanes instories.ymland both steps ofTest / Bug-bash sandboxinpr.yml. This PR changesbuild.shand the Dockerfile, which are comment-only changes, so its key changes. Its own CI therefore runs the local-build path in every sandbox job.Validation
bun test ./tests/bugbash ./tests/bugbash/sandbox: 356 pass.ensureImagehas a test inrunner.test.ts. The implementer broke each path in turn and saw its test fail. That was 10 breaks, plus 3 for the env allowlist.GH_TOKENandGH_CONFIG_DIRreach the fakegh. They never reach the fakedockeror thedocker runline.make static-check,make lint-eslint,make lint-zizmor, actionlint andmake crap-check: pass.make bug-bash-sandbox-checkpassed, 54 of 54. It built the image once for the new key and reused it for all 3 jobs, with one warning per job ("… is not published"). It passed again with no gh token.ghwith the runner's exact arguments and env:--source-ref refs/heads/nope: exit 1.Still to do after #7671 lands and publishes
inputs-2bf9…: run the self-check from a checkout with this runner and main's key, and show that it pulls and verifies the image with no build. After this PR merges, its new key gets published, and the next PR's sandbox jobs must pull that image without a build. I will link both runs here.Risks
gh attestation verify --bundle-from-ociin CI. If it is not, CI builds locally with a warning, about 40 s per job, and does not fail. I will check the first CI logs.make bug-bash,make mcp-apps-e2eandmake test-storiesnow builds the image when no verified one exists. That can take up to 25 minutes. Before, it refused.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh• Cost:$266.91