🤖 tests: resolve the sandbox image by stable tag and attestation, else build locally

.github/workflows/bugbash-sandbox-image.yml

@@ -7,9 +7,9 @@
 # key has that tag pushes nothing. Otherwise it builds fresh, pushes, attests that build's own
 # digest, and only then points the stable tag at the same digest. Nothing here deletes an image.
 #
-# Runners still trust only the digest in image.json: a person copies it from the job summary into
-# image.json in a pull request (no workflow writes to the repo). Until that PR merges, runners
-# refuse a checkout whose image inputs changed (stale inputs key).
+# Runners (tests/bugbash/sandbox/runner.ts) look up the stable tag of their checkout's key and
+# accept it only with this workflow's attestation from main. Without one they build the image
+# from their own checkout, so a pull request that changes an image input passes before the merge.
 name: Bug-bash sandbox image
 
 on:
@@ -65,7 +65,7 @@ jobs:
       contents: read
       # Push ghcr.io/coder/xum-bugbash-sandbox. Only this job, only on main.
       packages: write
-      # Build provenance for audit (the runner trusts the digest in image.json).
+      # Build provenance: runners verify it before they use the stable tag's digest.
       id-token: write
       attestations: write
     steps:
@@ -94,21 +94,23 @@ jobs:
             cat "$RUNNER_TEMP/inspect.err" >&2
             exit 1
           fi
-          make -s bugbash-sandbox-publish > image.json
-          # The subject of the attestation is this build's own output (image.json). Assignments, not
-          # $(...) inside echo: a failed jq must fail the step, or an empty digest would skip Attest.
-          image=$(jq -er .image image.json)
-          digest=$(jq -er .digest image.json)
+          record="$RUNNER_TEMP/publish.json"
+          make -s bugbash-sandbox-publish > "$record"
+          # The subject of the attestation is this build's own output (the publish record).
+          # Assignments, not $(...) inside echo: a failed jq must fail the step, or an empty
+          # digest would skip Attest.
+          image=$(jq -er .image "$record")
+          digest=$(jq -er .digest "$record")
           {
             echo "image=$image"
             echo "digest=$digest"
             echo "tag=$tag"
           } >> "$GITHUB_OUTPUT"
           {
-            echo '### tests/bugbash/sandbox/image.json'
+            echo '### Publish record'
             echo
             echo '```json'
-            cat image.json
+            cat "$record"
             echo '```'
           } >> "$GITHUB_STEP_SUMMARY"
       - name: Attest
@@ -119,6 +121,9 @@ jobs:
           subject-name: ${{ steps.push.outputs.image }}
           subject-digest: ${{ steps.push.outputs.digest }}
           push-to-registry: true
+          # Runners verify the attestation bundle in GHCR and read no artifact storage record.
+          # Creating one needs artifact-metadata: write, and without it the action warns.
+          create-storage-record: false
       # The stable tag marks the key as published, so it comes only after the attestation: a failed
       # Attest leaves no tag, and a re-run builds and attests again instead of skipping the key.
       - name: Tag inputs-<key>

.github/workflows/pr.yml

@@ -843,33 +843,35 @@ jobs:
   bugbash-sandbox:
     # The bug-bash sandbox (#5714, #5931): its self-check (isolation, proxy and export refusals,
     # against a fake upstream) and the repros of fixed bugs through its launcher, on the mock app
-    # AI. No secrets and no model. The launcher pulls the pinned image by digest without
-    # credentials, and exits nonzero on a refusal, a failed check or a container left `unknown`.
+    # AI. No secrets and no model. The launcher pulls main's attested image by digest without
+    # registry credentials (or builds one from this checkout), and exits nonzero on a refusal, a
+    # failed check or a container left `unknown`.
     # E2E shard 1 still runs the same repros on the host until this job proves stable.
     name: Test / Bug-bash sandbox
     needs: [changes]
     # Queue only on PR pushes, unless the PR changes tests/bugbash (see test-e2e).
     if: >-
       needs.changes.outputs.tests == 'true' && github.event.inputs.test_filter == '' &&
       (github.event_name != 'pull_request' || needs.changes.outputs.e2e_tests == 'true')
-    # 45, not 30: a stale image key adds a local image build of up to 25 minutes (below).
+    # 45, not 30: a PR that changes the image inputs has no published image yet (main publishes
+    # it after the merge), so the launcher builds one locally, for up to 25 minutes.
     timeout-minutes: 45
     runs-on: ${{ needs.changes.outputs.runner-4 }}
-    # Stale-key local build: only main publishes the image, so a PR that changes its inputs (Dockerfile, build.sh,
-    # playwright-core) has a stale image.json. With this opt-in the launcher then builds the image
-    # locally with build.sh, for these two jobs only, and prints a warning. Model-driven runs
-    # (make bug-bash, make mcp-apps-e2e) ignore it and keep the pinned digest.
-    env:
-      BUGBASH_SANDBOX_BUILD_STALE: "1"
     steps:
       - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
         with:
           persist-credentials: false
       - uses: ./.github/actions/setup-xum
+      # GH_TOKEN: the launcher's gh verifies the published image's attestation on the host. It
+      # never reaches Docker or a job container (the container env drops it).
       - name: Sandbox self-check
         run: make bug-bash-sandbox-check
+        env:
+          GH_TOKEN: ${{ github.token }}
       - name: Repros in the sandbox
         run: make test-bugbash-repros-sandbox
+        env:
+          GH_TOKEN: ${{ github.token }}
 
   test-windows:
     name: Test / Windows

.github/workflows/stories.yml

@@ -107,9 +107,6 @@ jobs:
       # failed download apart from the jobs.
       - name: make electron-binary
         run: make electron-binary
-      # make test-stories refuses a stale sandbox image key (a PR that changes the Dockerfile):
-      # BUGBASH_SANDBOX_BUILD_STALE does not apply to model-driven jobs (launch.ts), so such a PR
-      # fails here until main publishes the image and image.json pins it.
       # One more run only when every failed job got a provider 5xx (summary.ts --upstream-fault):
       # the user's rule, so that an overloaded provider does not fail a queue entry. A test
       # failure, a 4xx or a refusal fails at once. Before that run, the first run's outputs move
@@ -135,6 +132,9 @@ jobs:
           # The cap per job, so one run spends at most three times this, six with the retry. $3
           # unless the caller passes budget_usd ($1 for the PR lane).
           BUGBASH_BUDGET_USD: ${{ inputs.budget_usd || '3' }}
+          # The launcher's gh verifies the published sandbox image's attestation on the host. It
+          # never reaches Docker or a job container (the container env drops it).
+          GH_TOKEN: ${{ github.token }}
       # The e2e page of each job, as in the PR comment (summary.ts).
       - name: Summary
         if: always()

.xum/skills/bug-bash/SKILL.md

@@ -26,20 +26,22 @@ Xum harness. Where the two differ, follow this skill.
   `tests/bugbash/sandbox/launch.ts`, one container per job.
 - The image `ghcr.io/coder/xum-bugbash-sandbox` (about 1.4 GB, linux/amd64) holds only tools:
   Node 22, Bun, git, the Chromium of `@e2e-dev/web`'s playwright-core, and Xvfb plus libgtk-3
-  for Xum Electron (the Electron binary comes from the mounted node_modules). The runner pulls it only
-  by the digest in `tests/bugbash/sandbox/image.json`. It builds it only in the case below.
+  for Xum Electron (the Electron binary comes from the mounted node_modules).
 - The inputs key (`make bugbash-sandbox-key`) is a hash of the Dockerfile, `build.sh` and the
-  playwright-core version in `bun.lock`. The runner refuses an image whose key differs from the
-  checkout. One exception: with `BUGBASH_SANDBOX_BUILD_STALE=1` (exactly `1`), which only CI's
-  `Test / Bug-bash sandbox` job sets, the self-check and the exact-step repros build the image
-  locally with `build.sh` (local tag, no push, no registry login), check its inputs label, and
-  print a warning (a `::warning::` annotation in GitHub Actions). Explore charters, the MCP Apps
-  suite, the story tests, the agent repros and the preflight of `make bug-bash`,
-  `make mcp-apps-e2e` and `make test-stories` always refuse a stale key.
-- An image change lands in three steps: the PR that changes the Dockerfile, `build.sh` or
-  playwright-core passes CI on a local build. Its merge push runs the workflow "Bug-bash sandbox
+  playwright-core version in `bun.lock`. Every job, in CI and on your machine, looks up the tag
+  `inputs-<key>` anonymously and verifies its attestation on the host with
+  `gh attestation verify` (main's workflow "Bug-bash sandbox image", `refs/heads/main`). gh needs
+  your `gh auth login` or `GH_TOKEN`. The token stays on the host: no job container gets it.
+  A verified image is pulled by digest.
+- Any other outcome (no tag, a lookup error, no gh, no token, a failed verification) logs the
+  reason, builds the image locally with `build.sh` (local tag, no push, no registry login) or
+  reuses that build, and prints a LOCAL IMAGE warning (a `::warning::` annotation in GitHub
+  Actions). Both images must carry the checkout's key as their inputs label. A local build adds
+  no power: each job already runs its checkout's files, `dist/` and `node_modules/`.
+- An image change lands in one PR: the PR that changes the Dockerfile, `build.sh` or
+  playwright-core runs on a local build. Its merge push runs the workflow "Bug-bash sandbox
   image" on main, which publishes the image once for each inputs key (a manual run on main
-  retries a failed publish). Then a reviewed PR updates `image.json` to pin the new digest. An e2e update that keeps playwright-core keeps the key, so the image stays valid.
+  retries a failed publish). An e2e update that keeps playwright-core keeps the key, so the image stays valid.
 - The container gets `--network none`, `--read-only`, `--cap-drop ALL`, no-new-privileges, a
   non-root user, 4 GB memory and tmpfs scratch folders. It reads copies of the git-listed files,
   plus `dist/` and `node_modules/` read-only. An env allowlist passes no credential.
@@ -184,7 +186,7 @@ artifact `stories-reports-<attempt>` and, on a failure, the `results/` folders a
 `stories-evidence-<attempt>` (both kept 7 days), and, except in the merge queue, saves the next seed: `bun
 tests/stories/summary.ts --next-seed <dir>` (the caches of the newest web and Electron story
 jobs, less the seed entries either one deleted, at most 1000 entries and 128 MiB). GitHub scopes a saved cache to its ref, and a queue ref is temporary, so a
-merge-queue run saves none. Only the nightly and on-demand runs on main change what main replays. A PR that changes the sandbox image inputs fails there until main publishes the image.
+merge-queue run saves none. Only the nightly and on-demand runs on main change what main replays. A PR that changes the sandbox image inputs has no verified image yet, so each of its jobs builds the image locally (up to 25 minutes) and warns.
 After the PR lane, pr.yml's `Test / Stories (Haiku) comment` job posts the same page as one PR
 comment, which each push edits (`tests/stories/prComment.ts`; `--dry-run` prints the body). It
 runs the base commit's copy with no install scripts, writes only while the run's commit is still
@@ -222,8 +224,8 @@ removes only those whose launcher is provably dead on this boot, and exits 3 whe
 for one. Check that container's `xum.bugbash.owner` and `xum.bugbash.checkout` labels with
 `docker inspect`, then `docker rm -f <id>` it. A SIGKILLed launcher also leaves
 `$TMPDIR/xum-bugbash-sandbox/<checkout>/<job>` and a `xum-bugbash-docker-*` folder in `$TMPDIR`:
-remove both by hand. A run with `BUGBASH_SANDBOX_BUILD_STALE=1` on a stale key leaves a local
-image of about 1.4 GB: remove it with `docker image rm xum-bugbash-sandbox:inputs-<key>`.
+remove both by hand. A run without a verified image leaves a local image
+of about 1.4 GB: remove it with `docker image rm xum-bugbash-sandbox:inputs-<key>`.
 
 ## 5. Triage and verify
 

Makefile

@@ -236,14 +236,14 @@ bug-bash: ## Agent bug bash in the bug-bash sandbox: e2e explore charters x mode
 	@bun tests/bugbash/run.ts $(BUGBASH_ARGS)
 
 # The bug-bash sandbox image (tests/bugbash/sandbox/build.sh, #5714). Each prints one value on
-# stdout. Publishing runs only in the manual "Bug-bash sandbox image" workflow on main.
+# stdout. Publishing runs only in the "Bug-bash sandbox image" workflow on main.
 bugbash-sandbox-key: ## Print the inputs key of the bug-bash sandbox image for this checkout
 	@./tests/bugbash/sandbox/build.sh --key
 
 bugbash-sandbox-image: ## Build the bug-bash sandbox image locally (linux/amd64) and print its image ID
 	@./tests/bugbash/sandbox/build.sh
 
-bugbash-sandbox-publish: ## Workflow only: build and push the bug-bash sandbox image, print its image.json record
+bugbash-sandbox-publish: ## Workflow only: build and push the bug-bash sandbox image, print its publish record
 	@./tests/bugbash/sandbox/build.sh --push
 
 # The sandbox self-check (tests/bugbash/sandbox/selfCheckRun.ts, #5714): three jobs through the real

docs/AGENTS.md

@@ -259,7 +259,7 @@ Freely make breaking changes, and reorganize / cleanup IPC as needed.
 
 - To run a bug bash, the MCP Apps suite, the story tests or the sandbox checks, read the `bug-bash` skill (`.xum/skills/bug-bash/SKILL.md`). It covers the host needs, the env, costs, options, results, exit codes and leftovers.
 - Host pause (#5714): model-driven runs (`make bug-bash`, `make mcp-apps-e2e`, `make test-stories`, any `e2e explore` and every e2e command with `e2e.mcpapps.config.ts`, `tests/stories/e2e.config.ts` or `tests/stories/e2e.electron.config.ts`) run only inside the bug-bash sandbox, one Docker container per job. A model-driven e2e process on the host refuses with exit 2 (`tests/bugbash/hostPause.ts`), and outside the sandbox an `agent.*` step fails with `MODEL_UNAVAILABLE`. No override exists.
-- The sandbox image is pinned by digest in `tests/bugbash/sandbox/image.json`. A change to `tests/bugbash/sandbox/Dockerfile`, `tests/bugbash/sandbox/build.sh` or the playwright-core version of `@e2e-dev/web` in `bun.lock` changes the inputs key (`make bugbash-sandbox-key`), and the runner then refuses the image until the "Bug-bash sandbox image" workflow publishes a new one (it runs on the push to main) and a PR updates `image.json`. One exception: with `BUGBASH_SANDBOX_BUILD_STALE=1`, which only CI's `Test / Bug-bash sandbox` job sets, the self-check and the exact-step repros build the image locally with `build.sh` and warn. So an image change lands in three steps: the Dockerfile PR passes CI on a local build, the merge push makes main publish the image (a manual run on main retries a failed publish), and an `image.json` PR pins the new digest. `make bug-bash`, `make mcp-apps-e2e`, `make test-stories` and their preflight always refuse a stale key.
+- Each sandbox job, in CI and on a developer machine, looks up the image tag `ghcr.io/coder/xum-bugbash-sandbox:inputs-<key>` of its inputs key (`make bugbash-sandbox-key`: a hash of `tests/bugbash/sandbox/Dockerfile`, `tests/bugbash/sandbox/build.sh` and the playwright-core version of `@e2e-dev/web` in `bun.lock`). It uses the tag's digest only when `gh attestation verify` proves that main's "Bug-bash sandbox image" workflow built it (gh needs your login or `GH_TOKEN`). Otherwise it builds the image locally with `build.sh` and warns. That workflow publishes each new key once, on the push to main (a manual run on main retries a failed publish), so a PR that changes an image input runs on a local build until it merges.
 - Every model-driven job and every real-AI app run with `XUM_DISABLE_AGENT_TOOLS=1`, `XUM_DISABLE_TERMINALS=1` and `XUM_DISABLE_PROJECT_AUTOMATION=1`, because AI replies are untrusted input for the explorer. Do not turn them off for a bug bash without a security review. Callers with their own `streamText` tools (refinement, memory harvest, continuous compaction) do not check `XUM_DISABLE_AGENT_TOOLS`. None of them runs shell commands in the seeded app, so a new caller of that kind needs the same check. Settings can still add stdio MCP servers and custom editor commands, so charters must keep explorers out of them.
 - One reviewed exception (#7027, #7029): `BUGBASH_SCENARIO=scripted-tools` runs the app with agent tools on, so repros can see the real tool block. Its only model is a fixed loopback script (`tests/bugbash/scriptedModel.ts`) that ignores typed text and calls only `todo_write` and `file_read` of the seeded README.md. `startApp.ts` refuses it outside the sandbox, in a model-driven job and with a mounted proxy socket, and keeps terminals and project automation off. Only the exact-step sandbox job (`make test-bugbash-repros-sandbox`: no network, no proxy, no explorer model) selects its `tools-on` repros. Do not add tool calls to that script, or run it with a model, without a new security review.
 - A finding is a bug only once a repro fails for the reported reason. Put it in `tests/bugbash/repros/<slug>.e2e.ts` (tracked). Use exact `screen`/`browser`/`expect` steps where they work. A repro may use `agent.*` steps only when each of its tests has the `agent` tag (`tests/bugbash/reproRules.test.ts` enforces this): the free runs pass `--exclude-tag agent`, so only a model-driven sandbox job runs it: `make test-stories` (needs `BUGBASH_BUDGET_USD`). CI's `Stories` workflow (`.github/workflows/stories.yml`) runs that target with Sonnet 5.5 in the merge queue (pr.yml's `Test / Stories` job, which `Required` counts only when the repo variable `STORIES_MODE` is `block`), nightly on main and on demand, and with Haiku 5.5 on each push to a PR that is not a draft (`Test / Stories (Haiku)`, where Sonnet 5.5 judges the story tests' asserts and Haiku the agent repros', and which `Required` counts only when `STORIES_PR_MODE` is `block`). That target also runs the story tests, user stories that an agent drives in `tests/stories/*.e2e.ts`, and the Electron stories in `tests/stories/electron/*.e2e.ts` on Xum Electron under Xvfb (it runs `make electron-binary` first). Prove it: restore the bug in a scratch edit, rebuild, see `ASSERTION_FAILED` on the bug's own assertion, then restore and see it pass.