diff --git a/.github/workflows/pullfrog.yml b/.github/workflows/pullfrog.yml index 5d516c9c..efea91c9 100644 --- a/.github/workflows/pullfrog.yml +++ b/.github/workflows/pullfrog.yml @@ -29,6 +29,33 @@ jobs: with: fetch-depth: 1 persist-credentials: false + # Edited: what the review needs to run the repository's checks and + # tests. The environment admits only main, so the dev shell comes from + # the default branch, not a pull request's head. + - uses: DeterminateSystems/nix-installer-action@1d87d45818068401a10cf16bdc5f00b24994a83f # main + with: + determinate: false + - name: Put the dev shell's tools on PATH + # bash, not the default shell, for pipefail: a failing print-dev-env + # must fail the step, not write an empty PATH. + shell: bash + run: | + nix print-dev-env --json | + jq -r '.variables.PATH.value' | + tr ':' '\n' | grep '^/nix/store/' | paste -sd: >>"$GITHUB_PATH" + # The installer wrote the job's token into world-readable nix.conf and + # nix.custom.conf, and into /nix/receipt.json's plan, where the agent's + # shell could read it. + - name: Remove the job token from Nix's config + env: + TOKEN: ${{ github.token }} + run: | + sudo find /etc/nix -type f -exec sed -i '/^access-tokens/d' {} + + sudo chmod 0600 /nix/receipt.json + if sudo grep -rqF "$TOKEN" /etc/nix; then + echo "the job token is still under /etc/nix" >&2 + exit 1 + fi - name: Run agent uses: pullfrog/pullfrog@0741f79ecebfa4469a010644d2aed2d96d32d489 # v0.1.127 with: diff --git a/quest/m0/pullfrog-setup.md b/quest/m0/pullfrog-setup.md index ca8ab580..2418a662 100644 --- a/quest/m0/pullfrog-setup.md +++ b/quest/m0/pullfrog-setup.md @@ -1,4 +1,4 @@ -# [S] Pullfrog reviews have what their checks need +# [XS] Pullfrog reviews have what their checks need ## Goal @@ -9,75 +9,36 @@ repository's checks and tests instead of reasoning around their absence. ## Plan -Decided 2026-11-12 (user, /quest-plan): ✅ both a reference-line fallback to -the rules on GitHub, now in `quest init`, and a setup step here -(rejected: only the fallback; only this setup step). - -Pullfrog reviews with Claude since 2026-11-12, its model access configured in -the Pullfrog console, and its workflow is back in moq-dev/quest (#659), -kixel-ai/rind, moq-dev/rust2, moq-dev/studio, and moq-dev/web-transport after -the 2026-11-10 removal [token-waste](/quest/m0/token-waste/README.md) records. -Since 2026-11-13 its approval, which an identical `Pullfrog approval` ruleset -requires in every moq-dev and kixel-ai repository (kept in kixel-ai/dotfiles' -`bin/repo-settings`), gates merges in place of the DeepSeek review, so what its -runs can check now decides merges. - -Evidence from the trimmed Pullfrog learnings: rind's bot could not fetch its -test vector submodules because `.git` is read-only, so about 20 tests failed; -studio, rust2, quest, and web-transport each recorded a missing `quest`, -`nix`, or `just`. - -Decided 2026-11-12 (user, iterating -[kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)): ✅ files a -repository's tests read are checked in, so a plain clone runs every test -(rejected: fetching them in a Pullfrog setup step on every run; pinning them -as Nix flake inputs, which still needs Nix in the reviewer). That PR copies -the Wycheproof, CAVP, and CCTV files its tests open into `thirdparty/vectors/` -with `admin/sync-vectors`, and rind's Pullfrog workflow gets no setup step: -`ubuntu-latest` already has cargo, and vectors were its only recorded -problem. Until that PR merges, rind's `main` still has the submodules. - -Remaining: - -- Rind: once its PR merges, dispatch one Pullfrog run that runs rind's - tests (`gh workflow run pullfrog.yml -R kixel-ai/rind -f prompt='Run cargo - test --workspace on main and report any failure'`; the workflow takes its - task only from `prompt`) and check its log for passing tests. Rind is done when that run passes, and needs this quest's - setup work only if it shows another gap. -- Studio, rust2, quest, and web-transport, whose gaps are tools, not - files. Prefer making the bot need no tool it lacks, as rind did, over - installing one. Where a repository's checks still need Nix or `just`, add - only that. - -Trust boundary: the run holds `id-token: write` to mint the token Pullfrog -trades for model access, and `assets/CONTEXT.md` forbids running code from a -PR whose author lacks write access. Recommended: on such a PR the setup step -enters the dev shell from the default branch's checkout, and the bot runs no -checks or tests from the PR's head; confirm how Pullfrog checks out a fork -PR's head and that the agent's instructions say so. - -Token exposure: `DeterminateSystems/nix-installer-action` writes the job's -`GITHUB_TOKEN` into the world-readable `/etc/nix/nix.conf` as -`access-tokens`, where the agent's shell can read it, as Pullfrog's review of -[kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)'s first -approach found. A setup step that installs Nix must delete that line before -the agent starts, whether or not it builds a dev shell. - -Open for the implementer, with recommendations: - -- Where the step runs. Pullfrog's per-repository `hooks.setup` setting - (`pullfrog config set hooks.setup ... --repo /`) or steps - before the action in `.github/workflows/pullfrog.yml`, whose header says to - edit only where indicated. Recommended: workflow steps, since the Pullfrog - CLI could not read a repository's settings without the maintainer's login; - `hooks.setup` if that changes and it has the privileges a Nix install needs. -- Which repositories. Measure the added setup time on one run; recommended: - every repository whose reviews would run its tests, and none where the - fallback alone covers what reviews need. +Done: rind needs no setup. With its test vectors vendored +([kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)), a +dispatched run of `cargo test --workspace` on its `main` passed on +2026-11-15 (1785 passed, 0 failed, 53 ignored on purpose). + +Studio, rust2, quest, and web-transport check through their Nix dev shells, +so their `pullfrog.yml` gains steps before the agent +([moq-dev/quest#751](https://github.com/moq-dev/quest/pull/751) and the PRs +under `Waiting`): install Nix, put the `/nix/store` entries of `main`'s +`nix print-dev-env` `PATH` on `GITHUB_PATH` (Pullfrog's restricted shell +keeps `PATH` but drops other dev-shell variables, so `nix develop --command` +stays the way to anything needing them), and delete the job token the +installer writes into `/etc/nix/nix.conf`, failing if it remains. The +`pullfrog` environment admits only `main`, so these steps always build the +default branch's shell; `checkout_pr` moves to a PR's head only afterwards, +and `quest context`, now on the agent's PATH, carries the rule against +running a non-writer's PR code. + +Remaining: once a repository's workflow change is on `main`, dispatch one +run that runs its checks, such as `gh workflow run pullfrog.yml -R +moq-dev/studio -f prompt='Run just check and just test on main and report +any failure, and how long the workflow steps before the agent took'`, and +check its log. A run that shows another missing tool or variable gets it +added; the setup steps' time goes in the PR that finishes this quest. ## Waiting -- kixel-ai/rind: merge [kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714), - which vendors rind's test vectors; check that rind's `main` has - `thirdparty/vectors/{wycheproof,cavp,cctv}/` and none of those three - submodules. The review run the Plan's Remaining list names follows it. +- kixel-ai/rust2: merge [kixel-ai/rust2#841](https://github.com/kixel-ai/rust2/pull/841), + which adds the setup steps; then dispatch its run. +- moq-dev/studio: merge [moq-dev/studio#182](https://github.com/moq-dev/studio/pull/182), + which adds the setup steps; then dispatch its run. +- moq-dev/web-transport: merge [moq-dev/web-transport#545](https://github.com/moq-dev/web-transport/pull/545), + which adds the setup steps; then dispatch its run.