From 5f8509f6eb11a9edf588661c1966225d0e4d379b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 19:23:52 -0800 Subject: [PATCH 1/6] chore(quest): claim quest/m0/pullfrog-setup.md Quest-Claim: 4ab0f0a294b24007 From 5863fb30689bf28207b34ba7f521e7d24672b70b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 12 Nov 2026 19:24:33 -0800 Subject: [PATCH 2/6] quest: record rind's Pullfrog setup, narrow to the rest Co-Authored-By: Claude Opus 5.5 --- quest/m0/pullfrog-setup.md | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/quest/m0/pullfrog-setup.md b/quest/m0/pullfrog-setup.md index d6c338e7..21ae0e7b 100644 --- a/quest/m0/pullfrog-setup.md +++ b/quest/m0/pullfrog-setup.md @@ -37,14 +37,17 @@ enters the dev shell from the default branch's checkout, and the bot runs no checks or tests from the PR's head; confirm how Pullfrog checks out a fork PR's head and that the agent's instructions say so. -Open for the implementer, with recommendations: - -- Where the step runs. Pullfrog's per-repository `hooks.setup` setting - (`pullfrog config set hooks.setup ... --repo moq-dev/`) or steps - before the action in `.github/workflows/pullfrog.yml`, whose header says to - edit only where indicated. Recommended: `hooks.setup` if it runs with the - privileges a Nix install and a submodule fetch need; check Pullfrog's docs - and one run's log. -- Which repositories. Measure the added setup time on one run; recommended: - every repository whose reviews would run its tests (rind first), and none - where the fallback alone covers what reviews need. +Done for rind in kixel-ai/rind#714: steps before the action in +`pullfrog.yml` (its `nix` and `vectors` actions, then `nix develop --command +true`). The workflow is dispatch-only from the default branch, so its checkout +is trusted. `hooks.setup` was not used, since the CLI could not read the +repository's settings without the maintainer's Pullfrog login. + +Remaining: once #714 merges, measure one review run's added setup time and +confirm `just test` passes. Then apply the same steps to studio, rust2, quest, +and web-transport where their reviews would run tests. + +## Waiting + +- Maintainer: merge kixel-ai/rind#714, then dispatch one Pullfrog review in + rind and check its log for the setup time and a passing `just test` From 35653b7a37cfa18b9df3227e98e69d47cb6f1a1d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 09:55:47 -0800 Subject: [PATCH 3/6] chore(quest): claim quest/m0/pullfrog-setup.md Quest-Claim: c5b5931035e623f5 From d732e67f6acefddcd48034c6c63c2639c654269a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 10:10:19 -0800 Subject: [PATCH 4/6] ci(pullfrog): give reviews the dev shell's tools Rind needs no setup now that its vectors are vendored; a dispatched test run on its main passed. Studio, rust2, and web-transport get the same steps in their own PRs; the quest waits on those and the post-merge runs. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/pullfrog.yml | 22 +++++++ quest/m0/pullfrog-setup.md | 101 ++++++++++----------------------- 2 files changed, 53 insertions(+), 70 deletions(-) diff --git a/.github/workflows/pullfrog.yml b/.github/workflows/pullfrog.yml index 5d516c9c..efc7a881 100644 --- a/.github/workflows/pullfrog.yml +++ b/.github/workflows/pullfrog.yml @@ -29,6 +29,28 @@ jobs: with: fetch-depth: 1 persist-credentials: false + # Edited: what the review needs to run the repository's checks and + # tests. The environment admits only main, so the dev shell comes from + # the default branch, not a pull request's head. + - uses: DeterminateSystems/nix-installer-action@1d87d45818068401a10cf16bdc5f00b24994a83f # main + with: + determinate: false + - name: Put the dev shell's tools on PATH + run: | + nix print-dev-env --json | + jq -r '.variables.PATH.value' | + tr ':' '\n' | grep '^/nix/store/' | paste -sd: >>"$GITHUB_PATH" + # The installer wrote the job's token into world-readable nix.conf, + # where the agent's shell could read it. + - name: Remove the job token from Nix's config + env: + TOKEN: ${{ github.token }} + run: | + sudo sed -i '/^access-tokens/d' /etc/nix/nix.conf + if sudo grep -rqF "$TOKEN" /etc/nix; then + echo "the job token is still under /etc/nix" >&2 + exit 1 + fi - name: Run agent uses: pullfrog/pullfrog@0741f79ecebfa4469a010644d2aed2d96d32d489 # v0.1.127 with: diff --git a/quest/m0/pullfrog-setup.md b/quest/m0/pullfrog-setup.md index ca8ab580..2418a662 100644 --- a/quest/m0/pullfrog-setup.md +++ b/quest/m0/pullfrog-setup.md @@ -1,4 +1,4 @@ -# [S] Pullfrog reviews have what their checks need +# [XS] Pullfrog reviews have what their checks need ## Goal @@ -9,75 +9,36 @@ repository's checks and tests instead of reasoning around their absence. ## Plan -Decided 2026-11-12 (user, /quest-plan): ✅ both a reference-line fallback to -the rules on GitHub, now in `quest init`, and a setup step here -(rejected: only the fallback; only this setup step). - -Pullfrog reviews with Claude since 2026-11-12, its model access configured in -the Pullfrog console, and its workflow is back in moq-dev/quest (#659), -kixel-ai/rind, moq-dev/rust2, moq-dev/studio, and moq-dev/web-transport after -the 2026-11-10 removal [token-waste](/quest/m0/token-waste/README.md) records. -Since 2026-11-13 its approval, which an identical `Pullfrog approval` ruleset -requires in every moq-dev and kixel-ai repository (kept in kixel-ai/dotfiles' -`bin/repo-settings`), gates merges in place of the DeepSeek review, so what its -runs can check now decides merges. - -Evidence from the trimmed Pullfrog learnings: rind's bot could not fetch its -test vector submodules because `.git` is read-only, so about 20 tests failed; -studio, rust2, quest, and web-transport each recorded a missing `quest`, -`nix`, or `just`. - -Decided 2026-11-12 (user, iterating -[kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)): ✅ files a -repository's tests read are checked in, so a plain clone runs every test -(rejected: fetching them in a Pullfrog setup step on every run; pinning them -as Nix flake inputs, which still needs Nix in the reviewer). That PR copies -the Wycheproof, CAVP, and CCTV files its tests open into `thirdparty/vectors/` -with `admin/sync-vectors`, and rind's Pullfrog workflow gets no setup step: -`ubuntu-latest` already has cargo, and vectors were its only recorded -problem. Until that PR merges, rind's `main` still has the submodules. - -Remaining: - -- Rind: once its PR merges, dispatch one Pullfrog run that runs rind's - tests (`gh workflow run pullfrog.yml -R kixel-ai/rind -f prompt='Run cargo - test --workspace on main and report any failure'`; the workflow takes its - task only from `prompt`) and check its log for passing tests. Rind is done when that run passes, and needs this quest's - setup work only if it shows another gap. -- Studio, rust2, quest, and web-transport, whose gaps are tools, not - files. Prefer making the bot need no tool it lacks, as rind did, over - installing one. Where a repository's checks still need Nix or `just`, add - only that. - -Trust boundary: the run holds `id-token: write` to mint the token Pullfrog -trades for model access, and `assets/CONTEXT.md` forbids running code from a -PR whose author lacks write access. Recommended: on such a PR the setup step -enters the dev shell from the default branch's checkout, and the bot runs no -checks or tests from the PR's head; confirm how Pullfrog checks out a fork -PR's head and that the agent's instructions say so. - -Token exposure: `DeterminateSystems/nix-installer-action` writes the job's -`GITHUB_TOKEN` into the world-readable `/etc/nix/nix.conf` as -`access-tokens`, where the agent's shell can read it, as Pullfrog's review of -[kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)'s first -approach found. A setup step that installs Nix must delete that line before -the agent starts, whether or not it builds a dev shell. - -Open for the implementer, with recommendations: - -- Where the step runs. Pullfrog's per-repository `hooks.setup` setting - (`pullfrog config set hooks.setup ... --repo /`) or steps - before the action in `.github/workflows/pullfrog.yml`, whose header says to - edit only where indicated. Recommended: workflow steps, since the Pullfrog - CLI could not read a repository's settings without the maintainer's login; - `hooks.setup` if that changes and it has the privileges a Nix install needs. -- Which repositories. Measure the added setup time on one run; recommended: - every repository whose reviews would run its tests, and none where the - fallback alone covers what reviews need. +Done: rind needs no setup. With its test vectors vendored +([kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)), a +dispatched run of `cargo test --workspace` on its `main` passed on +2026-11-15 (1785 passed, 0 failed, 53 ignored on purpose). + +Studio, rust2, quest, and web-transport check through their Nix dev shells, +so their `pullfrog.yml` gains steps before the agent +([moq-dev/quest#751](https://github.com/moq-dev/quest/pull/751) and the PRs +under `Waiting`): install Nix, put the `/nix/store` entries of `main`'s +`nix print-dev-env` `PATH` on `GITHUB_PATH` (Pullfrog's restricted shell +keeps `PATH` but drops other dev-shell variables, so `nix develop --command` +stays the way to anything needing them), and delete the job token the +installer writes into `/etc/nix/nix.conf`, failing if it remains. The +`pullfrog` environment admits only `main`, so these steps always build the +default branch's shell; `checkout_pr` moves to a PR's head only afterwards, +and `quest context`, now on the agent's PATH, carries the rule against +running a non-writer's PR code. + +Remaining: once a repository's workflow change is on `main`, dispatch one +run that runs its checks, such as `gh workflow run pullfrog.yml -R +moq-dev/studio -f prompt='Run just check and just test on main and report +any failure, and how long the workflow steps before the agent took'`, and +check its log. A run that shows another missing tool or variable gets it +added; the setup steps' time goes in the PR that finishes this quest. ## Waiting -- kixel-ai/rind: merge [kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714), - which vendors rind's test vectors; check that rind's `main` has - `thirdparty/vectors/{wycheproof,cavp,cctv}/` and none of those three - submodules. The review run the Plan's Remaining list names follows it. +- kixel-ai/rust2: merge [kixel-ai/rust2#841](https://github.com/kixel-ai/rust2/pull/841), + which adds the setup steps; then dispatch its run. +- moq-dev/studio: merge [moq-dev/studio#182](https://github.com/moq-dev/studio/pull/182), + which adds the setup steps; then dispatch its run. +- moq-dev/web-transport: merge [moq-dev/web-transport#545](https://github.com/moq-dev/web-transport/pull/545), + which adds the setup steps; then dispatch its run. From cedeb9af730c393ad68a7771038f56646379aaa0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 10:15:09 -0800 Subject: [PATCH 5/6] ci(pullfrog): fail the PATH step when print-dev-env fails Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/pullfrog.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/pullfrog.yml b/.github/workflows/pullfrog.yml index efc7a881..dee5f519 100644 --- a/.github/workflows/pullfrog.yml +++ b/.github/workflows/pullfrog.yml @@ -36,6 +36,9 @@ jobs: with: determinate: false - name: Put the dev shell's tools on PATH + # bash, not the default shell, for pipefail: a failing print-dev-env + # must fail the step, not write an empty PATH. + shell: bash run: | nix print-dev-env --json | jq -r '.variables.PATH.value' | From 4f903210afdc40108ce76285d86a5fbfa93a6934 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 15 Nov 2026 10:16:21 -0800 Subject: [PATCH 6/6] ci(pullfrog): scrub the job token from nix.custom.conf and the receipt Co-Authored-By: Claude Sonnet 5.5 --- .github/workflows/pullfrog.yml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/pullfrog.yml b/.github/workflows/pullfrog.yml index dee5f519..efea91c9 100644 --- a/.github/workflows/pullfrog.yml +++ b/.github/workflows/pullfrog.yml @@ -43,13 +43,15 @@ jobs: nix print-dev-env --json | jq -r '.variables.PATH.value' | tr ':' '\n' | grep '^/nix/store/' | paste -sd: >>"$GITHUB_PATH" - # The installer wrote the job's token into world-readable nix.conf, - # where the agent's shell could read it. + # The installer wrote the job's token into world-readable nix.conf and + # nix.custom.conf, and into /nix/receipt.json's plan, where the agent's + # shell could read it. - name: Remove the job token from Nix's config env: TOKEN: ${{ github.token }} run: | - sudo sed -i '/^access-tokens/d' /etc/nix/nix.conf + sudo find /etc/nix -type f -exec sed -i '/^access-tokens/d' {} + + sudo chmod 0600 /nix/receipt.json if sudo grep -rqF "$TOKEN" /etc/nix; then echo "the job token is still under /etc/nix" >&2 exit 1