Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/pullfrog.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,33 @@ jobs:
with:
fetch-depth: 1
persist-credentials: false
# Edited: what the review needs to run the repository's checks and
# tests. The environment admits only main, so the dev shell comes from
# the default branch, not a pull request's head.
- uses: DeterminateSystems/nix-installer-action@1d87d45818068401a10cf16bdc5f00b24994a83f # main
with:
determinate: false
- name: Put the dev shell's tools on PATH
# bash, not the default shell, for pipefail: a failing print-dev-env
# must fail the step, not write an empty PATH.
shell: bash
run: |
Comment thread
pullfrog[bot] marked this conversation as resolved.
Resolved
nix print-dev-env --json |
jq -r '.variables.PATH.value' |
tr ':' '\n' | grep '^/nix/store/' | paste -sd: >>"$GITHUB_PATH"
# The installer wrote the job's token into world-readable nix.conf and
# nix.custom.conf, and into /nix/receipt.json's plan, where the agent's
# shell could read it.
- name: Remove the job token from Nix's config
env:
TOKEN: ${{ github.token }}
run: |
sudo find /etc/nix -type f -exec sed -i '/^access-tokens/d' {} +
sudo chmod 0600 /nix/receipt.json
if sudo grep -rqF "$TOKEN" /etc/nix; then
echo "the job token is still under /etc/nix" >&2
exit 1
fi
- name: Run agent
uses: pullfrog/pullfrog@0741f79ecebfa4469a010644d2aed2d96d32d489 # v0.1.127
with:
Expand Down
101 changes: 31 additions & 70 deletions quest/m0/pullfrog-setup.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# [S] Pullfrog reviews have what their checks need
# [XS] Pullfrog reviews have what their checks need

## Goal

Expand All @@ -9,75 +9,36 @@ repository's checks and tests instead of reasoning around their absence.

## Plan

Decided 2026-11-12 (user, /quest-plan): ✅ both a reference-line fallback to
the rules on GitHub, now in `quest init`, and a setup step here
(rejected: only the fallback; only this setup step).

Pullfrog reviews with Claude since 2026-11-12, its model access configured in
the Pullfrog console, and its workflow is back in moq-dev/quest (#659),
kixel-ai/rind, moq-dev/rust2, moq-dev/studio, and moq-dev/web-transport after
the 2026-11-10 removal [token-waste](/quest/m0/token-waste/README.md) records.
Since 2026-11-13 its approval, which an identical `Pullfrog approval` ruleset
requires in every moq-dev and kixel-ai repository (kept in kixel-ai/dotfiles'
`bin/repo-settings`), gates merges in place of the DeepSeek review, so what its
runs can check now decides merges.

Evidence from the trimmed Pullfrog learnings: rind's bot could not fetch its
test vector submodules because `.git` is read-only, so about 20 tests failed;
studio, rust2, quest, and web-transport each recorded a missing `quest`,
`nix`, or `just`.

Decided 2026-11-12 (user, iterating
[kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)): ✅ files a
repository's tests read are checked in, so a plain clone runs every test
(rejected: fetching them in a Pullfrog setup step on every run; pinning them
as Nix flake inputs, which still needs Nix in the reviewer). That PR copies
the Wycheproof, CAVP, and CCTV files its tests open into `thirdparty/vectors/`
with `admin/sync-vectors`, and rind's Pullfrog workflow gets no setup step:
`ubuntu-latest` already has cargo, and vectors were its only recorded
problem. Until that PR merges, rind's `main` still has the submodules.

Remaining:

- Rind: once its PR merges, dispatch one Pullfrog run that runs rind's
tests (`gh workflow run pullfrog.yml -R kixel-ai/rind -f prompt='Run cargo
test --workspace on main and report any failure'`; the workflow takes its
task only from `prompt`) and check its log for passing tests. Rind is done when that run passes, and needs this quest's
setup work only if it shows another gap.
- Studio, rust2, quest, and web-transport, whose gaps are tools, not
files. Prefer making the bot need no tool it lacks, as rind did, over
installing one. Where a repository's checks still need Nix or `just`, add
only that.

Trust boundary: the run holds `id-token: write` to mint the token Pullfrog
trades for model access, and `assets/CONTEXT.md` forbids running code from a
PR whose author lacks write access. Recommended: on such a PR the setup step
enters the dev shell from the default branch's checkout, and the bot runs no
checks or tests from the PR's head; confirm how Pullfrog checks out a fork
PR's head and that the agent's instructions say so.

Token exposure: `DeterminateSystems/nix-installer-action` writes the job's
`GITHUB_TOKEN` into the world-readable `/etc/nix/nix.conf` as
`access-tokens`, where the agent's shell can read it, as Pullfrog's review of
[kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)'s first
approach found. A setup step that installs Nix must delete that line before
the agent starts, whether or not it builds a dev shell.

Open for the implementer, with recommendations:

- Where the step runs. Pullfrog's per-repository `hooks.setup` setting
(`pullfrog config set hooks.setup ... --repo <owner>/<repo>`) or steps
before the action in `.github/workflows/pullfrog.yml`, whose header says to
edit only where indicated. Recommended: workflow steps, since the Pullfrog
CLI could not read a repository's settings without the maintainer's login;
`hooks.setup` if that changes and it has the privileges a Nix install needs.
- Which repositories. Measure the added setup time on one run; recommended:
every repository whose reviews would run its tests, and none where the
fallback alone covers what reviews need.
Done: rind needs no setup. With its test vectors vendored
([kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714)), a
dispatched run of `cargo test --workspace` on its `main` passed on
2026-11-15 (1785 passed, 0 failed, 53 ignored on purpose).

Studio, rust2, quest, and web-transport check through their Nix dev shells,
so their `pullfrog.yml` gains steps before the agent
([moq-dev/quest#751](https://github.com/moq-dev/quest/pull/751) and the PRs
under `Waiting`): install Nix, put the `/nix/store` entries of `main`'s
`nix print-dev-env` `PATH` on `GITHUB_PATH` (Pullfrog's restricted shell
keeps `PATH` but drops other dev-shell variables, so `nix develop --command`
stays the way to anything needing them), and delete the job token the
installer writes into `/etc/nix/nix.conf`, failing if it remains. The
`pullfrog` environment admits only `main`, so these steps always build the
default branch's shell; `checkout_pr` moves to a PR's head only afterwards,
and `quest context`, now on the agent's PATH, carries the rule against
running a non-writer's PR code.

Remaining: once a repository's workflow change is on `main`, dispatch one
run that runs its checks, such as `gh workflow run pullfrog.yml -R
moq-dev/studio -f prompt='Run just check and just test on main and report
any failure, and how long the workflow steps before the agent took'`, and
check its log. A run that shows another missing tool or variable gets it
added; the setup steps' time goes in the PR that finishes this quest.

## Waiting

- kixel-ai/rind: merge [kixel-ai/rind#714](https://github.com/kixel-ai/rind/pull/714),
which vendors rind's test vectors; check that rind's `main` has
`thirdparty/vectors/{wycheproof,cavp,cctv}/` and none of those three
submodules. The review run the Plan's Remaining list names follows it.
- kixel-ai/rust2: merge [kixel-ai/rust2#841](https://github.com/kixel-ai/rust2/pull/841),
which adds the setup steps; then dispatch its run.
- moq-dev/studio: merge [moq-dev/studio#182](https://github.com/moq-dev/studio/pull/182),
which adds the setup steps; then dispatch its run.
- moq-dev/web-transport: merge [moq-dev/web-transport#545](https://github.com/moq-dev/web-transport/pull/545),
which adds the setup steps; then dispatch its run.
Loading