Skip to content

ci(pullfrog): give reviews the dev shell's tools - #751

Merged
kixelated merged 7 commits into
mainfrom
quest/m0/pullfrog-setup
Nov 15, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m0/pullfrog-setup

Conversation

@kixelated

@kixelated kixelated commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Works quest/m0/pullfrog-setup.md as far as it can go before outside merges.

  • Rind: done, no setup. kixel-ai/rind#714 merged; rind's main has thirdparty/vectors/{wycheproof,cavp,cctv}/, no .gitmodules, and no submodule entries. The dispatched Pullfrog run (run 43743879414) ran cargo test --workspace --no-fail-fast on main: 1785 passed, 0 failed, 53 ignored on purpose (BINSEC, SDE, slow RustCrypto).
  • Quest, studio, rust2, web-transport: setup steps. All four check through their Nix dev shells (quest, just, taplo, nextest, cargo-hack...), so the rind route of needing no tool isn't available. Their pullfrog.yml now, before the agent:
    1. installs Nix (the pinned DeterminateSystems installer, or the repo's ./.github/actions/nix wrapping it),
    2. puts the /nix/store entries of main's nix print-dev-env PATH on GITHUB_PATH,
    3. deletes the access-tokens line the installer writes with the job's GITHUB_TOKEN into the world-readable files under /etc/nix (nix.conf and nix.custom.conf), restricts /nix/receipt.json, which holds the install plan, to root, and fails if the token is still anywhere under /etc/nix.
      Steps 2 and 3 run under shell: bash (pipefail), so a failed nix print-dev-env or an empty PATH fails the step.
  • This repository's change is in this PR; the others:
  • The quest is rewritten to what remains: once each workflow change is on main, one dispatched run that runs its checks, with its setup time, and Waiting items for the three outside merges. Resized S to XS.

Choices made without asking (the Plan's "Open for the implementer" items):

  • Workflow steps, not hooks.setup. Pullfrog runs hooks.setup in its shell sandbox (setpriv --no-new-privs, all capabilities dropped, per its utils/lifecycle.ts and utils/sandbox.ts), so it cannot sudo to install Nix, and setting it needs the maintainer's console login; workflow steps run before the action with the runner's privileges.
  • PATH on GITHUB_PATH, not the whole dev env on GITHUB_ENV. Pullfrog's restricted shell (filterEnv in its utils/secrets.ts) passes PATH but drops variables outside its allowlist, so GITHUB_ENV exports would never reach the agent. nix itself is on PATH too, so nix develop --command still covers anything needing the shell's other variables.
  • Trust boundary. The pullfrog environment admits only main, so the dev shell is always built from the default branch; Pullfrog's checkout_pr moves to a PR's head only after these steps. The rule against running a non-writer's PR code reaches the agent through the repositories' quest context reference line, which now resolves since quest is on PATH.
  • Which repositories. All four: each one's checks need the dev shell. Setup time is measured on the first post-merge run of each (Nix install alone takes 5 to 7 seconds in their CI); it cannot be measured earlier because the environment refuses dispatches from a branch.

Public API

None.

Validation

  • Rind: the dispatched run above.
  • actionlint passes on all four workflows.
  • Locally, the step's nix print-dev-env | jq | grep /nix/store pipeline on each repository's flake, with only that PATH plus /usr/bin:/bin, resolves just and quest in all four, and each repository's own check tools (studio and rust2: cargo, bun, taplo, cargo-nextest, shfmt; web-transport: cargo with its wasm32 target, bun, cargo-hack, cargo-shear, cargo-sort, wasm-bindgen; quest: bun, actionlint, shfmt).
  • Read Pullfrog v0.1.127's source at the pinned SHA for the env filter, sandbox PATH restoration, ACTIONS_ID_TOKEN_REQUEST_* removal, and when checkout_pr runs; read the pinned installer's source for where it writes access-tokens.
  • Pipefail: false | jq | grep | paste exits 1 under bash -eo pipefail and 0 under bash -e. The token scrub's find ... -exec sed leaves no token in a mock nix.conf plus nix.custom.conf. The installer source (latest, which the action downloads) puts extra conf, including access-tokens, in nix.custom.conf and serializes it into the receipt (pending_nix_config), which is written at the default mode.
  • Not exercised: the steps on a real runner, which only a run from main can do; that is the quest's remaining work.
  • just check and just test pass.

Follow-ups

None.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm A)
Implementer: Claude Code, claude-sonnet-5-5, unknown effort (iterate, XS, arm A)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 4f90321

  • Base: origin/main at 361549a

  • Risk: high, it changes .github/workflows/pullfrog.yml, which is always high: workflows run with the repository's secrets and decide what merges and ships

  • Classified: high by the always-high paths

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1e067-c81a-7082-903c-f6d0fa8a08de, round 2

  • Score: 8/10, The incremental workflow changes appear sound by inspection with no actionable defects, but execution was not verified because builds and tests were prohibited.

  • Rounds: 6; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, low effort, shadow

  • Session: 01a1d32d-ba0c-7733-9a31-ed2f0f32eab9

  • Rounds: 1; findings 0: fixed 0, refuted 0, open 0

kixelated and others added 4 commits November 12, 2026 19:23
Quest-Claim: 4ab0f0a294b24007
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Quest-Claim: c5b5931035e623f5
…/pullfrog-setup

# Conflicts:
#	quest/m0/pullfrog-setup.md
Rind needs no setup now that its vectors are vendored; a dispatched test
run on its main passed. Studio, rust2, and web-transport get the same
steps in their own PRs; the quest waits on those and the post-merge runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Pullfrog reviews have what their checks need ci(pullfrog): give reviews the dev shell's tools Nov 15, 2026
@kixelated
kixelated marked this pull request as ready for review November 15, 2026 18:11
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1a35d3a6-48f8-465f-a563-39cd3112e094
Commits

Reviewing files that changed from the base of the PR and between f3bf9ca and 4f90321.

Files selected for processing: 2

  • .github/workflows/pullfrog.yml
  • quest/m0/pullfrog-setup.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ One rough edge worth a look — nothing blocking.

Reviewed changes

Reviewed the Pullfrog workflow's new Nix setup steps and the rewritten pullfrog-setup quest.

  • Dev shell on the agent's PATH: installs Nix with the same pinned installer check.yml uses, then prepends the /nix/store entries of main's nix print-dev-env PATH through GITHUB_PATH, colon-joined on one line so order is preserved.
  • Job token scrubbed: deletes the installer's access-tokens line from /etc/nix/nix.conf after the dev shell is built, and fails the job if github.token remains anywhere under /etc/nix.
  • Quest narrowed: quest/m0/pullfrog-setup.md is resized to XS, records rind as done, and leaves one dispatched run per repository plus the three outside merges under Waiting.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread .github/workflows/pullfrog.yml Resolved
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 15, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the one commit since the prior Pullfrog review (cedeb9a).

  • Failing print-dev-env now fails the step: the PATH step sets shell: bash, which runs with -eo pipefail. A failed nix print-dev-env no longer writes an empty line to GITHUB_PATH. This addresses the prior thread, which is resolved.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the one commit since the prior Pullfrog review (cedeb9a).

  • Token scrub widened: the step now strips access-tokens from every file under /etc/nix (including nix.custom.conf), not just nix.conf.
  • Receipt locked down: /nix/receipt.json, whose plan also records the token, is set to 0600. The existing grep -rqF guard over /etc/nix still fails the job if the token survives there.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 920812a Nov 15, 2026
21 checks passed
@kixelated
kixelated deleted the quest/m0/pullfrog-setup branch November 15, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant