From 47ebaaafc77488aaa3ad3c0ff80b642f21330492 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 01:51:36 -0800 Subject: [PATCH 1/4] chore(quest): claim quest/m0/ci/pullfrog-timeout.md Quest-Claim: 4f9e77bbc594cde6 From cbb49cd91d70cf7e76f16e7bd3a4a0414b7f4d68 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 01:54:21 -0800 Subject: [PATCH 2/4] ci: request medium-effort Pullfrog reviews within one hour Co-Authored-By: GPT-6.1 Sol --- .github/workflows/pullfrog.yml | 2 + quest/m0/ci/pullfrog-timeout.md | 105 +++++++++++++++++++------------- 2 files changed, 63 insertions(+), 44 deletions(-) diff --git a/.github/workflows/pullfrog.yml b/.github/workflows/pullfrog.yml index f698e046c9b..c28dbcc5b78 100644 --- a/.github/workflows/pullfrog.yml +++ b/.github/workflows/pullfrog.yml @@ -33,6 +33,8 @@ jobs: uses: pullfrog/pullfrog@895164fc90f5c493e72693c919c6cf269af158eb # v0.1.110 with: prompt: ${{ inputs.prompt }} + effort: medium + timeout: 1h env: # add at least one provider API key CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} diff --git a/quest/m0/ci/pullfrog-timeout.md b/quest/m0/ci/pullfrog-timeout.md index 19a1d0537c7..b72536ffe6b 100644 --- a/quest/m0/ci/pullfrog-timeout.md +++ b/quest/m0/ci/pullfrog-timeout.md @@ -2,52 +2,69 @@ ## Goal -A Pullfrog review of a large PR finishes instead of failing on the -action's run limit: `.github/workflows/pullfrog.yml` sets a `timeout` -and `effort` for the `pullfrog/pullfrog` step that fit the reviews it is -dispatched for, and the choice is recorded in the PR with the run -durations that support it. +A Pullfrog review of a large PR finishes inside the action's run limit, +with its effective effort and duration recorded as evidence. ## Plan -Decided 2026-10-31 (user, /quest-plan of the follow-ups from the 2026-10-31 /quest-complete run). Milestone: ✅ m0 (user: "use recommended for any CI/flakes"). The fix is to raise the timeout or lower the effort. - -Facts from the last 100 runs of the workflow (58 runs, all -`workflow_dispatch` reviews; 55 passed): - -- [41178266611](https://github.com/moq-dev/moq/actions/runs/41178266611) - ("Review #7202"): `agent run timed out after 1h`. The log shows - `effort: high` and `timeout: 1h (default)`. -- [41188250827](https://github.com/moq-dev/moq/actions/runs/41188250827) - ("Review #7234", 56 min): aborted by the action's activity watchdog after - the model sent nothing for 903 s. Not the run limit, but a lower effort - likely shortens those silent turns too. -- [41197083466](https://github.com/moq-dev/moq/actions/runs/41197083466) - ("Review #7211", 23 min): failed installing dependencies - (`@moq/libav@0.1.0 is not published to npm yet, and building it needs - Nix`). Unrelated to the limit; note it in the PR as a follow-up rather - than fixing it here. - -It still recurs after this plan: runs -[41362378749](https://github.com/moq-dev/moq/actions/runs/41362378749) -("Review #7586") and -[41404718134](https://github.com/moq-dev/moq/actions/runs/41404718134) -("Review #7599") both failed with `agent run timed out after 1h` at -`effort: high`. - -The action takes `timeout` ("e.g. 10m, 1h30m. Default: 1h") and `effort` -(`low` to `max`, or a position in [0, 1]) inputs; today neither is set in -`with:` (`pullfrog.yml:34-35`, at be39188000), so the `high` comes from -the repo's Pullfrog settings. The file's header allows -edits only where indicated; `with:` is the place. Recommended: set -`effort: medium` first, since it bounds cost as well as time, and raise -`timeout` only if medium-effort reviews of the largest recent -PRs still approach an hour. Keep the job's own `timeout-minutes` (default -360) above whatever `timeout` is set. - -[moq-dev/moq#7317](https://github.com/moq-dev/moq/pull/7317) (merged, pins -the action) and [moq-dev/moq#7424](https://github.com/moq-dev/moq/pull/7424) -(merged, gates its secrets to main through the `pullfrog` environment) -already reshaped this file; edit on top of them. +Decided 2026-10-31 (user, /quest-plan): use the recommended effort-first +budget. The workflow now requests `effort: medium` and explicitly retains +`timeout: 1h` in the action's allowed `with:` block. Do not raise the limit +without medium-effort evidence from the largest recent PRs. GitHub's +unchanged default job limit is 360 minutes. + +Refreshed 2026-11-03 against pinned action +[`895164fc90f5c493e72693c919c6cf269af158eb`](https://github.com/pullfrog/pullfrog/blob/895164fc90f5c493e72693c919c6cf269af158eb/action.yml). +Its `effort` input overrides repository settings, but JSON prompt fields +have higher precedence in +[`utils/payload.ts`](https://github.com/pullfrog/pullfrog/blob/895164fc90f5c493e72693c919c6cf269af158eb/utils/payload.ts#L337). +More importantly, +[`main.ts`](https://github.com/pullfrog/pullfrog/blob/895164fc90f5c493e72693c919c6cf269af158eb/main.ts#L496) +unconditionally replaces effort for OSS proxy runs with the funded model's +`high` rung. The current +[run 41614007828](https://github.com/moq-dev/moq/actions/runs/41614007828) +uses `proxy: oss` and logs `effort: high`. Setting the input alone therefore +does not establish an effective medium-effort review on this route. + +Existing run evidence, measured from the jobs API's `Run agent` step: + +- [41178266611](https://github.com/moq-dev/moq/actions/runs/41178266611): + 60m29s, 16 files, high effort, `agent run timed out after 1h`. +- [41362378749](https://github.com/moq-dev/moq/actions/runs/41362378749): + 60m29s, 14 files, high effort, the same run-limit failure. +- [41404718134](https://github.com/moq-dev/moq/actions/runs/41404718134): + 60m40s, 3 files, high effort, the same run-limit failure. Size alone + does not explain the overruns. +- [41188250827](https://github.com/moq-dev/moq/actions/runs/41188250827): + 56m00s, activity-watchdog failure after 903 seconds without events, + distinct from the run limit. +- [41590494064](https://github.com/moq-dev/moq/actions/runs/41590494064): + 25m54s, 12 files, high effort, success. This is not medium-effort proof. + +The old +[41197083466](https://github.com/moq-dev/moq/actions/runs/41197083466) +logged the dependency-install error but ultimately failed the activity +watchdog at 23m07s. The install error still appears in successful +[41591680387](https://github.com/moq-dev/moq/actions/runs/41591680387) and +[41614007828](https://github.com/moq-dev/moq/actions/runs/41614007828): +`@moq/libav@0.1.0` is unpublished and Nix is absent. The npm version endpoint +still returns 404 on 2026-11-03. Publication is already covered by +[Claim the new npm names](/quest/m2/release/npm-first-publish.md); report the +Pullfrog setup gap to that work's planner rather than duplicating it here. + +Do not dispatch or request online reviews to validate this attempt. Use +existing run evidence; leave unavailable runtime proof explicit. Public API: none. Wire: none. + +## Waiting + +- Pullfrog: effective medium effort is unavailable on the current OSS proxy + route because the pinned action forces high effort. Recommended: resolve + that upstream constraint before considering a larger timeout; changing + provider or funding policy needs a maintainer decision. Confirm in the + action source and effective-effort log, without dispatching a validation run. +- moq-dev/moq: an existing review of a large recent PR must demonstrate the + effective effort requested by this workflow and finish inside one hour. + Record its URL and `Run agent` duration before closing this quest. No such + medium-effort runtime evidence was available for this attempt. From 370158d6b7e42fa8549b58de2caf885323315159 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 01:57:40 -0800 Subject: [PATCH 3/4] docs(quest): assign the effective-effort decision to the maintainer Co-Authored-By: GPT-6.1 Sol --- quest/m0/ci/pullfrog-timeout.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/quest/m0/ci/pullfrog-timeout.md b/quest/m0/ci/pullfrog-timeout.md index b72536ffe6b..93bf6582f2f 100644 --- a/quest/m0/ci/pullfrog-timeout.md +++ b/quest/m0/ci/pullfrog-timeout.md @@ -8,7 +8,12 @@ with its effective effort and duration recorded as evidence. ## Plan Decided 2026-10-31 (user, /quest-plan): use the recommended effort-first -budget. The workflow now requests `effort: medium` and explicitly retains +budget. Reconfirmed 2026-11-03 (user, /quest-spawn implementation brief): +apply that configuration, publish the attempt, keep unavailable medium-effort +runtime proof as explicit Waiting, and do not raise timeouts without the +required evidence or dispatch online validation. + +The workflow now requests `effort: medium` and explicitly retains `timeout: 1h` in the action's allowed `with:` block. Do not raise the limit without medium-effort evidence from the largest recent PRs. GitHub's unchanged default job limit is 360 minutes. @@ -59,11 +64,13 @@ Public API: none. Wire: none. ## Waiting -- Pullfrog: effective medium effort is unavailable on the current OSS proxy - route because the pinned action forces high effort. Recommended: resolve - that upstream constraint before considering a larger timeout; changing - provider or funding policy needs a maintainer decision. Confirm in the - action source and effective-effort log, without dispatching a validation run. +- Maintainer: choose how to make the effort input effective on the current + OSS proxy route, where the pinned action forces high effort. Recommended: + request upstream support for the input; alternative: choose a different + provider or funding route. No upstream request has been filed by this + attempt. Keep this PR draft pending that decision, then confirm the chosen + path in action source and effective-effort logs without dispatching a + validation run. Do not increase the timeout without the required evidence. - moq-dev/moq: an existing review of a large recent PR must demonstrate the effective effort requested by this workflow and finish inside one hour. Record its URL and `Run agent` duration before closing this quest. No such From 72665d431419ee243f42617ab1509c224cb855f7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 15:33:36 -0800 Subject: [PATCH 4/4] ci: finish the Pullfrog timeout quest on DeepSeek V4.1 Flash Pullfrog's OSS proxy already runs DeepSeek V4.1 Flash (the funded default, chosen server-side) and forces high effort, so the workflow's effort input was a no-op. Drop it, and delete the quest: a 135-file review now posts in 11 minutes, inside the one-hour run limit. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/pullfrog.yml | 2 - quest/m0/ci/README.md | 1 - quest/m0/ci/pullfrog-timeout.md | 77 --------------------------------- 3 files changed, 80 deletions(-) delete mode 100644 quest/m0/ci/pullfrog-timeout.md diff --git a/.github/workflows/pullfrog.yml b/.github/workflows/pullfrog.yml index c28dbcc5b78..f698e046c9b 100644 --- a/.github/workflows/pullfrog.yml +++ b/.github/workflows/pullfrog.yml @@ -33,8 +33,6 @@ jobs: uses: pullfrog/pullfrog@895164fc90f5c493e72693c919c6cf269af158eb # v0.1.110 with: prompt: ${{ inputs.prompt }} - effort: medium - timeout: 1h env: # add at least one provider API key CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 8c47322eee4..fdd08593905 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -165,5 +165,4 @@ The historical samples locate candidates; they are not controlled benchmarks. - [The queue host keeps the queue's disk](/quest/m0/ci/queue-host-disk.md) - agent sessions can't starve the queue runner of disk, and a crashed runner restarts - [The bench comparison builds moq-gst's catalog bench on its base](/quest/m0/ci/bench-base-build.md) - A paired bench comparison that selects `rust/gst/moq-gst/catalog` against `origin/main` builds the base side and reports a comparison, instead of marking it unavailable with "base build failed" - [The workspace-hack refuses trybuild edges and spares non-media builds](/quest/m0/ci/hakari-followups.md) - `hakari.py` refuses trybuild dev-dependency edges, and non-media Linux test builds stop needing libpipewire and libclang -- [Pullfrog reviews finish inside their time limit](/quest/m0/ci/pullfrog-timeout.md) - A Pullfrog review of a large PR finishes instead of failing on the action's run limit - [The flake lane skips hosts that can't map user namespaces](/quest/m0/ci/userns-uid-map.md) - `check flake` skips instead of failing where `/proc/self/uid_map` is unwritable diff --git a/quest/m0/ci/pullfrog-timeout.md b/quest/m0/ci/pullfrog-timeout.md deleted file mode 100644 index 93bf6582f2f..00000000000 --- a/quest/m0/ci/pullfrog-timeout.md +++ /dev/null @@ -1,77 +0,0 @@ -# [XS] Pullfrog reviews finish inside their time limit - -## Goal - -A Pullfrog review of a large PR finishes inside the action's run limit, -with its effective effort and duration recorded as evidence. - -## Plan - -Decided 2026-10-31 (user, /quest-plan): use the recommended effort-first -budget. Reconfirmed 2026-11-03 (user, /quest-spawn implementation brief): -apply that configuration, publish the attempt, keep unavailable medium-effort -runtime proof as explicit Waiting, and do not raise timeouts without the -required evidence or dispatch online validation. - -The workflow now requests `effort: medium` and explicitly retains -`timeout: 1h` in the action's allowed `with:` block. Do not raise the limit -without medium-effort evidence from the largest recent PRs. GitHub's -unchanged default job limit is 360 minutes. - -Refreshed 2026-11-03 against pinned action -[`895164fc90f5c493e72693c919c6cf269af158eb`](https://github.com/pullfrog/pullfrog/blob/895164fc90f5c493e72693c919c6cf269af158eb/action.yml). -Its `effort` input overrides repository settings, but JSON prompt fields -have higher precedence in -[`utils/payload.ts`](https://github.com/pullfrog/pullfrog/blob/895164fc90f5c493e72693c919c6cf269af158eb/utils/payload.ts#L337). -More importantly, -[`main.ts`](https://github.com/pullfrog/pullfrog/blob/895164fc90f5c493e72693c919c6cf269af158eb/main.ts#L496) -unconditionally replaces effort for OSS proxy runs with the funded model's -`high` rung. The current -[run 41614007828](https://github.com/moq-dev/moq/actions/runs/41614007828) -uses `proxy: oss` and logs `effort: high`. Setting the input alone therefore -does not establish an effective medium-effort review on this route. - -Existing run evidence, measured from the jobs API's `Run agent` step: - -- [41178266611](https://github.com/moq-dev/moq/actions/runs/41178266611): - 60m29s, 16 files, high effort, `agent run timed out after 1h`. -- [41362378749](https://github.com/moq-dev/moq/actions/runs/41362378749): - 60m29s, 14 files, high effort, the same run-limit failure. -- [41404718134](https://github.com/moq-dev/moq/actions/runs/41404718134): - 60m40s, 3 files, high effort, the same run-limit failure. Size alone - does not explain the overruns. -- [41188250827](https://github.com/moq-dev/moq/actions/runs/41188250827): - 56m00s, activity-watchdog failure after 903 seconds without events, - distinct from the run limit. -- [41590494064](https://github.com/moq-dev/moq/actions/runs/41590494064): - 25m54s, 12 files, high effort, success. This is not medium-effort proof. - -The old -[41197083466](https://github.com/moq-dev/moq/actions/runs/41197083466) -logged the dependency-install error but ultimately failed the activity -watchdog at 23m07s. The install error still appears in successful -[41591680387](https://github.com/moq-dev/moq/actions/runs/41591680387) and -[41614007828](https://github.com/moq-dev/moq/actions/runs/41614007828): -`@moq/libav@0.1.0` is unpublished and Nix is absent. The npm version endpoint -still returns 404 on 2026-11-03. Publication is already covered by -[Claim the new npm names](/quest/m2/release/npm-first-publish.md); report the -Pullfrog setup gap to that work's planner rather than duplicating it here. - -Do not dispatch or request online reviews to validate this attempt. Use -existing run evidence; leave unavailable runtime proof explicit. - -Public API: none. Wire: none. - -## Waiting - -- Maintainer: choose how to make the effort input effective on the current - OSS proxy route, where the pinned action forces high effort. Recommended: - request upstream support for the input; alternative: choose a different - provider or funding route. No upstream request has been filed by this - attempt. Keep this PR draft pending that decision, then confirm the chosen - path in action source and effective-effort logs without dispatching a - validation run. Do not increase the timeout without the required evidence. -- moq-dev/moq: an existing review of a large recent PR must demonstrate the - effective effort requested by this workflow and finish inside one hour. - Record its URL and `Run agent` duration before closing this quest. No such - medium-effort runtime evidence was available for this attempt.