[Fleet Monitor] combsmith/.github-private — .github/workflows/daily-pr-review-health.yml #2446

Activity

1. don-petry

   added

   health-checkAutomated health check reportAutomated health check report

   dev-leadFor dev-lead agent pickupFor dev-lead agent pickup

   fleet-trackerTracked by the Actions Fleet MonitorTracked by the Actions Fleet Monitor

   on Nov 3, 2026
2. don-petry commented on Nov 3, 2026

   

   don-petry

   on Nov 3, 2026

   CollaboratorAuthor

   More actions

   Dev-Lead: issue #2446 implementation failed — will retry

   The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

   • Cause: engine-error
   • Run: https://github.com/combsmith/.github-private/actions/runs/41656914428

   Last 40 lines of engine output (redacted)

       at throwErrorIfNotOK (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:267922:24)
       at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
       at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:267685:7
       at async Models.generateContentStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:268781:16)
       at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:312891:19
       at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:289124:23
       at async retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:309277:23)
       at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:334390:28)
       at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:334085:29)
       at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:335129:24) {
     status: 503
   }
   Attempt 10 failed: This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.. Max attempts reached
   Error when talking to Gemini API Full report available at: /tmp/gemini-client-error-Turn.run-sendMessageStream-2026-11-03T14-57-13-062Z.json RetryableQuotaError: This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.
       at classifyGoogleError (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:308569:14)
       at retryWithBackoff (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:309300:29)
       at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
       at async GeminiChat.makeApiCallAndProcessStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:334390:28)
       at async GeminiChat.streamWithRetries (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:334085:29)
       at async Turn.run (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:335129:24)
       at async GeminiClient.processTurn (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:349109:22)
       at async GeminiClient.sendMessageStream (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/chunk-RZ3ZVTZB.js:349206:14)
       at async file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-22EOGL6O.js:12219:26
       at async main (file:///home/runner/.npm-global/lib/node_modules/@google/gemini-cli/bundle/gemini-22EOGL6O.js:17833:5) {
     cause: {
       code: 503,
       message: 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.',
       details: []
     },
     retryDelayMs: undefined,
     status: 503
   }
   {
     "session_id": "60726d00-e957-41d4-8c9f-5dfec8f9a2da",
     "error": {
       "type": "Error",
       "message": "This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.",
       "code": 503
     }
   }
   
   
3. don-petry commented on Nov 3, 2026

   

   don-petry

   on Nov 3, 2026

   CollaboratorAuthor

   More actions

   Dev-Lead Implementation Plan

   Issue: #2446 — [Fleet Monitor] combsmith/.github-private — .github/workflows/daily-pr-review-health.yml

   Root cause

   The failing run (41649787742, 2026-11-03) failed in Run health check: the Claude log-analysis call in scripts/pr_review_health.sh returned You've hit your weekly limit · resets 4pm (UTC), and the script exits 1 on any Claude failure. A failed step also skips everything after it: the issue for the report, plus the runaway, stall and merge-ready scans. Those scans make no model call. The AGENTS_PAUSED gate (#2249) only helps once the budget breaker has paused the fleet. This run happened before the pause took effect.

   (The earlier failures on 10-28, 10-29 and 10-30 were the context7 MCP NO_HANDSHAKE check in Assert MCP connectivity. They are outside the current window and not part of this fix.)

   Scope

   Make pr_review_health.sh degrade gracefully when the Claude call is refused for quota or a usage limit. In that case it emits a ::warning::, adds a "log analysis skipped" note to the report, and exits 0, so the report sections that don't use the model and the later scan steps still run. Any other Claude failure still fails the step.

   Implementation Checklist

   • scripts/pr_review_health.sh: capture the Claude stdout/stderr in temp files instead of appending straight to the report. If the call fails and the captured output matches a quota/usage-limit pattern, warn, add a note and continue. Any other failure still exits 1.
   • .github/workflows/lint.yml: register the new bats file in the bats job list (AGENTS.md exception).

   Tests to Write

   • tests/pr_review_health.bats (stubbed gh / claude on PATH):
     • quota refusal ("You've hit your weekly limit") → exit 0, ::warning::, report keeps the parts that don't use the model and adds the skipped note
     • non-quota Claude failure → exit 1 (unchanged)
     • success → model output appended to the report

   Test Command

   scripts/run-bats.sh tests/pr_review_health.bats + bash .dev-lead/scripts/dev-lead-lint.sh
4. don-petry commented on Nov 3, 2026

   

   don-petry

   on Nov 3, 2026

   CollaboratorAuthor

   More actions

   Dev-Lead: Implementation Complete — PR #2448

   The implementation for issue #2446 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

   • Pull request: fix: treat Claude quota refusals as non-fatal in PR review health report #2448
   • Head commit: 0548cd88779f19a5981a7c638f3133deaa21a0ee
   • Net change against main: 3 file(s), +133/-7 lines
   • Run: https://github.com/combsmith/.github-private/actions/runs/41675215286

   Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
5. don-petry

   added a commit that references this issue on Nov 4, 2026

   fix(bot): remove out-of-scope feature-ideation.yml changes — defer au…

   ...

   a8a0ba9
6. don-petry commented on Nov 6, 2026

   

   don-petry

   on Nov 6, 2026

   CollaboratorAuthor

   More actions

   ✅ Auto-resolved by Fleet Monitor. This workflow's failure rate has been below the 10% threshold (or the workflow was removed) since 2026-11-03 (3 days) — the monitor stopped flagging it. Closing. It will reopen automatically if the workflow crosses the threshold again.
7. don-petry

   closed this as completedon Nov 6, 2026