Skip to content

[release/17.0] Raise PR Documentation Check max-turns from 50 to 80 - #22005

Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-07-pr21840from
mitchdenny-backport-17-0-08-pr21846
Closed

Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-07-pr21840from
mitchdenny-backport-17-0-08-pr21846

Conversation

@mitchdenny

Copy link
Copy Markdown
Member

Description

Backport of #21846 to release/17.0 (cherry-picked from cfda6e1 with no modifications). Layer 8 of 55 in the release/17.0 backport stack; stacked on #22004.

Backport recommendation: No — Workflows run from main.

Backport of #21846

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

## Failure

PR Documentation Check runs were failing with `429 Maximum LLM
invocations exceeded (50 / 50)`
(`failureClass=invocation_cap_exceeded`). Failure triage mislabelled
them as provider rate limits. **14 cap hits** between 2026-10-02 and
2026-10-20, e.g.
[39318742372](https://github.com/microsoft/aspire/actions/runs/39318742372),
[39362924453](https://github.com/microsoft/aspire/actions/runs/39362924453),
[36976170750](https://github.com/microsoft/aspire/actions/runs/36976170750),
[37537997917](https://github.com/microsoft/aspire/actions/runs/37537997917),
[38540541504](https://github.com/microsoft/aspire/actions/runs/38540541504)
(duplicates #20679, #20768, #20826, #20863, #20941, #20994, #21023,
#21056). Capped runs were doing real work. For example, 39362924453 was
cut off after 5m32s with `+120 -7` already staged.

Fixes #21157

## Change

- `pr-docs-check.md`: `max-turns: 50` → `80`, with a comment pointing at
#21157.
- `pr-docs-check.md` prompt: "Execution budget and completion" now says
`80-invocation limit` instead of `50-invocation limit`. The
early-finalization target (invocation 35) is unchanged.
- `pr-docs-check.lock.yml` recompiled with the pinned gh-aw
**v0.89.17**. Apart from the header `body_hash`, the only changes are
the two rails `max-turns` compiles to:

```diff
-          printf '%s\n' "{… \"maxRuns\":50 …}" > "${RUNNER_TEMP}/gh-aw/awf-config.json"
+          printf '%s\n' "{… \"maxRuns\":80 …}" > "${RUNNER_TEMP}/gh-aw/awf-config.json"
-          GH_AW_MAX_TURNS: 50
+          GH_AW_MAX_TURNS: 80
```

## Confidence

- **Root cause — high.** The harness logs say `LLM invocation cap
saturated` / `Maximum LLM invocations exceeded (50 / 50)`, not a
provider 429.
- **Fix correctness — high.** 80 gives ~60% headroom. Agent p90 was 461s
in the capped window (Oct 6–20, 13 hits / 53 runs). It has been 169s
since then (0 hits / 74 runs), after `auto` model routing changed. Since
the model is `auto` and the CLI is unpinned, the pressure can return.
- **Regression risk — low.** Runaway protection is unchanged: the 25M
`maxEffectiveTokens` proxy rail, the 20-minute job timeout, the prompt's
no-retry rule for deterministic failures, and outcome validation. The
worst case is more invocations on rare long runs. Those runs already
burned ~156–167 AIC before failing and being retried by hand.
- **Validation — compile and actionlint.** `gh aw compile pr-docs-check`
(v0.89.17): `✓ … 1 succeeded`. `actionlint` on the lock file reports the
same 3 pre-existing findings as `main` (`concurrency.queue`,
`copilot-requests`), nothing new. `auto-sec/publication-guard.js`
passes. This isn't verified in a live run yet. Urgency is low because
there have been no hits since Oct 20.
- **CI:** the repo's `Compile, lint, and test agentic workflows`, `Lint
handwritten workflows` and `zizmor` checks pass on this PR.

**How we stop this recurring:** keep the turn cap as a backstop with
headroom above the observed p90, and have failure triage classify
`invocation_cap_exceeded` as a turn-cap problem instead of an engine
429.

<details><summary>Source diff (<code>pr-docs-check.md</code>)</summary>

```diff
diff --git a/.github/workflows/pr-docs-check.md b/.github/workflows/pr-docs-check.md
index f03f126..154552f 100644
--- a/.github/workflows/pr-docs-check.md
+++ b/.github/workflows/pr-docs-check.md
@@ -45,7 +45,9 @@ if: >-
 # The prompt reserves the final calls for completion, queues the notification
 # before terminal PR creation, and prohibits retrying deterministic failures.
 # Keep the cap as a backstop; incomplete output still fails outcome validation.
-max-turns: 50
+# Raised from 50 to 80 after large docs PRs hit "Maximum LLM invocations
+# exceeded (50 / 50)" (#21157).
+max-turns: 80

 checkout:
   # Check out aspire.dev EXACTLY ONCE, as the current workspace, because that is
```

</details>

**🤖 Posted by AspireAgent (AI Agent)**

---------

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit cfda6e1)
@mitchdenny

Copy link
Copy Markdown
Member Author

Approved for inclusion in release/17.0. Do not merge yet.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22005

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22005"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants