From cbe00d4a5bb6a6f73c91e68adb70042d4bc1fd2f Mon Sep 17 00:00:00 2001 From: don-petry <36422719+don-petry@users.noreply.github.com> Date: Sun, 11 Oct 2026 20:34:09 -0500 Subject: [PATCH 1/5] docs(fleet-pause): a pause set by hand is three ordered calls, and a read-after-write check replaces the drill The hand-pause drill asked for before arming needs a test organisation, and there is none. The two races it covers are closed by procedure instead: delete the pause source, set the pause, delete the source again. The first delete removes a leftover source that would make the pause read as the breaker's own; the last covers a trip that had already passed its final check. A sixth live check, that a read of the variable list sees the write before it, replaces the drill in the arming preconditions. Also warns against gh variable set --org for this: its default visibility is private, which reaches none of this organisation's public repositories. Refs #2251 Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 60 +++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8433667c1..6abcb7b85 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -846,6 +846,50 @@ only an **organisation** variable pauses every repository, and a repository vari same name overrides the organisation's for that repository. GitHub compares the value without regard to case, so `TRUE` pauses too. +#### Setting a pause by hand: three calls, in this order (#2251) + +A pause set by hand is one that carries no `AGENTS_PAUSE_SOURCE`. The budget breaker lifts only +a pause that carries its own source, so a hand pause is always **delete the source, set the +pause, delete the source again**, in that order, whether or not the breaker is armed: + +```bash +gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE +``` + +```bash +gh api -X PATCH orgs/petry-projects/actions/variables/AGENTS_PAUSED -f value=true +``` + +```bash +gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE +``` + +- A `DELETE` answers 404 when there is no source to delete. That is the normal case and it is + fine; run the next call anyway. +- The `PATCH` answers 404 when the variable does not exist yet. Create it instead, visible to + every repository: + `gh api -X POST orgs/petry-projects/actions/variables -f name=AGENTS_PAUSED -f value=true -f visibility=all`. +- **Do not use `gh variable set --org` for this without `--visibility all`.** Its default + visibility is `private`, and it sends that on an update as well as on a create. Every + repository in this organisation is public, so a `private` variable reaches none of them and + the fleet keeps running while the variable reads `true`. A value-only `PATCH` leaves the + visibility alone. +- Why three calls. Ownership of a pause is the source and nothing else, and the breaker can + run between any two of these calls. + - The first delete removes a source left over from an earlier failed trip. Without it, a + breaker run that lands just after the pause is set would read the pause as its own and, + if the leftover end time has passed, lift it. + - The last delete covers a trip in progress. A trip writes its source before its final + check; if the pause is set in the one call between that check and the trip's own write + of the pause, the trip's source is still there, and this delete removes it. If the pause + is set any earlier, the trip's check sees it and the trip takes back what it wrote. + - After the last delete the pause has no source, and nothing the breaker does afterwards + adds one to a fleet that is already paused. +- This is why no drill of those races is required before arming. What it rests on is that a + read of the variable list sees a write made just before it, which is one of the live checks + under "Arming preconditions". +- To lift a hand pause, set `AGENTS_PAUSED` to `false` the same way (or delete the variable). + #### Which workflows honour the pause (#2249) Until #2249 only dev-lead read the variable, so a pause stopped one agent while pr-review, the @@ -1154,9 +1198,11 @@ as soon as the workflow is on `main`. first), sets nothing else, and raises a `::warning::` in a green run. This roll-back is the only write a run makes outside its plan, and it removes only what the same run wrote and that still holds the value it wrote. Two limits to know: - - The window is narrowed, not closed: a pause set by hand in the one API call between that - second read and the write of `AGENTS_PAUSED` is still claimed. To be sure a pause set by - hand stays, check afterwards that `AGENTS_PAUSE_SOURCE` is absent, and delete it if not. + - The window is narrowed, not closed, by the breaker alone: a pause set by hand in the one + API call between that second read and the write of `AGENTS_PAUSED` would still be + claimed. The hand-pause procedure closes it from the other side: delete + `AGENTS_PAUSE_SOURCE` before and after setting the pause, every time ("Setting a pause by + hand: three calls, in this order"). - If the roll-back itself fails, the run is red with one `::error::` that says what is left. A failed delete of the source leaves `AGENTS_PAUSE_SOURCE=rate-limit-observed` **and** `AGENTS_PAUSE_UNTIL` in place on purpose (a pause carrying the breaker's source @@ -1191,7 +1237,7 @@ as soon as the workflow is on `main`. 1. "Budget breaker: close the gaps found in review before it is armed" (#2251) is closed. 2. "AGENTS_PAUSED stops only dev-lead" (#2249) has landed: until then a pause set by the breaker stops dev-lead's dispatch, relay and resume jobs and nothing else. - 3. The five live checks below are recorded on "Auto-pause companion" (#1565), each with + 3. The six live checks below are recorded on "Auto-pause companion" (#1565), each with the date and what was observed. They are mocked in the tests and must be confirmed against the real API by the maintainer, with a **throwaway organisation variable** (for example `BUDGET_BREAKER_PROBE`), **never** with `AGENTS_PAUSED`, @@ -1201,6 +1247,12 @@ as soon as the workflow is on `main`. - the variable list paginates as the code expects: `gh api --paginate` on `orgs/petry-projects/actions/variables?per_page=30` with a `--jq` filter returns every variable when there are more than 30 (the filter runs once per page); + - a read of the variable list sees a write made just before it. The pause guard and the + hand-pause procedure both rest on this, and it replaces a drill of the two hand-pause + races (there is no test organisation to run one in). Twenty times: `PATCH` the + throwaway variable to a new value, then read + `orgs/petry-projects/actions/variables?per_page=30` and compare; one stale read fails + the check, and a drill is then needed after all; - the dedicated write credential can read, create, update and delete an organisation variable, and nothing else; - with the read credential the workflow uses, dev-lead's quota markers read as trusted: From 1ac066360c1e8ac62b7c45f513675bc5063ca287 Mon Sep 17 00:00:00 2001 From: don-petry <36422719+don-petry@users.noreply.github.com> Date: Tue, 13 Oct 2026 21:46:34 -0500 Subject: [PATCH 2/5] docs(fleet-pause): the hand pause forces visibility, ends with a check, and the live check covers deletes From review. The three calls do not cover a resume the breaker has already decided on: it can set the pause back to false after them, with every call looking successful. Nothing from outside can stop that write, so the procedure now ends by reading the state a minute later and repeating once if it is not as expected. The pause call sets visibility to all every time, because a value-only update keeps a private visibility that reaches no public repository. The live check that replaces the drill now creates, updates and deletes, since the guarantee depends on a deleted source being gone from the next read, and it is stated as blocking arming. Refs #2251 Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 65 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4cff8df67..15a8a3dc5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -846,34 +846,45 @@ only an **organisation** variable pauses every repository, and a repository vari same name overrides the organisation's for that repository. GitHub compares the value without regard to case, so `TRUE` pauses too. -#### Setting a pause by hand: three calls, in this order (#2251) +#### Setting a pause by hand: three calls, then a check (#2251) A pause set by hand is one that carries no `AGENTS_PAUSE_SOURCE`. The budget breaker lifts only a pause that carries its own source, so a hand pause is always **delete the source, set the -pause, delete the source again**, in that order, whether or not the breaker is armed: +pause, delete the source again, then check**, in that order, whether or not the breaker is +armed: ```bash gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE ``` ```bash -gh api -X PATCH orgs/petry-projects/actions/variables/AGENTS_PAUSED -f value=true +gh api -X PATCH orgs/petry-projects/actions/variables/AGENTS_PAUSED -f value=true -f visibility=all ``` ```bash gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE ``` +Wait a minute, then read the state: + +```bash +gh api "orgs/petry-projects/actions/variables?per_page=100" --jq '[.variables[] | select(.name == "AGENTS_PAUSED" or .name == "AGENTS_PAUSE_SOURCE") | "\(.name)=\(.value) (\(.visibility))"] | join(" ")' +``` + +The answer must be exactly `AGENTS_PAUSED=true (all)`, with no `AGENTS_PAUSE_SOURCE`. **If it +is anything else, run the three calls again and check again.** The check is part of the +procedure, not an extra. + - A `DELETE` answers 404 when there is no source to delete. That is the normal case and it is fine; run the next call anyway. -- The `PATCH` answers 404 when the variable does not exist yet. Create it instead, visible to - every repository: +- The `PATCH` answers 404 when the variable does not exist yet. Create it instead: `gh api -X POST orgs/petry-projects/actions/variables -f name=AGENTS_PAUSED -f value=true -f visibility=all`. -- **Do not use `gh variable set --org` for this without `--visibility all`.** Its default - visibility is `private`, and it sends that on an update as well as on a create. Every - repository in this organisation is public, so a `private` variable reaches none of them and - the fleet keeps running while the variable reads `true`. A value-only `PATCH` leaves the - visibility alone. +- **Visibility must be `all`, and the `PATCH` sets it every time.** A variable with visibility + `private` reaches only private repositories, and every repository in this organisation is + public (14 of 14 on 2026-10-14), so it would pause nothing while reading `true`. A + value-only `PATCH` would keep whatever visibility the variable already had. **Do not use + `gh variable set --org` without `--visibility all`:** its default is `private`, and it sends + that on an update as well as on a create. - Why three calls. Ownership of a pause is the source and nothing else, and the breaker can run between any two of these calls. - The first delete removes a source left over from an earlier failed trip. Without it, a @@ -883,11 +894,18 @@ gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE check; if the pause is set in the one call between that check and the trip's own write of the pause, the trip's source is still there, and this delete removes it. If the pause is set any earlier, the trip's check sees it and the trip takes back what it wrote. - - After the last delete the pause has no source, and nothing the breaker does afterwards - adds one to a fleet that is already paused. + - After the last delete the pause has no source, and the breaker never trips over or + resumes a pause with no source. +- Why the check. The three calls do not cover a **resume** in progress. A breaker run that + has made its last read of the state ("my pause, and its time is up") goes on to set + `AGENTS_PAUSED=false`; if the first two calls land between that read and that write, the + fleet ends up running and every call still looks successful. Nothing done from outside can stop a write the breaker has + already decided on. What can be done is to look afterwards: a breaker run that has read the + state finishes within seconds, and once it has resumed, its source is gone and it will not + touch a hand pause. So one repeat is enough, and the minute is there to let that run finish. - This is why no drill of those races is required before arming. What it rests on is that a - read of the variable list sees a write made just before it, which is one of the live checks - under "Arming preconditions". + read of the variable list sees a write or a delete made just before it, which is one of the + live checks under "Arming preconditions" and blocks arming until it has passed. - To lift a hand pause, set `AGENTS_PAUSED` to `false` the same way (or delete the variable). #### Which workflows honour the pause (#2249) @@ -1201,8 +1219,9 @@ as soon as the workflow is on `main`. - The window is narrowed, not closed, by the breaker alone: a pause set by hand in the one API call between that second read and the write of `AGENTS_PAUSED` would still be claimed. The hand-pause procedure closes it from the other side: delete - `AGENTS_PAUSE_SOURCE` before and after setting the pause, every time ("Setting a pause by - hand: three calls, in this order"). + `AGENTS_PAUSE_SOURCE` before and after setting the pause, then check the state a + minute later and repeat if it is not as expected ("Setting a pause by hand: three calls, + then a check"). - If the roll-back itself fails, the run is red with one `::error::` that says what is left. A failed delete of the source leaves `AGENTS_PAUSE_SOURCE=rate-limit-observed` **and** `AGENTS_PAUSE_UNTIL` in place on purpose (a pause carrying the breaker's source @@ -1247,12 +1266,14 @@ as soon as the workflow is on `main`. - the variable list paginates as the code expects: `gh api --paginate` on `orgs/petry-projects/actions/variables?per_page=30` with a `--jq` filter returns every variable when there are more than 30 (the filter runs once per page); - - a read of the variable list sees a write made just before it. The pause guard and the - hand-pause procedure both rest on this, and it replaces a drill of the two hand-pause - races (there is no test organisation to run one in). Twenty times: `PATCH` the - throwaway variable to a new value, then read - `orgs/petry-projects/actions/variables?per_page=30` and compare; one stale read fails - the check, and a drill is then needed after all; + - a read of the variable list sees a write, and a delete, made just before it. The pause + guard and the hand-pause procedure both rest on this, and it replaces a drill of the + hand-pause races (there is no test organisation to run one in). Twenty times, with the + throwaway variable: create it, read `orgs/petry-projects/actions/variables?per_page=30` + and find it; `PATCH` a new value, read and find that value; delete it, read and find it + gone. Once the dedicated write credential exists, make the reads with it, since it is + the breaker's reader. **Arming is blocked until this has passed. One stale read fails + it, and arming then stays blocked until a drill has been run after all;** - the dedicated write credential can read, create, update and delete an organisation variable, and nothing else; - with the read credential the workflow uses, dev-lead's quota markers read as trusted: From b0e97beecd5b9c00ce7759543d93bb1eddfbe43f Mon Sep 17 00:00:00 2001 From: don-petry <36422719+don-petry@users.noreply.github.com> Date: Tue, 13 Oct 2026 22:02:24 -0500 Subject: [PATCH 3/5] docs(fleet-pause): wait for breaker runs under way before the check, repeat until it passes, and point the breaker's notices at the procedure An independent exhaustive interleaving of the procedure against the breaker's read and write sequence found it safe with one repeat, but only if the check comes after any run that had already read the state has finished: a check made while such a run is still going can pass just before it lifts the pause. A minute is not a bound (no call has a timeout below the 10-minute job limit), so the procedure now waits on the runs that were in progress, and repeats until the check passes. The breaker's trip notice and tracking-issue text still gave the two-call advice with no check; both now point at the procedure. Refs #2251 Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 42 ++++++++++++++++++++-------- scripts/budget_breaker.sh | 2 +- scripts/lib/budget-breaker-writer.sh | 2 +- 3 files changed, 32 insertions(+), 14 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 15a8a3dc5..04b7a88f2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -846,12 +846,12 @@ only an **organisation** variable pauses every repository, and a repository vari same name overrides the organisation's for that repository. GitHub compares the value without regard to case, so `TRUE` pauses too. -#### Setting a pause by hand: three calls, then a check (#2251) +#### Setting a pause by hand: three calls, a wait, then a check (#2251) A pause set by hand is one that carries no `AGENTS_PAUSE_SOURCE`. The budget breaker lifts only a pause that carries its own source, so a hand pause is always **delete the source, set the -pause, delete the source again, then check**, in that order, whether or not the breaker is -armed: +pause, delete the source again, wait for any breaker run already under way, then check**, in +that order, whether or not the breaker is armed: ```bash gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE @@ -865,15 +865,23 @@ gh api -X PATCH orgs/petry-projects/actions/variables/AGENTS_PAUSED -f value=tru gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE ``` -Wait a minute, then read the state: +Then wait until every Budget Breaker run that was already under way has finished. This +command waits for each run in progress now and returns when they are done (a run that starts +later reads your pause and leaves it alone): + +```bash +for id in $(gh run list -R petry-projects/.github-private --workflow budget-breaker.yml --status in_progress --json databaseId --jq '.[].databaseId'); do gh run watch "$id" -R petry-projects/.github-private >/dev/null 2>&1; done; echo "no earlier breaker run is still in progress" +``` + +Then read the state: ```bash gh api "orgs/petry-projects/actions/variables?per_page=100" --jq '[.variables[] | select(.name == "AGENTS_PAUSED" or .name == "AGENTS_PAUSE_SOURCE") | "\(.name)=\(.value) (\(.visibility))"] | join(" ")' ``` The answer must be exactly `AGENTS_PAUSED=true (all)`, with no `AGENTS_PAUSE_SOURCE`. **If it -is anything else, run the three calls again and check again.** The check is part of the -procedure, not an extra. +is anything else, run the three calls, the wait and the check again, until it is.** The wait +and the check are part of the procedure, not extras. - A `DELETE` answers 404 when there is no source to delete. That is the normal case and it is fine; run the next call anyway. @@ -900,9 +908,19 @@ procedure, not an extra. has made its last read of the state ("my pause, and its time is up") goes on to set `AGENTS_PAUSED=false`; if the first two calls land between that read and that write, the fleet ends up running and every call still looks successful. Nothing done from outside can stop a write the breaker has - already decided on. What can be done is to look afterwards: a breaker run that has read the - state finishes within seconds, and once it has resumed, its source is gone and it will not - touch a hand pause. So one repeat is enough, and the minute is there to let that run finish. + already decided on. What can be done is to look afterwards, once that run has finished: + checking while it is still running can show a pause it is about to lift, which is why the + wait comes first. After a resume its source is gone and it will not touch a hand pause. + A different run can trip again in the meantime; the next round of three calls covers that, + and the breaker's minimum hold (30 minutes by default, `BUDGET_BREAKER_MIN_HOLD_MIN`) stops + it from lifting that new pause straight away, so one repeat is normally enough. Repeat + until the check passes anyway. +- How this was checked: an exhaustive interleaving of these steps against the breaker's own + read and write sequence (taken from `scripts/budget_breaker.sh` and + `scripts/lib/budget-breaker-writer.sh`), with up to four breaker runs and every starting + state, found no run that ends anything but paused with no source when the wait and one + repeat are kept. Without the repeat the resume race leaves the fleet running; without the + wait the check can pass just before a resume lands. - This is why no drill of those races is required before arming. What it rests on is that a read of the variable list sees a write or a delete made just before it, which is one of the live checks under "Arming preconditions" and blocks arming until it has passed. @@ -1219,9 +1237,9 @@ as soon as the workflow is on `main`. - The window is narrowed, not closed, by the breaker alone: a pause set by hand in the one API call between that second read and the write of `AGENTS_PAUSED` would still be claimed. The hand-pause procedure closes it from the other side: delete - `AGENTS_PAUSE_SOURCE` before and after setting the pause, then check the state a - minute later and repeat if it is not as expected ("Setting a pause by hand: three calls, - then a check"). + `AGENTS_PAUSE_SOURCE` before and after setting the pause, wait for any breaker run + already under way, then check the state and repeat until it is as expected ("Setting a + pause by hand: three calls, a wait, then a check"). - If the roll-back itself fails, the run is red with one `::error::` that says what is left. A failed delete of the source leaves `AGENTS_PAUSE_SOURCE=rate-limit-observed` **and** `AGENTS_PAUSE_UNTIL` in place on purpose (a pause carrying the breaker's source diff --git a/scripts/budget_breaker.sh b/scripts/budget_breaker.sh index bd7205b8c..e2ab1214b 100755 --- a/scripts/budget_breaker.sh +++ b/scripts/budget_breaker.sh @@ -307,7 +307,7 @@ run_url() { # announces a pause. Its first line is the pause's marker, which is how a later # run finds it (bbw_pause_notice_posted). pause_notice_body() { - printf '%s\n**%s**\n\n- **Why:** %s\n- **Until:** `%s` (the breaker lifts its own pause at the first run after that time)\n- **Set:** `%s=true`, `%s=%s`, `%s=%s` on the organisation `%s`\n\nItems that reported a quota refusal (up to 20):\n%s\n\nTo keep the fleet paused by hand, set `%s=true` and delete `%s`: the breaker never touches a pause without its own source. To end this pause early, set `%s=false` and delete `%s`.' \ + printf '%s\n**%s**\n\n- **Why:** %s\n- **Until:** `%s` (the breaker lifts its own pause at the first run after that time)\n- **Set:** `%s=true`, `%s=%s`, `%s=%s` on the organisation `%s`\n\nItems that reported a quota refusal (up to 20):\n%s\n\nTo keep the fleet paused by hand, follow "Setting a pause by hand" in AGENTS.md (set `%s=true` and delete `%s` in the order given there, then check): the breaker never touches a pause without its own source, but it can lift this one between your calls. To end this pause early, set `%s=false` and delete `%s`.' \ "$(bbw_pause_marker "$1")" "$2" "$3" "$1" \ "$BBW_VAR_PAUSED" "$BBW_VAR_SOURCE" "$BUDGET_BREAKER_PAUSE_SOURCE" "$BBW_VAR_UNTIL" "$1" "$ORG" \ "${4:-- none listed}" "$BBW_VAR_PAUSED" "$BBW_VAR_SOURCE" "$BBW_VAR_PAUSED" "$BBW_VAR_SOURCE" diff --git a/scripts/lib/budget-breaker-writer.sh b/scripts/lib/budget-breaker-writer.sh index 72efd16c7..e5dac345e 100644 --- a/scripts/lib/budget-breaker-writer.sh +++ b/scripts/lib/budget-breaker-writer.sh @@ -930,7 +930,7 @@ The budget breaker (`.github/workflows/budget-breaker.yml`, #2153) posts one com It pauses the fleet when the fleet itself reports that the model provider is refusing work for quota, by setting the organisation Actions variables `AGENTS_PAUSED=true`, `AGENTS_PAUSE_SOURCE=rate-limit-observed` and `AGENTS_PAUSE_UNTIL`. It only ever lifts a pause that carries its own source. -- To keep the fleet paused by hand: set `AGENTS_PAUSED=true` and delete `AGENTS_PAUSE_SOURCE`. The breaker never touches a pause without its own source. +- To keep the fleet paused by hand: follow "Setting a pause by hand" in `AGENTS.md`. It sets `AGENTS_PAUSED=true` and deletes `AGENTS_PAUSE_SOURCE` in a set order and then checks, because a breaker run already under way can lift a pause between two hand-made calls. The breaker never touches a pause without its own source. - To end a breaker pause early: set `AGENTS_PAUSED=false` and delete `AGENTS_PAUSE_SOURCE`. - To switch the breaker off: set the repository variable `BUDGET_BREAKER_ARMED` to `false`, or delete it. Only the exact lowercase value `true` arms it. From df9e779469b42f95d5bf598704d038d292ebab94 Mon Sep 17 00:00:00 2001 From: don-petry <36422719+don-petry@users.noreply.github.com> Date: Wed, 14 Oct 2026 06:19:34 -0500 Subject: [PATCH 4/5] docs(fleet-pause): the wait fails closed, the check reads each variable, and every route to a hand pause goes through the procedure From review. The wait now reports success only if the listing and every wait worked, and says to use a personal gh login (gh run watch does not accept a fine-grained token). The check reads the two variables one at a time instead of a list that was asked for 100 per page and not paginated. 'Overriding a breaker pause by hand' still told the operator to keep a pause by deleting the source alone, which a decided resume can undo; it now points at the procedure. The live check's reads paginate as the breaker does. Refs #2251 Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 04b7a88f2..05b4ec1a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -866,20 +866,30 @@ gh api -X DELETE orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE ``` Then wait until every Budget Breaker run that was already under way has finished. This -command waits for each run in progress now and returns when they are done (a run that starts -later reads your pause and leaves it alone): +command waits for each run in progress now and says so only if the listing and every wait +worked (a run that starts later reads your pause and leaves it alone). Run it with your own +`gh` login: `gh run watch` does not accept a fine-grained token. ```bash -for id in $(gh run list -R petry-projects/.github-private --workflow budget-breaker.yml --status in_progress --json databaseId --jq '.[].databaseId'); do gh run watch "$id" -R petry-projects/.github-private >/dev/null 2>&1; done; echo "no earlier breaker run is still in progress" +ok=1 +ids=$(gh run list -R petry-projects/.github-private --workflow budget-breaker.yml \ + --status in_progress --json databaseId --jq '.[].databaseId') || ok=0 +while read -r id; do + [ -n "$id" ] || continue + gh run watch "$id" -R petry-projects/.github-private >/dev/null 2>&1 /dev/null); then echo "AGENTS_PAUSED=$p"; else echo "AGENTS_PAUSED could not be read"; fi +if v=$(gh api orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE --jq .value 2>/dev/null); then echo "AGENTS_PAUSE_SOURCE=$v"; else echo "no AGENTS_PAUSE_SOURCE"; fi ``` -The answer must be exactly `AGENTS_PAUSED=true (all)`, with no `AGENTS_PAUSE_SOURCE`. **If it +The answer must be exactly `AGENTS_PAUSED=true (all)` and `no AGENTS_PAUSE_SOURCE`; any other +answer, including an error from the first read, is a failed check. **If it is anything else, run the three calls, the wait and the check again, until it is.** The wait and the check are part of the procedure, not extras. @@ -1287,8 +1297,8 @@ as soon as the workflow is on `main`. - a read of the variable list sees a write, and a delete, made just before it. The pause guard and the hand-pause procedure both rest on this, and it replaces a drill of the hand-pause races (there is no test organisation to run one in). Twenty times, with the - throwaway variable: create it, read `orgs/petry-projects/actions/variables?per_page=30` - and find it; `PATCH` a new value, read and find that value; delete it, read and find it + throwaway variable: create it, read the list as the breaker does (`gh api --paginate` + on `orgs/petry-projects/actions/variables?per_page=30`) and find it; `PATCH` a new value, read and find that value; delete it, read and find it gone. Once the dedicated write credential exists, make the reads with it, since it is the breaker's reader. **Arming is blocked until this has passed. One stale read fails it, and arming then stays blocked until a drill has been run after all;** @@ -1321,12 +1331,14 @@ as soon as the workflow is on `main`. described above on every run.) This stops further writes; it does not lift a pause that is already set (see the next point). - **Overriding a breaker pause by hand.** To **keep** the fleet paused past the breaker's - time, delete `AGENTS_PAUSE_SOURCE` (leave `AGENTS_PAUSED=true`): the pause is then a human - one and automation never lifts it. To **end** a breaker pause early, set + time, follow "Setting a pause by hand: three calls, a wait, then a check": the pause then + carries no source, and automation never lifts it. Deleting `AGENTS_PAUSE_SOURCE` on its own + is not enough, because a resume the breaker has already decided on can still set + `AGENTS_PAUSED=false` afterwards. To **end** a breaker pause early, set `AGENTS_PAUSED=false` **and delete `AGENTS_PAUSE_SOURCE`**; if the provider is still refusing work the next refusals trip it again, so disarm first if you want it to stay up. - To hold the fleet paused regardless of the breaker, set `AGENTS_PAUSED=true` **without** - `AGENTS_PAUSE_SOURCE` (the manual path). Ownership is the source and nothing else: a + To hold the fleet paused regardless of the breaker, use the same procedure (the manual + path: `AGENTS_PAUSED=true` **without** `AGENTS_PAUSE_SOURCE`). Ownership is the source and nothing else: a `AGENTS_PAUSE_SOURCE=rate-limit-observed` left behind on an unpaused fleet makes the next pause set by hand look like the breaker's, so the next armed run removes it (see "Leftover ownership markers are removed"); a run that is not armed raises a `::warning::` asking for From 96b9acde032f0dd393b8886eecb7a002909d79ca Mon Sep 17 00:00:00 2001 From: don-petry <36422719+don-petry@users.noreply.github.com> Date: Wed, 14 Oct 2026 06:22:05 -0500 Subject: [PATCH 5/5] docs(fleet-pause): only a 404 counts as no pause source in the check From review: the check printed the passing answer for any failed read of the source, including a rate limit or an expired login. It now tells a 404 from other failures and reports those as a failed check. Refs #2251 Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 05b4ec1a9..7820cdb1d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -885,11 +885,13 @@ Then read the state, one variable at a time (no list, so no page can be missed): ```bash if p=$(gh api orgs/petry-projects/actions/variables/AGENTS_PAUSED --jq '"\(.value) (\(.visibility))"' 2>/dev/null); then echo "AGENTS_PAUSED=$p"; else echo "AGENTS_PAUSED could not be read"; fi -if v=$(gh api orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE --jq .value 2>/dev/null); then echo "AGENTS_PAUSE_SOURCE=$v"; else echo "no AGENTS_PAUSE_SOURCE"; fi +if v=$(gh api orgs/petry-projects/actions/variables/AGENTS_PAUSE_SOURCE --jq .value 2>&1); then echo "AGENTS_PAUSE_SOURCE=$v" +elif grep -q 'HTTP 404' <<< "$v"; then echo "no AGENTS_PAUSE_SOURCE"; else echo "AGENTS_PAUSE_SOURCE could not be read"; fi ``` The answer must be exactly `AGENTS_PAUSED=true (all)` and `no AGENTS_PAUSE_SOURCE`; any other -answer, including an error from the first read, is a failed check. **If it +answer, including "could not be read" for either variable, is a failed check (only a 404 +counts as "no source"; a rate limit or an expired login does not). **If it is anything else, run the three calls, the wait and the check again, until it is.** The wait and the check are part of the procedure, not extras.