Repository navigation
Budget breaker depends on scheduled ticks that this repository barely receives: add a default-branch-only event trigger before arming #2259
Description
Problem
The budget breaker ("Budget breaker writer", #2153) is schedule-only, every 10 minutes, so that only default-branch code ever runs with its write credential. But this repository is receiving almost no scheduled ticks.
Observed 2026-10-11: the workflow merged at 01:55 UTC and is listed as active. By 03:10 UTC it had not run once (seven slots). Over the same period, across every scheduled workflow in this repository, GitHub delivered 2 scheduled run(s) since 01:00 UTC; the twice-hourly budget poller last ran at 01:00, and a 15-minute sweep ran once. "GitHub drops about 60% of scheduled ticks here" (#2160) measured 40% delivery; under load it is far lower.
A breaker that may not run for an hour or more cannot react to a quota outage in time, and once armed it also could not resume the fleet on time, which is worse: its own pause would outlive the reset until a tick happens to arrive.
Proposal
Keep the schedule as a backstop and add an event trigger that still runs only default-branch code:
- Add a
workflow_runtrigger on completion of the agent workflows that post quota markers (dev-lead first). GitHub runs aworkflow_runworkflow from the default branch's definition, so the property the schedule-only rule protects (no branch code with the write credential) is kept. Confirm that against GitHub's documentation and state it in the workflow header; the checkout must stay on the default branch and must never check out the triggering run's ref. - Bound the cost: the existing
budget-breakerconcurrency group already allows one running and one pending run. Add a cheap early exit when the previous breaker run finished less than a couple of minutes ago and the fleet is not paused. - For resume specifically, a pause must not depend on a tick arriving: consider having the paused agents' own
fleet-pausednotice jobs or the retry cron's next run be able to wake the breaker, or accept and document the worst-case delay. - Report, in the run summary, how long it has been since the previous breaker run, so poor delivery is visible.
Acceptance criteria
- The static test that pins "no trigger that can run branch code" is updated to allow exactly
scheduleandworkflow_run, and still rejectsworkflow_dispatch,pull_request,pushand the rest. - A test that the checkout step never uses the triggering run's ref or SHA.
- The documented worst-case reaction time and worst-case resume delay, with the measurement they are based on.
This should be decided before arming ("Auto-pause companion", #1565). Related: #2160, #2251.
Activity
github-actions commentedon Oct 11, 2026
Thanks for the detailed write-up and the delivery measurements, which make the problem easy to see. Before this is picked up, one question: for the resume case (point 3), do you prefer having the
fleet-pausednotice jobs or the retry cron wake the breaker, or would you rather accept and document a worst-case resume delay? That choice decides how much new wiring is needed and what the documented worst-case numbers will be.don-petry commentedon Oct 11, 2026
Second data point, 2026-10-11 04:55 UTC: three hours after merging, the breaker workflow has still not run once (eighteen 10-minute slots). Across every scheduled workflow in this repository, one scheduled run has been delivered since 02:00 UTC. The workflow is listed as active, so this is delivery, not configuration. Whatever cadence the schedule asks for, it cannot be relied on for reaction or for resume; the event trigger proposed above is the substance of this issue, not an optimisation.
don-petry commentedon Oct 11, 2026
dev-lead is withholding action on this item.
It is labeled
dev-lead:hands-off(deliberately kept off-limits to dev-lead (e.g. changes to the dev-lead automation itself)), so dev-lead will not pick it up while that label is present. This notice is posted once so the withhold is visible rather than looking like a stalled run.To re-enable automated pickup: remove the
dev-lead:hands-offlabel.fix(budget-breaker): a failed run does not cover a slot for the early…
feat(budget-breaker): wake on agent workflow runs, keep the schedule …
don-petry commentedon Oct 11, 2026
First 84 minutes under the new trigger (17:35 to 18:59 UTC, 2026-10-11)
It works as designed, and it costs more than the pull request estimated.
No self-trigger loop. Lined up second by second over a busy stretch (18:10 to 18:12), every breaker run was created 1 to 2 seconds after a Dev-Lead Agent run completed, and no Dev-Lead Agent run was created by a breaker run completing: the Dev-Lead Agent runs in that window were
issue_comment,pull_request_review,pull_request_review_comment,pull_requestandrepository_dispatchevents, with onecheck_run. The volume is real fleet activity, which was far burstier than the day's average the estimate was built on (21 Dev-Lead Agent runs in 3.5 minutes there).The early exit engages. Of six consecutive runs read (38162668958 onward), the first of the two-minute slot scanned and decided (
0 trusted, 0 dropped, decisionnone), and the next five took the early exit. The first run after the merge (38160312163) could not list earlier runs because the login's API quota was exhausted at that moment; it failed open and decided, as intended.What it costs while the breaker is disarmed and protecting nothing:
cancelled.Not yet observed: a breaker run following a Dev-Lead Agent run that ended
skipped(arming precondition 6). Onecheck_runDev-Lead Agent run endedskippedat about 18:10; I have not matched it to a breaker run.Options are with the maintainer: leave it and review after a day; lengthen the early exit's slot with the repository variable
BUDGET_BREAKER_MIN_INTERVAL_MIN(fewer organisation scans, same number of runs); or change the design so that fewer runs are created at all.test(budget-breaker): prove the next slot decides, guard the fixture'…
fix(budget-breaker): lengthen the early exit's default slot from two …