Skip to content

Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565

Open

Description

@don-petry

Story

As the maintainer of the agent fleet,
I want a scheduled job that reads the Claude subscription usage windows and flips AGENTS_PAUSED on and off automatically,
so that budget conservation stops being a manual act of noticing pressure and withholding a secret — the failure that produced the 2026-08-16→18 opaque fleet outage (#1525).


⚠️ Status update 2026-09-21 — the public half landed, this is now the only thing standing between the fleet and the budget

Epic petry-projects/.github#636 closed on 2026-09-07 with phases 1–6 merged. The library (scripts/lib/agent-rate-limit.sh), the config (standards/agent-rate-limits.json), the 5-hour static breaker and the 7-day glide path all exist and are tested. Two things keep the whole apparatus inert:

  1. Nothing produces telemetry — this issue.
  2. Nothing consults the breaker — scripts/agent-rate-limit-gate.sh contains no reference to arl_token_budget_gate or arl_token_weekly_glide_gate. Filed as petry-projects/.github#1155.

Measured on the fleet while both gaps were open:

session       33%  normal   resets 2026-09-21T05:20Z
weekly_all    79%  warning  resets 2026-09-22T16:00Z
weekly_scoped 36%  normal

79% of the weekly budget consumed with two days left in the window, against ~2,500 agent workflow runs over two days that merged nothing (see #1864). Nothing paused, nothing warned. The fleet's only other token-awareness check, check_provider_headroom, has never worked (#1862).

The integration contract is now fixed by the shipped library — build to it, do not invent a format. See the revised Scope §1.


Why this lives in the private repo

The public epic (petry-projects/.github#636) owns the decision (threshold config) and the gate (the library that defers a dispatch). It cannot own the actuation: the Claude credential (CLAUDE_CODE_OAUTH_TOKEN), the engine, and the token telemetry all live here. Epic #636 already lists this as an untracked cross-repo prerequisite. This issue tracks it.

Scope

  1. Telemetry adapter — scripts/lib/usage-telemetry.sh: one call to GET https://api.anthropic.com/api/oauth/usage, emitting the envelope shape the shipped public library already consumes (scripts/lib/agent-rate-limit.sh:485):

    { "status": <http_status>, "retry_after": <int?>, "observed_at": <epoch?>, "body": <raw upstream response> }
    • The adapter's job is the transport envelope, not normalization. body carries the upstream response unmodified; the library's own arl_token_extract_percent / arl_token_extract_resets_at / arl_token_window_active do the field extraction, preferring limits[] over the flattened five_hour / seven_day keys. Re-normalizing here would create a second, drifting parser — the exact failure model-pricing.tsv exists to prevent.
    • Publish it to the library through its existing seam: write the envelope to a file and export AGENT_TOKEN_BUDGET_TELEMETRY_FILE, or expose a command via AGENT_TOKEN_BUDGET_TELEMETRY_CMD. Do not add a new seam.
    • Headers are load-bearing: Authorization: Bearer $CLAUDE_CODE_OAUTH_TOKEN, anthropic-beta: oauth-2025-04-20, and User-Agent: claude-code/<version> — without the User-Agent the caller hits an aggressively rate-limited bucket and gets persistent 429s.
    • On a 429, populate retry_after and observed_at; the library's arl_token_transport_decision uses them to distinguish a fresh hard-cap signal (defer) from a stale one (allow).
    • Non-200 / malformed / missing-window ⇒ allow-with-warning, never a hard failure (per the ADR default in .github#637).
  2. Scheduled poller — an hourly workflow evaluating the configured thresholds and, on trip, setting the org Actions variable AGENTS_PAUSED=true plus AGENTS_PAUSE_SOURCE=weekly-glide (or five-hour).

  3. Auto-resume — after the window's resets_at passes, clear AGENTS_PAUSED only if AGENTS_PAUSE_SOURCE marks it as automation-set. A pause a human set by hand is never cleared by the poller.

  4. Fleet-monitor awareness — paused runs already count as healthy (Make 'agents intentionally paused' a first-class state — deliberate token disable presented as fleet-wide outage (2026-08-16) #1525 AC Optimize review: small-PR and incremental fast paths #4); extend the report to name why (which window, what percent) so a budget pause is legible at a glance rather than looking like a quiet fleet.

  5. Notification — one notice when the breaker trips and one when it clears. A pause nobody is told about reproduces the original Make 'agents intentionally paused' a first-class state — deliberate token disable presented as fleet-wide outage (2026-08-16) #1525 confusion in a new shape.

Acceptance Criteria

  1. scripts/lib/usage-telemetry.sh exists, is shellcheck-clean, and emits the envelope in Scope §1 verbatim; unit tests mock curl and cover 200 / 429-with-retry-after / other-non-200 / malformed / missing-window. No live network in tests.
  2. The poller runs on a schedule (hourly is far inside the endpoint's ≥180 s guidance) and supports a dry-run mode that logs the decision without writing any variable — this is the default until the epic's human sign-off.
  3. Setting/clearing the org variable works with a credential that actually carries the permission (see the credential note below), and a permission failure surfaces as one actionable error, not a silent no-op.
  4. Auto-resume never clears a human-set pause; asserted by a test that sets AGENTS_PAUSED=true without AGENTS_PAUSE_SOURCE and confirms the poller leaves it alone.
  5. AGENTS.md documents the automatic pause alongside the existing manual pause/resume paragraph, including how to override it.
  6. Verifies — and records — that the long-lived claude setup-token credential is accepted by the usage endpoint (see below).
  7. Round-trip proof against the real consumer. A test feeds the adapter's envelope to the shipped arl_token_budget_gate and arl_token_weekly_glide_gate and asserts the expected decision=. An envelope that the public library cannot read is the single most likely way this ships broken and nobody notices.
  8. Telemetry liveness is asserted, not assumed. The poller emits a positive "telemetry read OK, session=N% weekly_all=M%" record on every successful poll, and the fleet monitor raises a warning when no such record has appeared within a configurable staleness window. Every failure path in this design is fail-open, which means a silently broken poller is indistinguishable from a healthy under-budget fleet — the exact shape of the three-week gap this issue sat in. A breaker nobody can prove is alive is not a breaker.
  9. The dry-run decision is logged against live data before arming. Run the poller in dry-run for at least one full weekly window and record what it would have done, so the thresholds are calibrated against real utilization rather than assumed. The 79% observation above suggests the 5-hour breaker (90%) would rarely trip while the weekly glide path would be the binding control — confirm or refute that before arming either.

Dev Notes — two verification items found during the spike

Credential type is unproven. The endpoint was probed successfully (HTTP 200, live window data) using the short-lived OAuth access token from a local credential store — reconfirmed 2026-09-21. CI authenticates with the long-lived claude setup-token credential. Both carry the sk-ant-oat01- prefix and both are OAuth-issued, so acceptance is likely but not demonstrated. Verify by running the poller in dry-run in CI against the existing secret — do not mint a fresh setup-token locally to test, since that risks invalidating the credential the fleet currently runs on. If the long-lived token is rejected, the fallback is to run the poller with a dedicated short-lived credential refreshed by the workflow, which is a materially larger change and should be re-scoped before implementation.

Org-variable write needs a permission the fleet may not have. Writing an org-level Actions variable returns 403 — You must be an org admin or have the actions variables fine-grained permission for a classic token with repo / workflow / read:org, even when the caller is already an org admin. It requires admin:org (classic) or the fine-grained org Variables: write permission. Confirm whether GH_PAT_DON_PETRY carries this before implementation; if not, either widen that PAT or fall back to per-repo variables (writable under plain repo scope) at the cost of N writes and a slower fleet-wide stop.

Sequencing

This issue and petry-projects/.github#1155 are independently landable and jointly required — neither alone changes fleet behaviour. Land this first: the public gate can then be wired and armed against real telemetry rather than a mock. #1862 (the dead in-path headroom probe) should consume this adapter rather than build a second reader, and #1863 (weekly resets parsing as "unknown") should take its resets_at from this envelope instead of parsing prose.

References


Calibration evidence for AC #9 — the configured thresholds would NOT have fired this week

Observed on the live fleet, 2026-09-21 (weekly window resets 2026-09-22T16:00Z, i.e. ~1 day out):

session       3%   normal    ← 5-hour window had just reset
weekly_all   89%   warning
weekly_scoped 40%  normal

Against the thresholds currently in standards/agent-rate-limits.json:

Breaker Threshold at this moment Actual Would it pause?
session static 90% 3% no
weekly_all glide 100 − 2 × 1 day = 98% 89% no

So on the week that ran the budget from 79% to 89% while merging almost nothing, neither breaker would have tripped even if both were armed and fed. The 5-hour window never approached 90% because the spend was spread thin and continuous rather than bursty; the glide path is at its most permissive exactly when the week is nearly spent.

This is not an argument against the design — it is the calibration AC #9 asks for, arriving early and for free. Two things it suggests, to be confirmed by the required dry-run window rather than acted on blindly:

  1. reserve_pct_per_day = 2 may be too lax. It only bites in the final ~1–2 days, by which point the budget is already gone. A steeper reserve, or a floor well below 86%, would engage while there is still something to conserve.
  2. A continuous low-level drain is invisible to both breakers. Both are level-triggered on a percentage; neither looks at rate. A burn-rate signal ("at the current rate the weekly budget exhausts N hours before resets_at") would have flagged this week on about day two. Worth considering as a third, advisory-only signal before adding another pause-worthy one.

Record what the dry-run would have done across a full window before arming, and compare it against this observation.


Design question raised 2026-09-21 — should the breaker pause, or degrade to Gemini?

The ADR gives this breaker exactly one action: set AGENTS_PAUSED and stop the fleet. That was specified before the multi-key Gemini fallback (#1777) was in place. It now is, and it is idle:

  • Org secrets GOOGLE_API_KEY, GOOGLE_API_KEY_2, GOOGLE_API_KEY_3 all exist (three distinct keys).
  • Both agents receive them — dev-lead.yml and pr-review-trigger.yml use secrets: inherit, and dev-lead-reusable.yml / pr-review.yml map all three into env at every engine step.
  • engine.sh has three nested fallback layers: per-key rotation within a model, then the in-Gemini model chain, then cross-provider claude → gemini → copilot.
  • No DEV_LEAD_ENGINES kill-switch is set, so the full chain is live.

And it has never fired. The most recent weekly cost report is 100% Claude — zero Gemini calls, zero Copilot calls — across the week that carried weekly_all from 79% to 97%. The reason is that cross-provider fallback is triggered only by a failed call (exit 2). Below 100% nothing fails, so nothing falls over. The response is binary: full Claude until hard exhaustion.

So at the moment the breaker would trip (90% session, or the weekly glide), there are three unused Gemini keys available. Stopping the fleet is not obviously the right action when a working, cheaper engine is sitting idle.

Worth deciding before arming:

  1. Degrade before pausing. At the pause threshold, set the engine chain to start at gemini (the DEV_LEAD_ENGINES kill-switch already expresses exactly this, with no new mechanism) instead of halting. Reserve AGENTS_PAUSED for when every provider is exhausted or a human wants a hard stop.
  2. If degrading, say so loudly. Gemini output quality differs from Claude's; a silent engine swap changes review and implementation behaviour without anyone being told. It needs the same "one notice when it trips, one when it clears" treatment as a pause (Scope §5), naming the engine.
  3. Keep weekly_scoped out of it either way — per-model exhaustion is already the model-fallback chain's job (ADR §2.5).

This does not change the telemetry work in this issue, which is the prerequisite for either action. It changes what the gate does with the signal, so it likely belongs in petry-projects/.github#1155's scope as well. Raising it here so it is decided rather than inherited.

Activity

  1. github-actions commented on Aug 20, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up — the cross-repo context, the credential/permission findings from the spike, and the explicit non-goals make this very actionable. Before implementation begins, could you confirm whether GH_PAT_DON_PETRY currently carries the org-level Variables write permission (or admin:org), since that determines whether the poller can write directly to the org variable or needs to fall back to per-repo writes?

  2. don-petry commented on Sep 22, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1565 — Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget

    Scope of this PR — the "land this first" prerequisite (Sequencing §)

    Implement scripts/lib/usage-telemetry.sh: the transport-envelope adapter for
    GET https://api.anthropic.com/api/oauth/usage, emitting the exact envelope the shipped public
    library (petry-projects/.github scripts/lib/agent-rate-limit.sh) already consumes via its
    AGENT_TOKEN_BUDGET_TELEMETRY_FILE / AGENT_TOKEN_BUDGET_TELEMETRY_CMD seam — plus unit tests, a
    round-trip proof against a pinned copy of that consumer, and the AGENTS.md docs.

    Why the poller/actuation is deliberately NOT in this PR

    The consumer library does not exist in this private repo (it lives in the public
    petry-projects/.github), so the poller cannot source a breaker to consult here yet. Wiring the
    hourly poller + org-variable actuation also depends on three things a code-only PR cannot discharge,
    all called out in the issue's Dev Notes / ACs:

    The issue's own Sequencing says "Land this [adapter] first: the public gate can then be wired and
    armed against real telemetry."
    This PR delivers exactly that landable, hermetic unit; the poller,
    notifications, fleet-monitor awareness, and live actuation follow once the above are discharged.

    Implementation Checklist

    • scripts/lib/usage-telemetry.sh — one curl to the OAuth usage endpoint with the three
      load-bearing headers (Authorization: Bearer, anthropic-beta: oauth-2025-04-20,
      User-Agent: claude-code/<version>); emits {status, retry_after?, observed_at?, body} with
      body the unmodified upstream response (no re-normalization — the library owns extraction).
    • 429 path populates retry_after (from the Retry-After header) + observed_at.
    • Non-200 / transport error / missing token → status with no/empty body (library fails safe to
      allow-with-warning). Never a hard failure.
    • usage_telemetry_publish_file helper — writes the envelope + exports the existing
      AGENT_TOKEN_BUDGET_TELEMETRY_FILE seam (no new seam).
    • Register the two new bats files in .github/workflows/lint.yml.
    • AGENTS.md — document the (forthcoming) automatic pause alongside the manual pause paragraph.

    Tests to Write

    • tests/dev-lead/unit/test_usage_telemetry.bats — mock curl, cover 200 / 429-with-retry-after
      / other-non-200 / malformed / missing-window / missing-token
      . No live network.
    • tests/dev-lead/integration/test_usage_telemetry_roundtrip.bats — feed the adapter's envelope to
      the real arl_token_budget_gate and arl_token_weekly_glide_gate (pinned copy under
      tests/fixtures/agent-rate-limit/) and assert decision=allow/defer for session ≥/<90% and
      the weekly glide path (AC Workflow should skip no-op PRs in MAX_PRS count #7).

    Test Command

    bats tests/dev-lead/unit/test_usage_telemetry.bats tests/dev-lead/integration/test_usage_telemetry_roundtrip.bats
    then the full scripts/run-bats.sh … list + bash .dev-lead/scripts/dev-lead-lint.sh.

  3. don-petry commented on Sep 22, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Progress (in progress — not yet durable)

    Provisional. Nothing is pushed yet; the automation posts the authoritative
    completion record (with PR number + head SHA) after the push. Treat the notes
    below as intent, not delivered results.

    What this workspace contains (intended)

    The "land this first" prerequisite from the issue's Sequencing — the transport
    adapter and its proof of contract compatibility — plus docs:

    • scripts/lib/usage-telemetry.sh — one curl to GET …/api/oauth/usage with the
      three load-bearing headers (Authorization: Bearer, anthropic-beta: oauth-2025-04-20,
      User-Agent: claude-code/<version>), emitting {status, retry_after?, observed_at?, body}
      with body the unmodified upstream response. 429 populates retry_after +
      observed_at; non-200 / transport error / missing token → status-only envelope
      (library fails safe to allow). Publishes via the existing
      AGENT_TOKEN_BUDGET_TELEMETRY_FILE seam — no new seam.
    • tests/dev-lead/unit/test_usage_telemetry.bats — mocks curl; covers
      200 / 429-with-retry-after / other-non-200 / malformed / missing-window / missing-token.
    • tests/dev-lead/integration/test_usage_telemetry_roundtrip.bats + a pinned copy of
      the public consumer under tests/fixtures/agent-rate-limit/ — feeds the adapter's
      envelope through the file seam into the real arl_token_budget_gate /
      arl_token_weekly_glide_gate and asserts decision=allow/defer (AC Workflow should skip no-op PRs in MAX_PRS count #7).
    • AGENTS.md — documents the automatic budget pause + the human override, beside the
      existing manual pause paragraph (AC feat: add Copilot engine support via REVIEW_ENGINE toggle #5).
    • .github/workflows/lint.yml — registers the two new bats files.

    Deliberately deferred (not in this slice — see the plan comment for rationale)

    The scheduled poller + org-variable actuation, notification, and fleet-monitor
    awareness (Scope §2–5) are not included, because the consumer library is not
    co-located in this private repo yet and because arming depends on verifications a
    code-only change cannot discharge: the long-lived claude setup-token acceptance by
    the endpoint (AC #6 / Dev Note 1), GH_PAT_DON_PETRY carrying org Variables: write
    (AC #3 / Dev Note 2), and a full-weekly-window dry-run calibration (AC #9). The issue's
    own Sequencing says to land this adapter first and wire/arm the gate against real
    telemetry afterward.

    Local check output (provisional — not the pushed run)

    • New adapter unit + round-trip tests: all green locally (15 cases).
    • bash .dev-lead/scripts/dev-lead-lint.sh: clean (shellcheck on 154 files, profiles, CODEOWNERS).
    • Full CI bats list: the only failures are 11 pre-existing cross-file-interference
      cases in three pr-review files (they reproduce with none of my files involved and
      pass in isolation) — unrelated to this change.

    Notes

    The vendored tests/fixtures/agent-rate-limit/ copy is a pinned snapshot of the public
    consumer for hermetic round-trip testing; a diff there is the intended drift signal to
    refresh it. Follow-ups this unblocks: #1155 (wire the gate), #1862 (consume this adapter
    instead of the dead headroom probe), #1863 (take resets_at from this envelope).

  4. don-petry commented on Sep 22, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete — PR #1888

    The implementation for issue #1565 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  5. added 2 commits that reference this issue on Sep 22, 2026
  6. added a commit that references this issue on Oct 2, 2026
  7. don-petry commented on Oct 2, 2026

    @don-petry
    CollaboratorAuthor

    Maintainer decision recorded 2026-10-02: when the breaker trips the fleet degrades to the Gemini engine rather than pausing, and Gemini quota must be checked and respected the same as Claude's. Consequences for this issue: arming is blocked on Gemini metering (#2030); the actuation is likely a flip of the AI_ENGINES variable (e.g. to gemini,claude) rather than AGENTS_PAUSED, to be restored after the window resets, and it must refuse to degrade when Gemini itself is constrained. Order of work: #2030 (meter Gemini), #2029 (dry-run poller), then an actuation issue. Scope items 2 and 3 above (variable set/clear, auto-resume) are unchanged in shape but target AI_ENGINES.

  8. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Decision update, 2026-10-09 (supersedes the earlier note that the fleet would degrade to Gemini): pause is the primary control; Gemini is a small trickle for cheap work only, sized by data so spend can be raised later. Consequences for this issue: (1) arming means setting AGENTS_PAUSED with AGENTS_PAUSE_SOURCE, as originally scoped; the AI_ENGINES-flip idea is dropped. (2) The Gemini trickle is tracked in #2147 (default off, exempts only a budget-source pause, allowlisted cheap tiers, only when Gemini capacity exists). (3) Order of work: dry-run poller (#2029) for a full weekly window (AC 9 here), then arming, with the trickle layered on after #2041 and #2146. (4) Open prerequisite, not code: the org-variable write permission noted in this issue (needs admin:org or fine-grained org Variables: write); confirm GH_PAT_DON_PETRY carries it before the arming PR.

  9. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Constraint confirmed by the maintainer, 2026-10-09: GH_PAT_DON_PETRY is a classic token without organisation variable write. GitHub documents admin:org as the classic-token requirement for org variables and the repo scope for repository variables (the page does not document fine-grained or GitHub App permission names; confirm those in the token or App creation UI). Note AGENTS_PAUSED is read as vars.AGENTS_PAUSED inside the reusable workflows, which resolves in the CALLING repository context, so a repo-level variable pauses only that repo. Options under consideration for the arming step (decision pending): (a) a dedicated fine-grained PAT owned by the org with org Variables write, (b) a GitHub App token, (c) per-repo variables written with the existing repo scope for each consumer repo, (d) notify-only until a human sets the variable. Widening the shared classic token to admin:org is NOT proposed. The dry-run poller is unaffected and proceeds.

  10. don-petry commented on Oct 9, 2026

    @don-petry
    CollaboratorAuthor

    Decision recorded 2026-10-09: the arming credential is a dedicated fine-grained PAT owned by the org with organisation Variables write only (option A), stored as the secret BUDGET_BREAKER_PAT, available only to this repository. The writer is tracked in #2153 as a separate default-off workflow so the dry-run poller keeps its no-write-path guarantee. The shared classic token is not widened. The maintainer creates and stores the token; no automation touches its value.

  11. don-petry commented on Oct 10, 2026

    @don-petry
    CollaboratorAuthor

    Maintainer decision 2026-10-10: the breaker is reactive (it does not read the usage endpoint; see the evidence and decision on #2029). The pieces are "Reactive budget breaker, decision only" (#2221) and "Budget breaker writer" (#2153). For criterion 9 here, the dry-run evidence reviewed before arming now comes from the writer running disarmed and logging the write it would have made, not from poller records.

  12. don-petry commented on Oct 11, 2026

    @don-petry
    CollaboratorAuthor

    Arming status, 2026-10-11 17:15 UTC. The breaker is still disarmed and BUDGET_BREAKER_ARMED is unset.

    Blocker State
    "Budget breaker depends on scheduled ticks that this repository barely receives" (#2259) In review as #2293: adds a workflow_run wake-up, schedule kept as backstop. The maintainer agreed in principle today.
    Live checks on "Budget breaker: close the gaps found in review before it is armed" (#2251) 8, 11 and 13 observed and recorded there; 6, 7 and 9 have exact commands and are the maintainer's; 10 waits for a real dev-lead rate limit; 12 needs a test organisation (the variable names are fixed in the writer by design).
    New with #2293: that workflow_run fires after a Dev-Lead Agent run that ended skipped From GitHub's documentation only. It is what wakes a resume. Observable after merge without a pause; record the run id here (arming precondition 6).
    New: "Canary promotion gate counts a fleet-paused feature-ideation run as a healthy sample" (combsmith/.github#1310) Open. A pause set by the breaker would let one paused run satisfy a ring's sample gate. Should be fixed before a breaker-set pause is relied on.
    "feature-ideation reusable calls the model but ignores AGENTS_PAUSED" (combsmith/.github#1295) Merged today (combsmith/.github#1311). Consumers get it only as feature-ideation/v1-* channels move; most pin v1-stable.
    "compliance-audit-and-improvement.yml calls the model and ignores AGENTS_PAUSED" (combsmith/.github#1309) Open; a second ungated model call found while checking the reusables.
    A full weekly window of disarmed runs reviewed (criterion 9) Not started in earnest: the breaker ran twice in its first 13 hours. #2293 is what makes this window meaningful.

    Also merged today: "Three monitors misread a paused run" (#2266, by #2281), so the canaries and the fleet monitor report a paused run as paused.

  13. don-petry commented on Oct 14, 2026

    @don-petry
    CollaboratorAuthor

    Arming precondition 6: observed (2026-10-14)

    Both halves of "the event trigger has been seen working in disarmed runs" are now on record.

    1. A run woken by workflow_run whose previous run: line shows a time (so the read credential can list the workflow's runs). Budget Breaker run 38229601217, 2026-10-12:

    triggered by: workflow_run
    previous run: finished 2026-10-12T09:23:47Z (success), 0 min before this one
    mode: disarmed
    early exit: an earlier run was created in this 10 min slot (since 2026-10-12T09:20:00Z) and petry-projects/.github-private shows no quota marker since; no scan, no decision
    

    2. A Budget Breaker run that starts within seconds of a Dev-Lead Agent run that ended skipped (what a resume relies on, and until now taken from GitHub's documentation only).

    • Dev-Lead Agent run 38229596615 (check_run) ended skipped at 09:23:39 UTC on 2026-10-12.
    • Budget Breaker run 38229601217 was created at 09:23:41, two seconds later.
    • It was a quiet minute, so nothing else explains the wake-up: the only other Dev-Lead Agent run nearby finished at 09:23:56, and neither of the other two listed workflows ran in those four minutes.
    • Across 12 to 14 October, all 24 Dev-Lead Agent runs that ended skipped were followed by a workflow_run breaker run within five seconds; 17 of those breaker runs executed and the rest were replaced while waiting, as designed.

    So workflow_run does fire for a triggering run that concluded skipped. What this does not show is a paused run: no pause has been set, so these were runs skipped by their event filter. A paused Dev-Lead Agent run ends the same way (every job skipped at the job level), which is why this is the relevant observation, but the first real pause is still the first direct one.

    Also since the last status here: the early exit's slot is ten minutes (#2307), the trusted-records filter and the ledger fix have merged (#2305, #2311), and on 13 October the breaker ran 16 times in the day, against 711 on the 11th, because fleet activity itself fell from 2,500 Dev-Lead Agent runs to 8.

  14. don-petry commented on Oct 14, 2026

    @don-petry
    CollaboratorAuthor

    The dry-run budget poller (budget-poller.yml) is disabled (disabled_manually); its last run was 2026-10-14 07:39 UTC. Under the reactive design it could only ever record degraded (permission_error, the CI credential cannot read usage). Its two readers already fail open on a missing log: the retry sweep's budget hold logs budget hold fail-open (no readable budget-poller-log artifact), and the fleet monitor warns that no OK poll exists, as it did before. To bring it back: gh workflow enable budget-poller.yml -R petry-projects/.github-private.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions