Skip to content

check_provider_headroom has never worked: probes with an unset ANTHROPIC_API_KEY, reads per-minute TPM instead of the subscription window, and pr-review never calls it #1862

Open

Description

@don-petry

Summary

check_provider_headroom (scripts/engine.sh:257) is the fleet's only in-path token-awareness check. It has never been able to return anything but "ok" for Claude, for three independent reasons. Closed issue #206 delivered it; it has been inert since.

Defect 1 — it authenticates with a credential the fleet does not have

The Claude arm probes api.anthropic.com/v1/messages with:

-H "x-api-key: ${ANTHROPIC_API_KEY:-}"

ANTHROPIC_API_KEY is set nowhere in this repository — not in any workflow, not in any script. The fleet authenticates with CLAUDE_CODE_OAUTH_TOKEN, and engine.sh itself unsets ANTHROPIC_API_KEY on the gemini and copilot arms (scripts/engine.sh:1726, :1738).

So the probe sends an empty API key, receives a 401 with no x-ratelimit-* headers, leaves used_pct at its initialized 0, and reports [headroom] claude usage 0% — ok. Every time. The function is fail-open by design, so this produces no error and no warning — it looks identical to a healthy 0%-utilized fleet.

Defect 2 — even authenticated, it reads the wrong window

The probe reads x-ratelimit-remaining-tokens / x-ratelimit-limit-tokens. Those are the per-minute TPM bucket. They carry no information about the 5-hour or 7-day subscription windows, which are what actually gets exhausted.

engine.sh's own header already says this (scripts/engine.sh:17-21):

Does NOT help when the daily subscription cap is exhausted (cap is shared across all Claude models — see issue #206 for the proactive headroom guard).

The comment points at #206 as the fix for the gap, but #206 is what produced this function.

Defect 3 — the heaviest consumer never calls it

check_provider_headroom is invoked from exactly one place: run_writer_with_fallback (scripts/engine.sh:1561). That reaches it only via the three dev-lead writer scripts (dev-lead-fix-ci.sh, dev-lead-fix-issue.sh, dev-lead-fix-reviews.sh).

pr-review never calls it at all. review-one-pr.sh / review-batch.sh go through run_triage / run_agentic / run_duck, none of which check headroom — and pr-review is the larger consumer: a sweep every 15 minutes, up to MAX_PRS=10 reviews per batch, roughly five model tiers per review.

Acceptance Criteria

  1. The Claude headroom probe reads the subscription windows (session, weekly_all), not the per-minute TPM headers, via the same telemetry envelope Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 produces — no second, divergent telemetry path.
  2. It authenticates with a credential the fleet actually holds, or it is deleted. A probe that cannot authenticate must not silently report 0%.
  3. Telemetry that is absent or unreadable still allows the call (fail-open is correct here), but emits a distinct warning — headroom: telemetry unavailable, proceeding — never usage 0% — ok. An unmeasurable budget and an empty budget must never render the same.
  4. run_triage, run_agentic, and run_duck consult headroom on the same terms as run_writer, so pr-review is covered.
  5. DEV_LEAD_USAGE_THRESHOLD is either honoured against the real window or removed; a documented knob that cannot bind is worse than none.
  6. A bats test asserts: unavailable telemetry ⇒ allow plus the distinct warning; over-threshold telemetry ⇒ skip; and that no code path can report 0% from a failed probe.

Notes for the implementer

References


Dispatch sequencing — deliberately not yet released to dev-lead (2026-09-21)

This issue is fully specified but intentionally carries no dev-lead label yet. Two reasons:

  1. It is blocked by Auto-pause companion: usage-telemetry adapter + scheduled poller that sets/clears AGENTS_PAUSED from the Claude budget #1565 (this consumes its telemetry envelope).
  2. The weekly Claude budget stood at 79% with two days left in the window, and every new dev-lead run opens another PR into the rebase treadmill described in Merge protections are unsatisfiable at N>1 open PRs: one merge to main forces a full re-review of every other PR — ~154 forced re-reviews in 48h, zero merges #1864 — which multiplies re-review cost across all open PRs, not just the new one. Adding parallel agent work before pr-review re-runs the full tier stack for a conflict-free base merge — carry the prior verdict forward when the diff-vs-merge-base is unchanged #1865 lands makes the budget problem worse, not better.

Release condition: add the dev-lead label once #1865 has merged, or after the weekly window resets (resets_at from the usage endpoint — do not assume a weekday), whichever comes first, and once the blocking issue above is resolved.

Activity

  1. github-actions commented on Sep 21, 2026

    @github-actions
    Contributor

    Thank you for such a thorough and precisely documented report — the specific line numbers, failure modes, and the note that fail-open makes this invisible are exactly what makes an issue actionable. Before we scope the fix: has #1565 landed yet? The notes say to consume the telemetry envelope it produces, so whether that's merged determines whether we implement the full subscription-window check or land the warning-discipline layer first (and gate the rest behind absent-telemetry fail-open).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions