Skip to content

docs(rfc): queue and retry turns while the model provider throttles - #29430

Open
Pearcekieser wants to merge 1 commit into
mainfrom
pearcekieser/rfc-throttle-queue
Open

Pearcekieser wants to merge 1 commit into
mainfrom
pearcekieser/rfc-throttle-queue

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Main records, as an accepted RFC, the product owner's decision that a dashboard chat turn the model provider throttles waits in a queue and retries with an adaptive backoff, instead of ending on an error card, so the implementation PR has a decision on the base branch to land against.

Today a throttled chat turn re-sends 3 times on throttle_retry_delay and then ends on the throttle card, approximately 70 s after the first throttle. On one host, 99 turns ended on the throttle card in the 14 days to 2026-11-03, and 25 failures hit 18 sessions in one 13-minute window. Each session retries on its own clock, so a burst stays a burst.

Why it matters

The implementation changes what a dashboard chat turn does when the provider throttles. That is a product-shape change, and the First Principles lane reads the decision from the base commit. The decision belongs in a docs-only diff that a maintainer can read before the code.

Not a goal

What changed (motivation → approach → change)

Adds docs/request-for-change/rfc-provider-throttle-queue.md with status: accepted, naming @Pearcekieser as product owner, and its row in docs/request-for-change/README.md. The document extends rfc-overload-resilience §14.4 to dashboard chat turns. It was checked against 16ff876b5a.

The design is one in-memory gate per backend and model, with four states: open, cooling, probing and ramping. A dashboard chat turn asks the gate for admission where its session/prompt request goes out, on both ACP transports. A cooling scope holds chat turns in one queue, with full-jitter backoff and an AIMD release in arrival order. While a scope recovers, chat turns that no person typed, such as session_send deliveries, also start at no more than 2 per minute on that scope. The chat shows a queued notice with Retry now and Cancel. Each throttle row carries meta.error_kind, provider, model and scope, so tools need not match the card text. Six agent.throttle_queue_* keys configure it. The queue is off by default, and off gives today's behaviour exactly.

Every other prompt start keeps today's path with the key on. When the provider throttles one of them, the transport reports the throttle to the gate, so a cron or sub-agent throttle cools the scope that chat turns wait on. That start never waits in the gate. A chat turn waits only when Shared Crew fair admission did not admit it and its slot is attended, so a waiting turn never holds a Shared Crew admission or a background-turn permit. Stop, the turn ceiling and a responsive crewmate preempt end a wait without arming the cancelled-turn restore, and a waiting turn gives up before its whole-turn ceiling can fire. RFC §4.4 has a table of each call site that moves in v1 and each one that stays.

Review on 2026-11-05 narrowed the first draft, which gated every prompt start and had a park mode for coordinator callers. The product owner confirmed the narrowing the same day.

The design came from a search of open and closed throttle work. #28633 (attended-turn cap) is at the same dispatch point, so a turn that waits in the gate holds no attended-turn permit. #20750 (auto-resume) edits other arms of the same recovery module. #15543 (per-chat priority) is not merged, so the queue keeps a (priority, seq) key with priority 0.

A maintainer's review and merge of this PR records the acceptance.

Backwards compatibility

Compatible: documentation only. No code, config or contract changes.

Removes nothing: the diff adds one RFC file, one README row and one audit sentence.

Tests

python3 scripts/docs_lint.py passes. No code changed, so no code tests apply.

Manual verification

N/A: documentation only. I checked each claim about today's code against 16ff876b5a. That covers _TRANSIENT_RETRIES, THROTTLE_RETRY_BASE_DELAY, _shared_dependency_delay, stream_and_collect, _CRON_TRANSIENT_RETRIES, the DependencyCoordinator defaults, scope_for, the chat throttle arm in dashboard/chat_turn/acp_recovery.py and each file the §4.4 table names.

Related Issues

Implementation follows in its own PR once this merges.

Fixes #29429

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners November 4, 2026 01:45
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is docs only. It records an RFC for an off-by-default, reversible design, takes nothing away, and adds no surface that would need rendered evidence.

[DESIGN-REVIEWED] 64c22e9

Description read

[DESCRIPTION-READ] de33bc2653509cd22f6bd041ed6be4a76dd0c427d083b699bd4931103f42c5bb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 64c22e9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 64c22e9

Verdict parsed from the review's SHA-scoped output markers for commit 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 64c22e906ad22b1aaaf4e255d35ac872d8c79ab6 false-positive: <reason, 20 to 500 characters>

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the RFC. A first read of b81b524, which adds docs/request-for-change/rfc-provider-throttle-queue.md and its README row.

Design brief

  • What it does: a new in-memory ProviderThrottleGate, one record per provider:<backend>:<model> scope. A throttle opens a cool-off for that scope. Throttled chat, cron and stream_and_collect turns wait in a queue instead of failing after about 70 s, and the queue releases them in order with AIMD. The chat shows "queued, retrying at ..." with Retry now and Cancel. A turn that waits past throttle_queue_give_up_secs (2700 s) ends on today's card with Resume.
  • Pros: it targets a measured failure (107 throttle failures in 14 days on one host, 25 in one 13-minute burst). It shares the dependency coordinator's scope key and retry_at, so the two schedules agree. One key (agent.throttle_queue_enabled) restores today's paths, and a parity test pins them. Throttle rows get a structured error kind.
  • Cons: it is on by default, so every user's throttled turn changes from a fast failure to a wait of up to 45 minutes. The gate is in memory, so a restart drops the queue. The per-model scope costs one extra probe per model if throttles are really account-wide.
  • Tech debt: small. It adds one more admission point beside the coordinator, which the RFC justifies in §10, alternative 2.

Points for the decision:

  1. The front matter says status: accepted, by the author as product owner. The maintainer sign-off on this repo is still separate from that.
  2. Open questions 2 and 3 (the "selected model is unavailable" error, and channel turns on TurnDriver) change what users see, so they would be good to settle before the implementation PR.

Electron Shell Tests is red on two xattr and descriptor helper tests, which this docs-only change cannot affect. E2E is still running. I am passing the sign-off to the maintainers.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 4, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

note (non-blocking): maintainer sign-off

Agreed. The front matter records the product owner's decision only. A maintainer approval and merge of this PR records the repository's acceptance, as with #18535, and I ask for that here.

question (non-blocking): open questions 2 and 3

@Pearcekieser, as product owner, can you settle these two before the implementation PR opens? I recommend today's behaviour for both in the first version.

Question 2: the "selected model is unavailable" card also covers region rollouts and entitlement, not only capacity. Its usual fix is another model, and the card says so. A wait of up to 45 minutes would hide a model that does not come back. It was 8 of the 107 failures, and the new error_kind: model_unavailable field lets us measure it before we decide.

Question 3: a channel turn on TurnDriver gets admission, so it no longer starts into a cooling provider. A throttle still ends it with today's message. A Slack or Telegram thread has no Cancel button, so a silent wait of up to 45 minutes there is worse than a fast failure. Queueing there can follow once the dashboard path has shipped.

note (non-blocking): on by default

A queued chat turn shows the queued row with the time of the next try. Cancel is the existing Stop, and agent.throttle_queue_enabled = false restores today's path exactly. The default is the product owner's call in the RFC.

@Pearcekieser
Pearcekieser force-pushed the pearcekieser/rfc-throttle-queue branch from b81b524 to 286d4ad Compare November 4, 2026 10:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 4, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 4, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/rfc-throttle-queue branch from 286d4ad to 162ed4c Compare November 4, 2026 12:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 4, 2026
@Pearcekieser
Pearcekieser enabled auto-merge (squash) November 4, 2026 13:41

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 162ed4c (comment only, no sign-off on the full scope). The gate shows 96 checks passed and none red or pending.

The direction is good. A throttle that today ends a chat turn on an error card after about 70 seconds, with every session retrying on its own clock, is a real problem, and one shared schedule per scope with gradual release is a sound way to fix it. I read the whole document. The scope in this version is much wider than the problem it measures, so I would like it narrowed before it is accepted.

  1. Narrow v1 to dashboard chat turns. The 107 failures counted in section 1 are chat turns that ended on the throttle card. Section 4.1 puts admission under every prompt send (the chat runner, stream_and_collect, TurnDriver, sub-agents, TaskRunner steps, workflow agents, compaction, background one-liners, the auto-improvement runner and the code-review pool), and sections 4.4 and 4.5 add the park mode and the injected-start limit. Each extra path is a place where a start can now wait where it used to fail or park. Please keep the gate for the chat runner only in v1, and leave the other callers on today's behavior.

  2. Keep today's error card as the default. Section 7 sets agent.throttle_queue_enabled to true, so every existing user's throttled turn would start waiting instead of failing on the next release. Please ship the queue behind the setting with the default off (today's behavior exactly), so people opt in, and revisit the default after the evidence in section 9 exists.

  3. List the call sites that move in v1. Please add a table to the RFC naming each code path that calls the gate in v1 and each one that stays as it is today, so the implementation PR can be checked against it. With point 1 applied, I expect the list to be the chat runner's throttle arm and the admission point at dispatch.

With these changes I am happy to review again. I am not signing off on the full scope as written.

@Pearcekieser
Pearcekieser disabled auto-merge November 5, 2026 01:28
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 5, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author

Could a maintainer rerun the failed jobs in runs 42126420780, 42126421072, 42126420799, 42126420840, 42126420864, 42126416097, 42126420829 and 42126420950 on 724deb1? All eight failed on the CI app's installation rate limit before reaching a verdict.

@bolichen97

Copy link
Copy Markdown
Contributor

Signed: v1 scope as narrowed, default off, the other paths unchanged.

That is, at 724deb1 only the dashboard chat runner's send waits in the gate, the queue is off by default, and cron turns, the other stream_and_collect callers, TurnDriver channel turns, sub-agents and the coordinator's park keep today's behavior. This is a comment, not an approval.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 5, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/rfc-throttle-queue branch from 724deb1 to c2062d2 Compare November 5, 2026 23:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 5, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/rfc-throttle-queue branch from c2062d2 to 316e57a Compare November 7, 2026 01:01
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Nov 7, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 316e57a (comment only; an RFC sign-off stays with the maintainers).

Thanks for narrowing it. Compared with 162ed4c, all three points from my last review are in:

  1. v1 gates dashboard chat turns only. Goals 1 and 2 and the new non-goal limit the wait to turns the chat runner marks. Every other prompt start is report-only: it reports a throttle to the gate and keeps its own re-sends, timeouts and errors.
  2. Off by default. agent.throttle_queue_enabled is false (§7, §8), and open question 3 keeps the default change for later, once the §9 evidence exists.
  3. Call-site table. §4.4 names each path that moves in v1 and each one that stays, and says an unnamed caller is report-only, with a test pinning that an unmarked send never waits.

Design brief

  • What it does: with the key on, a dashboard chat turn that the provider throttles waits in a per-provider:<backend>:<model> queue and is re-sent from it, instead of ending on the throttle card after its fixed re-sends. Release is AIMD in arrival order. The chat shows "queued, retrying at ..." with Retry now (owner only, through require_owner_dashboard_request) and Cancel. The give-up window ends a waiter on today's card with Resume.
  • Pros: it targets the measured chat failures in §1. It is opt-in, so nothing changes for anyone who does not turn it on. Other callers only feed throttle reports in, so chat turns stop starting into a scope that a cron burst just found throttled. Throttle rows get a structured error kind either way (goal 7).
  • Cons: the gate is in memory, so a restart drops the queue. Report-only throttles from other callers can open a cool-off that holds chat turns while those callers keep their own retries.
  • Tech debt: small. One more admission point beside the dependency coordinator, now limited to one caller. Open questions 4 and 5 list what a later change would move.

The new Retry now route uses the existing owner check of the dashboard routes, so it adds no new restriction.

No further asks from me on the v1 scope.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (ci) at 316e57a.

One check is red on this head. Dependency License Gate (run 42321676132) failed twice with "Dependency review is not supported on this repository". That was a GitHub dependency-graph outage, and the same gate passed on other PRs later that day. This PR is docs only, so a re-run of that job should clear it. The required PR Readiness status is already green (06:32Z), so nothing in protected merge waits on this check. A real failure on the head still blocks approval until it is re-run.

The RFC itself is ready. The file is unchanged since 724deb1, the scope signed on 2026-11-05. v1 gates dashboard chat turns only, the queue is off by default (§8), and the call-site table names every path that stays report-only (§4.4). The code facts I spot-checked hold at this head. Design, Opus and GPT 6.1 report no findings.

— crew-pr-reviewer (r149)

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 7, 2026
@Pearcekieser
Pearcekieser force-pushed the pearcekieser/rfc-throttle-queue branch from 316e57a to 64c22e9 Compare November 8, 2026 01:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 8, 2026
@Pearcekieser
Pearcekieser enabled auto-merge (squash) November 8, 2026 02:33
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here.

To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/).

Review result on this head: no blocking findings. The diff is docs only: one new RFC and one README index row. No other open PR changes these files. It merges cleanly onto main. We checked the RFC's claims about today's code against main and they hold. These include the retry constants, scope_for and its provider:acp default, the CancelOutcome literal, RESERVED_ROW_META_KEYS, and the owner check on the proposed Retry now route. We ran python3 scripts/docs_lint.py on this head and it passed. One thing done well: the §4.4 call-site table names every prompt path and makes any unnamed caller report-only, so the implementation PR gets a concrete checklist.

Optional notes, for this RFC or the implementation PR:

  • Optional: The default Kiro CLI backend's runtime id is the empty string (ACP_BACKEND_KIRO = "" in backends.py). Read literally, §4.1 and the §6 table then write meta.provider = "" and the scope provider::<model> for the most common harness. You can say that the scope and meta.provider use the stored spelling from backend_to_stored() (kiro).
  • Optional: §4.5 says only chat send, Resume and Retry now are typed. On main, the owner's Regenerate and Rewind also pass _directive_user_origin=True, so the mechanical definition already covers them. The name "typed bit" is also close to the existing _message_is_typed kwarg in _run_chat. A name like owner_origin avoids that confusion.
  • Optional: The injected-start limit in §4.5 (goal 8, the per-minute config key, held_by, §9 item 6) is a second pacing layer on top of the AIMD window. The Goal holds without it. If the maintainer prefers a smaller v1, deleting it and releasing all chat waiters through the window in arrival order is the Subtraction.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs(rfc): queue and retry turns while the model provider throttles

4 participants