Repository navigation
docs(rfc): queue and retry turns while the model provider throttles - #29430
Pearcekieser wants to merge 1 commit into
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change is docs only. It records an RFC for an off-by-default, reversible design, takes nothing away, and adds no surface that would need rendered evidence. [DESIGN-REVIEWED] 64c22e9 Description read[DESCRIPTION-READ] de33bc2653509cd22f6bd041ed6be4a76dd0c427d083b699bd4931103f42c5bb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for the RFC. A first read of b81b524, which adds docs/request-for-change/rfc-provider-throttle-queue.md and its README row.
Design brief
- What it does: a new in-memory
ProviderThrottleGate, one record perprovider:<backend>:<model>scope. A throttle opens a cool-off for that scope. Throttled chat, cron andstream_and_collectturns wait in a queue instead of failing after about 70 s, and the queue releases them in order with AIMD. The chat shows "queued, retrying at ..." with Retry now and Cancel. A turn that waits pastthrottle_queue_give_up_secs(2700 s) ends on today's card with Resume. - Pros: it targets a measured failure (107 throttle failures in 14 days on one host, 25 in one 13-minute burst). It shares the dependency coordinator's scope key and
retry_at, so the two schedules agree. One key (agent.throttle_queue_enabled) restores today's paths, and a parity test pins them. Throttle rows get a structured error kind. - Cons: it is on by default, so every user's throttled turn changes from a fast failure to a wait of up to 45 minutes. The gate is in memory, so a restart drops the queue. The per-model scope costs one extra probe per model if throttles are really account-wide.
- Tech debt: small. It adds one more admission point beside the coordinator, which the RFC justifies in §10, alternative 2.
Points for the decision:
- The front matter says
status: accepted, by the author as product owner. The maintainer sign-off on this repo is still separate from that. - Open questions 2 and 3 (the "selected model is unavailable" error, and channel turns on
TurnDriver) change what users see, so they would be good to settle before the implementation PR.
Electron Shell Tests is red on two xattr and descriptor helper tests, which this docs-only change cannot affect. E2E is still running. I am passing the sign-off to the maintainers.
|
note (non-blocking): maintainer sign-off Agreed. The front matter records the product owner's decision only. A maintainer approval and merge of this PR records the repository's acceptance, as with #18535, and I ask for that here. question (non-blocking): open questions 2 and 3 @Pearcekieser, as product owner, can you settle these two before the implementation PR opens? I recommend today's behaviour for both in the first version. Question 2: the "selected model is unavailable" card also covers region rollouts and entitlement, not only capacity. Its usual fix is another model, and the card says so. A wait of up to 45 minutes would hide a model that does not come back. It was 8 of the 107 failures, and the new Question 3: a channel turn on note (non-blocking): on by default A queued chat turn shows the queued row with the time of the next try. Cancel is the existing Stop, and |
b81b524 to
286d4ad
Compare
286d4ad to
162ed4c
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed head 162ed4c (comment only, no sign-off on the full scope). The gate shows 96 checks passed and none red or pending.
The direction is good. A throttle that today ends a chat turn on an error card after about 70 seconds, with every session retrying on its own clock, is a real problem, and one shared schedule per scope with gradual release is a sound way to fix it. I read the whole document. The scope in this version is much wider than the problem it measures, so I would like it narrowed before it is accepted.
-
Narrow v1 to dashboard chat turns. The 107 failures counted in section 1 are chat turns that ended on the throttle card. Section 4.1 puts admission under every prompt send (the chat runner,
stream_and_collect,TurnDriver, sub-agents, TaskRunner steps, workflow agents, compaction, background one-liners, the auto-improvement runner and the code-review pool), and sections 4.4 and 4.5 add the park mode and the injected-start limit. Each extra path is a place where a start can now wait where it used to fail or park. Please keep the gate for the chat runner only in v1, and leave the other callers on today's behavior. -
Keep today's error card as the default. Section 7 sets
agent.throttle_queue_enabledtotrue, so every existing user's throttled turn would start waiting instead of failing on the next release. Please ship the queue behind the setting with the default off (today's behavior exactly), so people opt in, and revisit the default after the evidence in section 9 exists. -
List the call sites that move in v1. Please add a table to the RFC naming each code path that calls the gate in v1 and each one that stays as it is today, so the implementation PR can be checked against it. With point 1 applied, I expect the list to be the chat runner's throttle arm and the admission point at dispatch.
With these changes I am happy to review again. I am not signing off on the full scope as written.
|
Could a maintainer rerun the failed jobs in runs 42126420780, 42126421072, 42126420799, 42126420840, 42126420864, 42126416097, 42126420829 and 42126420950 on 724deb1? All eight failed on the CI app's installation rate limit before reaching a verdict. |
|
Signed: v1 scope as narrowed, default off, the other paths unchanged. That is, at 724deb1 only the dashboard chat runner's send waits in the gate, the queue is off by default, and cron turns, the other |
724deb1 to
c2062d2
Compare
c2062d2 to
316e57a
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed head 316e57a (comment only; an RFC sign-off stays with the maintainers).
Thanks for narrowing it. Compared with 162ed4c, all three points from my last review are in:
- v1 gates dashboard chat turns only. Goals 1 and 2 and the new non-goal limit the wait to turns the chat runner marks. Every other prompt start is report-only: it reports a throttle to the gate and keeps its own re-sends, timeouts and errors.
- Off by default.
agent.throttle_queue_enabledisfalse(§7, §8), and open question 3 keeps the default change for later, once the §9 evidence exists. - Call-site table. §4.4 names each path that moves in v1 and each one that stays, and says an unnamed caller is report-only, with a test pinning that an unmarked send never waits.
Design brief
- What it does: with the key on, a dashboard chat turn that the provider throttles waits in a per-
provider:<backend>:<model>queue and is re-sent from it, instead of ending on the throttle card after its fixed re-sends. Release is AIMD in arrival order. The chat shows "queued, retrying at ..." with Retry now (owner only, throughrequire_owner_dashboard_request) and Cancel. The give-up window ends a waiter on today's card with Resume. - Pros: it targets the measured chat failures in §1. It is opt-in, so nothing changes for anyone who does not turn it on. Other callers only feed throttle reports in, so chat turns stop starting into a scope that a cron burst just found throttled. Throttle rows get a structured error kind either way (goal 7).
- Cons: the gate is in memory, so a restart drops the queue. Report-only throttles from other callers can open a cool-off that holds chat turns while those callers keep their own retries.
- Tech debt: small. One more admission point beside the dependency coordinator, now limited to one caller. Open questions 4 and 5 list what a later change would move.
The new Retry now route uses the existing owner check of the dashboard routes, so it adds no new restriction.
No further asks from me on the v1 scope.
|
HOLD (ci) at 316e57a. One check is red on this head. The RFC itself is ready. The file is unchanged since 724deb1, the scope signed on 2026-11-05. v1 gates dashboard chat turns only, the queue is off by default (§8), and the call-site table names every path that stays report-only (§4.4). The code facts I spot-checked hold at this head. Design, Opus and GPT 6.1 report no findings. — crew-pr-reviewer (r149) |
316e57a to
64c22e9
Compare
|
This changes behaviour users already rely on, so it needs a design sign-off from a maintainer before it can merge. A maintainer will post the decision here. To propose a design, follow the request-for-change process in CONTRIBUTING.md (docs/request-for-change/). Review result on this head: no blocking findings. The diff is docs only: one new RFC and one README index row. No other open PR changes these files. It merges cleanly onto main. We checked the RFC's claims about today's code against main and they hold. These include the retry constants, Optional notes, for this RFC or the implementation PR:
|
Problem / Motivation
Goal: Main records, as an accepted RFC, the product owner's decision that a dashboard chat turn the model provider throttles waits in a queue and retries with an adaptive backoff, instead of ending on an error card, so the implementation PR has a decision on the base branch to land against.
Today a throttled chat turn re-sends 3 times on
throttle_retry_delayand then ends on the throttle card, approximately 70 s after the first throttle. On one host, 99 turns ended on the throttle card in the 14 days to 2026-11-03, and 25 failures hit 18 sessions in one 13-minute window. Each session retries on its own clock, so a burst stays a burst.Why it matters
The implementation changes what a dashboard chat turn does when the provider throttles. That is a product-shape change, and the First Principles lane reads the decision from the base commit. The decision belongs in a docs-only diff that a maintainer can read before the code.
Not a goal
agent.throttle_queue_enabled = false, and the default is revisited after the RFC's §9 evidence exists.What changed (motivation → approach → change)
Adds
docs/request-for-change/rfc-provider-throttle-queue.mdwithstatus: accepted, naming @Pearcekieser as product owner, and its row indocs/request-for-change/README.md. The document extendsrfc-overload-resilience§14.4 to dashboard chat turns. It was checked against16ff876b5a.The design is one in-memory gate per backend and model, with four states: open, cooling, probing and ramping. A dashboard chat turn asks the gate for admission where its
session/promptrequest goes out, on both ACP transports. A cooling scope holds chat turns in one queue, with full-jitter backoff and an AIMD release in arrival order. While a scope recovers, chat turns that no person typed, such assession_senddeliveries, also start at no more than 2 per minute on that scope. The chat shows a queued notice with Retry now and Cancel. Each throttle row carriesmeta.error_kind,provider,modelandscope, so tools need not match the card text. Sixagent.throttle_queue_*keys configure it. The queue is off by default, and off gives today's behaviour exactly.Every other prompt start keeps today's path with the key on. When the provider throttles one of them, the transport reports the throttle to the gate, so a cron or sub-agent throttle cools the scope that chat turns wait on. That start never waits in the gate. A chat turn waits only when Shared Crew fair admission did not admit it and its slot is attended, so a waiting turn never holds a Shared Crew admission or a background-turn permit. Stop, the turn ceiling and a responsive crewmate preempt end a wait without arming the cancelled-turn restore, and a waiting turn gives up before its whole-turn ceiling can fire. RFC §4.4 has a table of each call site that moves in v1 and each one that stays.
Review on 2026-11-05 narrowed the first draft, which gated every prompt start and had a park mode for coordinator callers. The product owner confirmed the narrowing the same day.
The design came from a search of open and closed throttle work. #28633 (attended-turn cap) is at the same dispatch point, so a turn that waits in the gate holds no attended-turn permit. #20750 (auto-resume) edits other arms of the same recovery module. #15543 (per-chat priority) is not merged, so the queue keeps a
(priority, seq)key with priority 0.A maintainer's review and merge of this PR records the acceptance.
Backwards compatibility
Compatible: documentation only. No code, config or contract changes.
Removes nothing: the diff adds one RFC file, one README row and one audit sentence.
Tests
python3 scripts/docs_lint.pypasses. No code changed, so no code tests apply.Manual verification
N/A: documentation only. I checked each claim about today's code against
16ff876b5a. That covers_TRANSIENT_RETRIES,THROTTLE_RETRY_BASE_DELAY,_shared_dependency_delay,stream_and_collect,_CRON_TRANSIENT_RETRIES, theDependencyCoordinatordefaults,scope_for, the chat throttle arm indashboard/chat_turn/acp_recovery.pyand each file the §4.4 table names.Related Issues
Implementation follows in its own PR once this merges.
Fixes #29429
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)