queen: bee key failover -- on 429/1305 the next attempt goes to another key, then the other pool, instead of retrying the same key until the bee dies #9993

Part of the dispatch roadmap epic #9976 (Phase 2, reliability). Industry reference and the measured gap are below.

Measured 2026-10-17 13:00-15:18Z in the Railway logs of trios-bee-runner-2:

• 93 bee turn profiles, 17 of them ended after 0 steps.
• At least 22 AI_RetryError on z.ai 429 {"code":"1305"} (temporary overload) in the last 200 error rows. They came in bursts at 13:53, 13:59, 14:08 (16 in one minute), 14:19, 14:47 and 14:50.
• trios-bee-runner-1 had about 6 of 48 turns end after 0 steps. NVIDIA, the reviewer's provider, answered 503 and 429 in the same window.

Example: the bee for #9967 started at 14:50:47 on pool-2 key 9/10 and died at 14:50:54. The AI SDK retried the same key 3 times, the turn closed "without a completion frame", and the Queen recorded the exit as "cause undetermined" and kept the claim.

The cause is that rotation happens only once per bee. resolveWorkerProvider (trios queen-dispatch.ts) picks a key round-robin when the bee is dispatched, and the bee is pinned to that key for its whole life. On 1305/1302/429 the request is retried on the same key, so 34 configured keys (24 in pool 1, 10 in pool 2) do not help a bee whose one key is overloaded. isTerminalProviderError correctly keeps 1305 retryable, but "retryable" today means "same key again".

The reviewer already does this right: queen-tick.ts ("a transient refusal tries the next lane too") moves to the next lane on 429/1302/1305/503/529, after the 2026-09-23 incident where it lost 16 of 16 calls to ZAI 1302/1305 while 15 NVIDIA credentials sat unused. Bee turns have no such failover.

Industry reference: client-side load balancing with outlier ejection. Envoy and gRPC retry a failed request on a different host (retry_host_predicate: previous_hosts), eject a host on consecutive 5xx for a short base time that grows on repeat, and fail over to the next priority level when a whole pool is unhealthy. LiteLLM's router does the same for LLM deployments: on 429 it moves to the next deployment, puts the failing one on a cooldown of seconds, and falls back to the next model group.

User Scenarios

• As a bee whose request gets 429/1305 on key 9/10, my next attempt goes to a different free key in the same pool, not to key 9/10 again, and my turn continues.
• As a bee when several distinct keys of one pool answer 1305 within a short window, I treat it as an overload of the pool's model (1305 is model-wide, not per key) and move to the other pool or the next model candidate instead of burning every key in the pool.
• As the Queen, a key that answered a transient error rests for seconds, not the 30-minute quota break. A key closes for the long break only on the quota-exhausted codes (1113, 1308-1321, 402).
• As the board, a bee that ran out of lanes ends with the named exit provider_overloaded (trios#1708), its claim is released (trios#1886), and it is never counted as "cause undetermined".

Requirements

• error_kind(status, zai_code) -> u8 with named kinds: transient (429 with 1302/1305, 503, 529, timeout), quota (402, 1113, 1308-1321), other.
• next_lane(error_kind, tried_mask, pool, pool_size, cursor) -> i32: the next untried, not-cooling key in the pool, or a value that means "go to the next pool". On a transient error it never returns the key that just failed.
• pool_overloaded(distinct_keys_1305, window_seconds) -> bool, with the threshold and the window as named constants and the reason in a comment.
• cooldown_seconds(error_kind, strikes) -> u32: seconds for transient errors, capped and growing with consecutive strikes. The quota break is a separate constant. Transient errors never get the quota break.
• max_attempts_per_turn as a named constant: a bound on the total attempts, so failover cannot spin.
• Write the measured numbers above into the header comment, as specs/queen/holds.t27 does.

Success Criteria

• Tests:
  • 1305 on key k picks a key other than k.
  • 1305 on 3 distinct keys in the window moves to the other pool.
  • 1113 closes the key for the quota break.
  • 1302 never closes a key for the quota break.
  • All keys tried ends in provider_overloaded.
  • Cooldown is capped.
• Invariants:
  • A key is never tried twice in a row after a transient error.
  • A transient error never retires work or closes a key for the quota break.
  • Attempts per turn stay at or below max_attempts_per_turn.
• 0 surviving mutants, 0 vacuous tests.

Rules for the bee

• Only .t27 is hand-written. The seal is tool-written (t27c seal --save after a fresh release build on the Railway lab). No .py, .rs, .ts, .sh or .yml edits.
• Every rule gets test blocks and at least one invariant. Run tri mutate spec on the new file (on the lab) and kill every surviving mutant. The bar is specs/queen/dispatch.t27 in PR test(queen): kill the 39 surviving mutants of specs/queen/dispatch.t27 #9961: 0 survivors, 0 vacuous tests.
• Wiring into the runtime (gHashTag/trios queen-dispatch.ts, the bee turn loop) is a separate step. It lands as generated code once gen-ts lowers pure function bodies (gen-ts: lower the bodies of pure functions, so a host imports a spec's rule instead of copying it #7871 / PR gen-ts: lower the bodies of pure functions #7872). Do not hand-write TypeScript here.

Boundary

• specs/queen/key_failover.t27
• .trinity/seals/queen_QueenKeyFailover.json