queen: a refused key rests and the next bee takes the next key (key_failover card)

[gHashTag]

Wires specs/queen/key_failover.t27 (gHashTag/t27#9998) into the live Queen dispatcher, the way dispatch_exit is wired. Tracking issue: gHashTag/t27#10403 (a closing keyword cannot work across repos and this PR does not target the default branch, so the issue is closed from the t27 evidence PR).

What it does

• drain (queen-dispatch.ts): a 429, 5xx or Z.ai 1302/1305 stream error closes the row as provider overloaded (http N | zai code N), chosen by the card's error_kind. The label carries no response body and no credential.
• dispatchBee: a key whose closed rows are still inside cooldown_seconds (15 s doubled per strike, cap 240 s; quota 1800 s) counts as full, so availableKeyIndex skips it and the bee takes the next key. The reading of the closed rows is restingKeys in queen-contributor-keys.ts.
• beeEndOfRow (actors path): the new label ends as EF_PROVIDER, so a key refusal is not charged to the issue.
• The Z.ai quota-code Set is deleted; classifyQuotaExhaustion asks the card's is_quota_code. The 14-code test keeps its assertions and adds 1302/1305 as non-quota.
• Everything is behind TRIOS_QUEEN_KEY_FAILOVER=on. Off is the previous behaviour (restingKeys returns [], classifyOverload returns null).

Own-language budget

Hand-written TS: 70 added, 70 deleted (shrink-only holds), no file over 40 added lines, no new foreign file. The card, its wasm (831 bytes, two builds, one hash) and queen-key-failover-card.gen.ts are tool-written; PIN lines added. No owner-approved-foreign label is needed.

Evidence and what is not verified

Run on the Railway lab (details in the t27 evidence PR): the vendored card passes its tests, and two mutations that restore the old behaviour each fail a named test. The TypeScript wiring itself has not been executed: the lab has no bun and the repo tests are bun test, and nothing heavy runs on the Mac. CI on this PR is the first run of the changed TS.

Not in this PR

• Pool-level next-pool move (pool_overloaded).
• keyed_batch (t27#10051): admission batching exists only on the actors path (off in production) and needs a restructure, not a thin edit.
• Production env values are redacted, so whether the flag should be on in each service is the owner's call.

🤖 Generated with Claude Code

[tri-claw]

Summary by t27-bees

Checked by the t27 compiler

_{t27c parse, parse-complete and typecheck on each changed .t27 file at b7e2ccc. Nothing was generated, built or run. A pass means the file is well formed, not that it is right.}

• trios/agent-server/specs/queen/key_failover.t27: parses and typechecks.

Read by the model

This PR wires the key_failover card into the Queen dispatcher so that when a key receives a transient refusal (HTTP 429/5xx, Z.ai 1302/1305), the key rests for a cooldown (15s base, doubling per strike, cap 240s) and the next bee picks the next available key. Quota/auth errors get a longer 30-minute break. The feature is gated by TRIOS_QUEEN_KEY_FAILOVER=on.

Worth a look

• trios/agent-server/apps/server/src/api/services/queen-contributor-keys.ts:1048–1068 — restingKeys counts all provider-overloaded/quota rows in the 30-minute window as strikes (count(*) - 1), but the card's cooldown_seconds expects only consecutive transient refusals. A quota error followed by a transient error would inflate the strike count and over-cool the key.

• trios/agent-server/apps/server/src/api/services/queen-dispatch.ts:4972–4982 — dispatchBee expands each resting key into POOL_MAX (32) duplicate entries via Array.fill so resolveWorkerProvider skips them. This works only if resolveWorkerProvider deduplicates its input; if it treats the array as an ordered list of candidates, the duplicates could shift indices and cause the wrong key to be chosen.

• trios/agent-server/apps/server/src/api/services/queen-dispatch.ts:4327–4337 — classifyOverload returns null for any error_kind other than KE_TRANSIENT (quota, auth, request). Quota is handled by classifyQuotaExhaustion, but auth/request errors that aren't caught by the quota classifier will fall through with no overload label and no key rest, potentially retrying the same bad key immediately.

• trios/agent-server/apps/server/src/api/services/queen-contributor-keys.ts:1053 — The SQL uses EXTRACT(EPOCH FROM now() - finished_at)::int which truncates fractional seconds. For the 15s base cooldown, a row aged 15.9s becomes a = 15, passes a < cooldown_seconds (15), and the key rests one more cycle; a row aged 15.0s becomes a = 15, fails the check, and the key is released up to 1s early.

• trios/agent-server/apps/server/tests/api/queen-dispatch.test.ts:2158–2175 — The test now asserts classifyQuotaExhaustion returns null for 1302/1305, but the test no longer verifies that classifyOverload does return a label for those codes. The overload path is untested in TypeScript (the description notes the TS wiring has not been executed).

---

_{Read: 8 files, +520 -70, the whole diff. Head b7e2ccc. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.}

[github-actions]

✅ Tests passed — 3915/3988

Suite	Passed	Failed	Skipped
✅ agent	429/429	0	0
✅ build	9/9	0	0
✅ cdp-protocol	5/5	0	0
✅ eval	93/93	0	0
✅ server-agent	280/280	0	0
✅ server-api	2273/2345	0	72
✅ server-browser	6/6	0	0
✅ server-integration	10/11	0	1
✅ server-lib	279/279	0	0
✅ server-pglive	166/166	0	0
✅ server-root	68/68	0	0
✅ server-sim	8/8	0	0
✅ server-skills	31/31	0	0
✅ server-tools	244/244	0	0
✅ shared	14/14	0	0

View workflow run