Repository navigation
queen: wire key_failover and keyed_batch into the live bee dispatcher (trios queen-dispatch) #10403
Description
The Queen's key allocator is live TypeScript in gHashTag/trios (trios/agent-server, branch queen), and specs/queen/key_failover.t27 (#9998, Closes #9993: a refused key, 429 or 503, moves the bee to the next key) was not called by it. Measured on the live path: Z.ai 1302/1305 end as "ended unexpectedly (cause undetermined)", the SDK retries a 429 three times on the same key, and endpointPoolProvider hands the next bee the same overloaded key. This wires the card the way dispatch_exit is wired: the .t27 vendored byte-identical, a wasm built with t27c gen-c + zig, constants from t27c gen-ts, and a thin host edit that only carries numbers in and out.
Scope after the first pass: the per-key half is wired. The next-pool half and the keyed_batch card (#10051) are parked, with the reasons under "Follow-ups".
Boundary
- In:
trios/agent-server/specs/queen/key_failover.{t27,wasm}, theirspecs/PINlines,queen-key-failover-card.gen.ts(tool-written), and thin edits to existingqueen-dispatch.ts(drain ending,dispatchBeekey pick),queen-contributor-keys.ts(resting keys from the closed rows) andqueen-bee-actors-round.ts(beeEndOfRow). A t27 docs/now note with the lab evidence. - Out: any NEW hand-written foreign file, the card itself (spec(queen): key_failover.t27 -- a refused key moves the bee to the next key, then the next pool (Closes #9993) #9998), Railway variable changes, turning
TRIOS_QUEEN_DISPATCH=actorson in production, the pool-level next-pool move, batch admission.
Scenarios
- A turn on pool 1 key 3 ends with Z.ai code 1305 (or HTTP 429/503): the row closes with a provider-overloaded label, key 3 rests for
cooldown_seconds(strikes), and the next dispatch takes the next free key, not key 3. - A quota or auth refusal closes the key for 1800 seconds; a request error (4xx other than 429) is the issue's fault, a key refusal never is.
- The failover flag is off: the allocator picks exactly as before (the negative control).
Requirements
- FR-001: The drain ending MUST classify a Z.ai 1302/1305 or a 429/5xx stream error with
key_failover.error_kindand close the row with a provider-overloaded label that carries no response body and no credential. - FR-002:
dispatchBeeMUST treat a key inside its cooldown as fully occupied, soavailableKeyIndexskips it. - FR-003: A key refusal MUST NOT end as a failure charged to the issue (
beeEndOfRowmaps the label to the provider ending). - FR-004: The change MUST add no more foreign lines than it deletes (shrink-only), at most 40 per existing file and no new hand-written foreign file; the decisions MUST run from the vendored wasm.
Success Criteria
- The card's own tests pass on the Railway lab, and each of two mutations that restore the old behaviour (next_free may return the failed key; 429 and 1302/1305 not transient) makes a named test fail (negative controls).
specs/PINhashes match the vendored source and wasm byte for byte (two builds, one hash).- The flag off takes the previous code path (
restingKeysreturns[],classifyOverloadreturnsnull).
Follow-ups (not in this issue)
- Pool-level move: three distinct keys of one pool answering 1305 inside 60 seconds moves the next dispatch to the next provider pool (
pool_overloaded). - keyed_batch (spec(queen): keyed_batch.t27 -- the bee admission admits a batch of ready issues per queend choose (Closes #10033) #10051): admission lives only on the actors path (
TRIOS_QUEEN_DISPATCH=actors, off in production); the legacy loop already batches throughTRIOS_QUEEN_DISPATCH_BATCH. Wiring needs a restructure ofadmissionSpec.receiveand a batched choose, which is not a thin edit.
Activity
docs/now: key_failover wired into the live Queen dispatcher, with lab…