Skip to content

queen: wire key_failover and keyed_batch into the live bee dispatcher (trios queen-dispatch) #10403

Closed

Description

@gHashTag

The Queen's key allocator is live TypeScript in gHashTag/trios (trios/agent-server, branch queen), and specs/queen/key_failover.t27 (#9998, Closes #9993: a refused key, 429 or 503, moves the bee to the next key) was not called by it. Measured on the live path: Z.ai 1302/1305 end as "ended unexpectedly (cause undetermined)", the SDK retries a 429 three times on the same key, and endpointPoolProvider hands the next bee the same overloaded key. This wires the card the way dispatch_exit is wired: the .t27 vendored byte-identical, a wasm built with t27c gen-c + zig, constants from t27c gen-ts, and a thin host edit that only carries numbers in and out.

Scope after the first pass: the per-key half is wired. The next-pool half and the keyed_batch card (#10051) are parked, with the reasons under "Follow-ups".

Boundary

  • In: trios/agent-server/specs/queen/key_failover.{t27,wasm}, their specs/PIN lines, queen-key-failover-card.gen.ts (tool-written), and thin edits to existing queen-dispatch.ts (drain ending, dispatchBee key pick), queen-contributor-keys.ts (resting keys from the closed rows) and queen-bee-actors-round.ts (beeEndOfRow). A t27 docs/now note with the lab evidence.
  • Out: any NEW hand-written foreign file, the card itself (spec(queen): key_failover.t27 -- a refused key moves the bee to the next key, then the next pool (Closes #9993) #9998), Railway variable changes, turning TRIOS_QUEEN_DISPATCH=actors on in production, the pool-level next-pool move, batch admission.

Scenarios

  • A turn on pool 1 key 3 ends with Z.ai code 1305 (or HTTP 429/503): the row closes with a provider-overloaded label, key 3 rests for cooldown_seconds(strikes), and the next dispatch takes the next free key, not key 3.
  • A quota or auth refusal closes the key for 1800 seconds; a request error (4xx other than 429) is the issue's fault, a key refusal never is.
  • The failover flag is off: the allocator picks exactly as before (the negative control).

Requirements

  • FR-001: The drain ending MUST classify a Z.ai 1302/1305 or a 429/5xx stream error with key_failover.error_kind and close the row with a provider-overloaded label that carries no response body and no credential.
  • FR-002: dispatchBee MUST treat a key inside its cooldown as fully occupied, so availableKeyIndex skips it.
  • FR-003: A key refusal MUST NOT end as a failure charged to the issue (beeEndOfRow maps the label to the provider ending).
  • FR-004: The change MUST add no more foreign lines than it deletes (shrink-only), at most 40 per existing file and no new hand-written foreign file; the decisions MUST run from the vendored wasm.

Success Criteria

  • The card's own tests pass on the Railway lab, and each of two mutations that restore the old behaviour (next_free may return the failed key; 429 and 1302/1305 not transient) makes a named test fail (negative controls).
  • specs/PIN hashes match the vendored source and wasm byte for byte (two builds, one hash).
  • The flag off takes the previous code path (restingKeys returns [], classifyOverload returns null).

Follow-ups (not in this issue)

Activity

  1. added a commit that references this issue on Oct 18, 2026
    25f0b15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions