Repository navigation
EPIC: best-in-industry dispatch -- why 32 of 70 bees idle with a full backlog, and the roadmap to fix it #9976
Copy link
Copy link
Closed as not planned
Closed as not planned
Copy link
Labels
shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens itClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it
Description
The owner asked for a roadmap that makes the Queen's dispatch the best in the industry, and for the reason the bees stand idle while the backlog is full. This epic answers both. It is a tracking issue: it has no Boundary on purpose and is never dispatched itself.
Where we stand (measured 2026-10-17 14:37Z, /queen/status)
| Metric | Now | Target |
|---|---|---|
| Lanes busy | 38 / 70 (32 idle) | idle < 5% while eligible work exists |
| Tick result | "nothing to choose": 826 of 826 open issues skipped | > 0 eligible whenever the backlog has work |
| Skips | completed 508, fileConflict 166, missingBoundary 79, claimed 70, incompleteSpec 3 | completed 0 (closed), fileConflict < 20 |
| Dispatched / finished / reviewed per hour | 45 / 21 / 24 | review >= finish rate |
| Unreviewed | 206 | < 30 |
| Finished dispatches that ended "unexpectedly (cause undetermined)" | 7 of 9 at 14:05Z, 1 of 11 at 14:37Z | < 2%, and every exit named |
| Bee PRs since 2026-10-10 | 115 merged, 41 open, 4 closed | |
| Bee PR lead time | p50 1.6 h, p90 49.6 h | p90 < 6 h |
Why the bees stand idle: the work exists, but it does not reach the dispatcher as eligible issues.
- Hot files. A few foreign files (
bootstrap/src/compiler.rs,bootstrap/src/main.rs,scripts/tri_loop/t27b.py,cli/tri/src/main.rs) sit in the Boundary of dozens of issues, so one held issue shuts out the rest: fileConflict 166. - The roadmap is not in the queue. Epics are not split into Queen-shaped sub-issues, so they count as missingBoundary (79), and the planner does not read the roadmap (Queen roadmap from one .t27 source, live from the epics, read by the planner; plus self-diagnosis rules #9022).
- Done work stays open. 508 issues are complete but open, and are re-skipped every tick.
- Master is red on duplicate-bodies, cli-tri and t27b-native-ratchet, so no bee PR lands (duplicate-bodies: share tree helpers across eight gen_w bench ports #9633, cli-tri: retire deleted-workflow test and refresh measured census pins #9634 / PR policy: record approval for the 19-line gates test deletion (Refs #9634) #9639, t27b-native ratchet is red on every master run since it landed: ledger 099ac2224 predates 16 improvements and 40+ new specs #9600 / PR t27b: repair crypto and rust_convert; distinguish PR ledger lag #9637).
- Contracts are not wired. Most fixes already exist as
specs/queen/*.t27cards, but the runtime (gHashTag/trios) does not run them yet, and hand-written TypeScript needs the owner label.
Phase 0 -- unstarve the swarm (days)
- Master green: duplicate-bodies: share tree helpers across eight gen_w bench ports #9633 (duplicate-bodies, pure
.t27, in progress), PR policy: record approval for the 19-line gates test deletion (Refs #9634) #9639 (cli-tri, owner label), PR t27b: repair crypto and rust_convert; distinguish PR ledger lag #9637 (t27b ratchet, owner label) - Give the missingBoundary issues a Boundary or mark them as epics (in progress)
- queen: reconcile issue state from PR linkage -- completed-but-open issues close one at a time, with evidence #9968 reconcile: close completed issues one at a time, with the PR as evidence
- queen: a hot file is named and split -- a path that shuts out more than N issues becomes a finding with a split plan #9975 hot files: name them, split them, and stop holding paths that bees may not edit by hand
- t27b ts: emit the Queen's rules as TypeScript with function bodies, from .t27 (supersedes the blocked gen-ts PR #7872) #9994
t27b ts: the bridge. It emits the cards as TypeScript with function bodies, from.t27(ts_emit.t27/ts_plan.t27, the shape oft27b verilog). It supersedes gen-ts: lower the bodies of pure functions, so a host imports a spec's rule instead of copying it #7871 / PR gen-ts: lower the bodies of pure functions #7872, which the own-language gate blocks for good (441 Rust lines against a 40/80 limit that no label lifts).
Phase 1 -- flow (Kanban, Little's law, merge queues)
- queen: service levels with an error budget -- written targets, and dispatch slows when review falls behind #9974 SLOs and error budget; dispatch follows review capacity
- queen: event-driven requeue -- a skipped issue wakes when its blocker releases, not by rescanning 826 issues every tick #9967 event-driven requeue (Kubernetes QueueingHint pattern)
- queen: a merge queue that batches green PRs and bisects a red batch, so landing keeps up with finishing #9973 merge queue: batch and bisect (Uber SubmitQueue, GitHub merge queue, Google TAP)
- Queen wiring: one chooser call per wake fills every free slot (offers_per_wake, resident queend) trios#1710 / feat: extend IGLA RACE cross-target to systolic PE + gen-backend findings #1881: one chooser call fills every free lane (
TRIOS_QUEEN_DISPATCH_BATCH, merged, flag state not visible) - Queen wiring: reviewer concurrency from the spec instead of a fixed 4 trios#1711: reviewer concurrency from
specs/queen/reviewer_sizing.t27 - Queen wiring: file owner actor admits disjoint edits instead of the fileConflict skip trios#1709: a file-owner actor admits disjoint edits instead of the fileConflict skip
- queen: wire conflict_step into the publisher (update-branch / redo conflicting bee PRs) #7573: publisher conflict step (update-branch, redo conflicting bee PRs)
Phase 2 -- reliability (Temporal, BEAM)
- queen: bee key failover -- on 429/1305 the next attempt goes to another key, then the other pool, instead of retrying the same key until the bee dies #9993 key failover: on 429/1302/1305 a bee moves to the next key, then the other pool (17 of 93 bee turns died after 0 steps on 2026-10-17)
- Queen wiring: record the exit reason of every dispatch in the activity event and run the spec's conformance rows trios#1708: record every dispatch's exit reason (
specs/queen/dispatch_exit.t27, 24 conformance rows) - queen: a runner bee that committed nothing is an empty attempt after the grace, not a 6-hour wait (reviewer.t27 no_bundle_is_empty, behind TRIOS_QUEEN_NO_BUNDLE_EMPTY) trios#1886: a bee that committed nothing is an empty attempt after the grace, not a 6-hour wait
- Queen: a redispatch after a spent ceiling should get a full set of send-backs (option (a) of #1769, depends on t27#8768) trios#1806 + queen: preserve bee worktrees and bound retries with a standalone policy card #9552: a retry policy instead of the retry-ceiling lock (
tools/queen/refile.pyre-files today) - Queen dispatches issues that already have an open PR closing them (4 duplicates on 2026-10-10) trios#1736: never dispatch an issue that already has an open PR closing it
- Queen deploy drain almost stops dispatch for ~30 min (rounds gated during drain) trios#1705: a deploy drain does not stop dispatch for 30 minutes
- queen: review rounds fail to read the diff of finished work (1105 lines in 46 min) #6116: the review round reads the diff of finished work
Phase 3 -- intelligence (where agent schedulers are going)
- queen: order work by value -- cost of delay over size (WSJF) and the number of issues it unblocks #9971 value: WSJF plus how many issues a task unblocks (critical path)
- queen: route each task to the cheapest tier that can finish it, and escalate one tier on send-back #9972 routing: the cheapest tier that can finish the task, one tier up per send-back (FrugalGPT, RouteLLM)
- queen: idle lanes run speculative attempts -- a second bee on the highest-value long-running issue instead of standing idle #9969 speculative attempts on idle lanes: best-of-N and backup tasks for stragglers (MapReduce, Codex, Devin)
- Queen roadmap from one .t27 source, live from the epics, read by the planner; plus self-diagnosis rules #9022: the roadmap as one
.t27source the planner reads
How a card lands
Each new card above is Queen-shaped, with a Boundary of one new spec file and its seal, so the cards never conflict with each other and bees can take all eight at once. The bar is specs/queen/dispatch.t27 in PR #9961: 0 surviving mutants, 0 vacuous tests. Wiring follows through t27b ts (#9994), not hand-written TypeScript.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens itClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it
Activity
github-actions commentedon Oct 17, 2026
Closed by the issue-shape gate (specs/policy/issue_shape.t27, #7498): an issue must have the template's sections. Edit the body; an edit that passes reopens it.
- specs/x.t27); without it the Queen reserves nothing and no bee can take the issueFR-001: ... MUST ...exit 0, or a backticked commandgHashTag commentedon Oct 17, 2026
Measured right after filing (2026-10-17 14:49-14:54Z): the Queen took the new cards within about 3 minutes, which confirms the swarm is starved for eligible issues rather than slowed by the dispatcher. #9967, #9969, #9968 and #9974 were dispatched, then each ended "unexpectedly (cause undetermined)" 1-2 minutes later. None pushed a
queen-Nbranch or opened a PR, and all four now sit in reviewwaitand count asclaimed. #9973 and #9975 were still running at 14:54Z.So the first blocker is not supply, it is bees that die within minutes and say nothing. Earlier today 7 of 9 finished dispatches ended the same way (14:05Z).
What this needs, in order:
specs/queen/dispatch_exit.t27has the contract and 24 conformance rows). Without it, every fix is a guess.specs/queen/holds.t27). Today it holds the issue inwait, so the issue is lost to every other bee.gHashTag commentedon Oct 17, 2026
Root cause for part of the idle and "cause undetermined" exits, from the runner logs on Railway (project 999, services
trios-bee-runner-1/trios-bee-runner-2, 2026-10-17 13:00-15:18Z).AI_RetryErroron z.ai429 code 1305(temporary overload), in bursts at 13:53, 13:59, 14:08 (16), 14:19, 14:47 and 14:50. Runner-1: about 6 of 48 turns ended after 0 steps.resolveWorkerProvider, round-robin) and keeps it. On 1305 the SDK retries the same key 3 times, the turn ends without a completion frame, and the claim is held. 34 keys sit in the two pools, but a bee never moves to another one.queen-tick.ts, after the 2026-09-23 incident). Bees do not.Card: #9993 (Phase 2). The exit should be named
provider_overloaded(gHashTag/trios#1708) and the claim released (gHashTag/trios#1886).How to read these logs: Railway MCP
get-logson project 999 in the shared workspace, with a filter such as@issue:9967or a quoted string like"1305". Results are capped at 200 lines per call.