Skip to content

EPIC: best-in-industry dispatch -- why 32 of 70 bees idle with a full backlog, and the roadmap to fix it #9976

Closed as not planned

Description

@gHashTag

The owner asked for a roadmap that makes the Queen's dispatch the best in the industry, and for the reason the bees stand idle while the backlog is full. This epic answers both. It is a tracking issue: it has no Boundary on purpose and is never dispatched itself.

Where we stand (measured 2026-10-17 14:37Z, /queen/status)

Metric Now Target
Lanes busy 38 / 70 (32 idle) idle < 5% while eligible work exists
Tick result "nothing to choose": 826 of 826 open issues skipped > 0 eligible whenever the backlog has work
Skips completed 508, fileConflict 166, missingBoundary 79, claimed 70, incompleteSpec 3 completed 0 (closed), fileConflict < 20
Dispatched / finished / reviewed per hour 45 / 21 / 24 review >= finish rate
Unreviewed 206 < 30
Finished dispatches that ended "unexpectedly (cause undetermined)" 7 of 9 at 14:05Z, 1 of 11 at 14:37Z < 2%, and every exit named
Bee PRs since 2026-10-10 115 merged, 41 open, 4 closed
Bee PR lead time p50 1.6 h, p90 49.6 h p90 < 6 h

Why the bees stand idle: the work exists, but it does not reach the dispatcher as eligible issues.

  1. Hot files. A few foreign files (bootstrap/src/compiler.rs, bootstrap/src/main.rs, scripts/tri_loop/t27b.py, cli/tri/src/main.rs) sit in the Boundary of dozens of issues, so one held issue shuts out the rest: fileConflict 166.
  2. The roadmap is not in the queue. Epics are not split into Queen-shaped sub-issues, so they count as missingBoundary (79), and the planner does not read the roadmap (Queen roadmap from one .t27 source, live from the epics, read by the planner; plus self-diagnosis rules #9022).
  3. Done work stays open. 508 issues are complete but open, and are re-skipped every tick.
  4. Master is red on duplicate-bodies, cli-tri and t27b-native-ratchet, so no bee PR lands (duplicate-bodies: share tree helpers across eight gen_w bench ports #9633, cli-tri: retire deleted-workflow test and refresh measured census pins #9634 / PR policy: record approval for the 19-line gates test deletion (Refs #9634) #9639, t27b-native ratchet is red on every master run since it landed: ledger 099ac2224 predates 16 improvements and 40+ new specs #9600 / PR t27b: repair crypto and rust_convert; distinguish PR ledger lag #9637).
  5. Contracts are not wired. Most fixes already exist as specs/queen/*.t27 cards, but the runtime (gHashTag/trios) does not run them yet, and hand-written TypeScript needs the owner label.

Phase 0 -- unstarve the swarm (days)

Phase 1 -- flow (Kanban, Little's law, merge queues)

Phase 2 -- reliability (Temporal, BEAM)

Phase 3 -- intelligence (where agent schedulers are going)

How a card lands

Each new card above is Queen-shaped, with a Boundary of one new spec file and its seal, so the cards never conflict with each other and bees can take all eight at once. The bar is specs/queen/dispatch.t27 in PR #9961: 0 surviving mutants, 0 vacuous tests. Wiring follows through t27b ts (#9994), not hand-written TypeScript.

Refs #6657 #9021 #9891 #6595 #9025

Activity

  1. added
    shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it
    on Oct 17, 2026
  2. github-actions commented on Oct 17, 2026

    @github-actions
    Contributor

    Closed by the issue-shape gate (specs/policy/issue_shape.t27, #7498): an issue must have the template's sections. Edit the body; an edit that passes reopens it.

    • missing: ## Boundary -- list every file this task may touch, one per line (- specs/x.t27); without it the Queen reserves nothing and no bee can take the issue
    • missing: ## User Scenarios -- at least one Given/When/Then scenario
    • missing: ## Requirements -- numbered obligations, FR-001: ... MUST ...
    • missing: ## Success Criteria -- an outcome a machine can settle: a number with a unit, a backticked file, exit 0, or a backticked command
  3. gHashTag commented on Oct 17, 2026

    @gHashTag
    OwnerAuthor

    Measured right after filing (2026-10-17 14:49-14:54Z): the Queen took the new cards within about 3 minutes, which confirms the swarm is starved for eligible issues rather than slowed by the dispatcher. #9967, #9969, #9968 and #9974 were dispatched, then each ended "unexpectedly (cause undetermined)" 1-2 minutes later. None pushed a queen-N branch or opened a PR, and all four now sit in review wait and count as claimed. #9973 and #9975 were still running at 14:54Z.

    So the first blocker is not supply, it is bees that die within minutes and say nothing. Earlier today 7 of 9 finished dispatches ended the same way (14:05Z).

    What this needs, in order:

    1. Queen wiring: record the exit reason of every dispatch in the activity event and run the spec's conformance rows trios#1708: write the exit reason on every dispatch (specs/queen/dispatch_exit.t27 has the contract and 24 conformance rows). Without it, every fix is a guess.
    2. Read access to the trios-agent-server Railway logs for whoever runs this epic. The project is not visible from the operator account used today.
    3. A dispatch that dies before its first commit must release its claim at once (queen: a runner bee that committed nothing is an empty attempt after the grace, not a 6-hour wait (reviewer.t27 no_bundle_is_empty, behind TRIOS_QUEEN_NO_BUNDLE_EMPTY) trios#1886, specs/queen/holds.t27). Today it holds the issue in wait, so the issue is lost to every other bee.
  4. gHashTag commented on Oct 17, 2026

    @gHashTag
    OwnerAuthor

    Root cause for part of the idle and "cause undetermined" exits, from the runner logs on Railway (project 999, services trios-bee-runner-1 / trios-bee-runner-2, 2026-10-17 13:00-15:18Z).

    • Runner-2: 93 bee turns, 17 ended after 0 steps. At least 22 AI_RetryError on z.ai 429 code 1305 (temporary overload), in bursts at 13:53, 13:59, 14:08 (16), 14:19, 14:47 and 14:50. Runner-1: about 6 of 48 turns ended after 0 steps.
    • A bee gets one key at dispatch (resolveWorkerProvider, round-robin) and keeps it. On 1305 the SDK retries the same key 3 times, the turn ends without a completion frame, and the claim is held. 34 keys sit in the two pools, but a bee never moves to another one.
    • The reviewer already fails over to the next lane on transient errors (queen-tick.ts, after the 2026-09-23 incident). Bees do not.

    Card: #9993 (Phase 2). The exit should be named provider_overloaded (gHashTag/trios#1708) and the claim released (gHashTag/trios#1886).

    How to read these logs: Railway MCP get-logs on project 999 in the shared workspace, with a filter such as @issue:9967 or a quoted string like "1305". Results are capped at 200 lines per call.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    shape-refusedClosed by the issue-shape gate (specs/policy/issue_shape.t27); a passing edit reopens it

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions