Skip to content

Queen moves to the actor model: the rule in every AGENTS.md, an MVP, a benchmark, then the swap #7851

Open

Description

@gHashTag

Why

Owner rule, 2026-10-08: the Queen and the swarm move to the actor model.

specs/queen/actors.t27 has carried the BEAM/OTP rules since #6963 (pids, bounded mailboxes, a control lane, links, monitors, supervisors with intensity and backoff). Nothing runs them. The Queen still works through timers and loops: the round on one clock, the reviewer on another (#7840), and the runners counting lanes. Each new kind of work adds one more loop.

The rule

  • Everything that runs concurrently in the Queen and the swarm becomes an actor. That covers a round, a reviewer, a poller and a runner lane. Each one has a pid and a mailbox, and a supervisor restarts it.

  • No new setInterval loops, thread pools, worker counters or ad-hoc promise fan-outs for Queen work.

  • Replacement happens in this order and no other:

    1. An MVP of the actor runtime runs the same workload.
    2. Its tests pass.
    3. A benchmark runs the current loop and the actor runtime on the same input, measuring throughput, latency and recovery from a crashed worker.
    4. The numbers are posted here.
    5. Only then does the swap happen.

    Never swap first and measure later.

  • The rule goes into every root AGENTS.md we own.

Where the code lives

The Queen server moves out of the gHashTag/BrowserOS fork into gHashTag/trios, on branches queen and queen-runners. The fork holds the browser extension and nothing else.

Plan

  • Write the rule into AGENTS.md in t27, trios, trinity, tri-net, trinity-fpga and leela, and into the fork (extension only).
  • Fix the one failing card test in actors.t27 under gen-c: ctl_post(0, 63) == 9223372036854775808.
  • MVP: a runtime that drives the generated actors card. It runs the reviewer as the first actor domain, with a supervisor tree from queen_tree.
  • A benchmark of the current reviewer loop against the actor reviewer on the same recorded queue. It measures reviews per hour, wait time to the first review, the time a crashed worker takes to come back, and the work lost.
  • Post the numbers here. The swap ships only if the actor runtime is at least as good on every measure.

Acceptance criteria

  • Every root AGENTS.md listed above contains the section "Actors, not threads".
  • t27c test-report specs/queen/actors.t27 reports 0 failing tests under the C backend.
  • A comment on this issue posts the benchmark table, with both runtimes measured on the same input.

Refs #6971, #6657, #7840

Activity

  1. gHashTag commented on Oct 8, 2026

    @gHashTag
    OwnerAuthor

    MVP and benchmark: gHashTag/trios#1698

    The actor runtime is queen-actors.ts; every decision comes from the actors.wasm card built from #7866. The reviewer runs as a tree on it. It starts only behind TRIOS_QUEEN_REVIEWER=actors, and nothing is swapped yet.

    The same seeded input runs on a virtual clock three ways:

    • loop: today's reviewer, one pass every 10 s.
    • actors: woken by the bus.
    • actors-hb-only: the same tree on its 10 s heartbeat only, which isolates the runtime.

    Times are in seconds.

    steady 45/h, 60 s reviews, faults (263 rows, horizon 7 h)

    | runtime | done | open | perHour | waitP50 | waitP95 | doneP50 | doneP95 | recoveryP50 | recoveryMax | stallsStarted | killed | gaveUp |
    | loop | 263 | 0 | 37.6 | 6 | 45 | 72 | 220 | 7 | 9 | 8 | 0 | 0 |
    | actors | 263 | 0 | 37.6 | 0 | 0 | 63 | 226 | 0 | 10 | 8 | 8 | 0 |
    | actors-hb-only | 263 | 0 | 37.6 | 6 | 10 | 69 | 220 | 0 | 16 | 8 | 8 | 0 |

    steady 45/h, 180 s reviews, faults (278 rows, horizon 7 h)

    | runtime | done | open | perHour | waitP50 | waitP95 | doneP50 | doneP95 | recoveryP50 | recoveryMax | stallsStarted | killed | gaveUp |
    | loop | 278 | 0 | 39.7 | 10 | 308 | 270 | 553 | 9 | 1181 | 6 | 0 | 0 |
    | actors | 278 | 0 | 39.7 | 0 | 240 | 228 | 453 | 0 | 294 | 6 | 5 | 0 |
    | actors-hb-only | 278 | 0 | 39.7 | 8 | 244 | 246 | 451 | 0 | 71 | 6 | 5 | 0 |

    steady 45/h, 180 s reviews, no faults (262 rows, horizon 7 h)

    | runtime | done | open | perHour | waitP50 | waitP95 | doneP50 | doneP95 | recoveryP50 | recoveryMax | stallsStarted | killed | gaveUp |
    | loop | 262 | 0 | 37.4 | 6 | 98 | 216 | 319 | 0 | 0 | 0 | 0 | 0 |
    | actors | 262 | 0 | 37.4 | 0 | 90 | 210 | 309 | 0 | 0 | 0 | 0 | 0 |
    | actors-hb-only | 262 | 0 | 37.4 | 6 | 98 | 214 | 315 | 0 | 0 | 0 | 0 | 0 |

    overload 120/h, 180 s reviews, faults (660 rows, horizon 7 h)

    | runtime | done | open | perHour | waitP50 | waitP95 | doneP50 | doneP95 | recoveryP50 | recoveryMax | stallsStarted | killed | gaveUp |
    | loop | 476 | 184 | 68 | 4669 | 8327 | 4828 | 8389 | 12259 | 24781 | 8 | 0 | 0 |
    | actors | 536 | 124 | 76.6 | 3067 | 6730 | 3248 | 6932 | 2045 | 24682 | 10 | 5 | 0 |
    | actors-hb-only | 526 | 134 | 75.1 | 3864 | 7612 | 4053 | 7895 | 8 | 13073 | 9 | 7 | 0 |

    burst 120 in 10 min + 45/h, 180 s, faults (365 rows, horizon 7 h)

    | runtime | done | open | perHour | waitP50 | waitP95 | doneP50 | doneP95 | recoveryP50 | recoveryMax | stallsStarted | killed | gaveUp |
    | loop | 365 | 0 | 52.1 | 2416 | 6201 | 2686 | 6551 | 657 | 11504 | 6 | 0 | 0 |
    | actors | 365 | 0 | 52.1 | 1607 | 5224 | 1845 | 5483 | 0 | 8531 | 6 | 6 | 0 |
    | actors-hb-only | 365 | 0 | 52.1 | 1401 | 4971 | 1622 | 5233 | 0 | 8520 | 7 | 7 | 0 |

    runtime cost

    • No faults: a tie (within 2%).
    • With faults (5% crash, 2% stall): actors cut done p95 from 553 s to 453 s and the worst recovery from 1181 s to 294 s. A stall holds a loop slot for 30 min; the actor runtime kills the turn at REVIEW_ROW_SECONDS.
    • Overload 120/h: +12.6% throughput with the same 4 slots.
    • Burst: the median wait falls from 2416 s to 1607 s.
    • Overhead: 3.9 us per message (100k messages through 1000 actors).

    Limits:

    • It is a simulation. The arrival rate is production's; the lengths and fault rates are assumptions.
    • An abandoned model call still holds its z.ai lane until its 120 s timeout.
    • The serialized checkout steps are not modelled.

    Next, on the owner's OK: set TRIOS_QUEEN_REVIEWER=actors in production and compare against the loop's last 24 h. The swap ships only if production agrees.

  2. gHashTag commented on Oct 8, 2026

    @gHashTag
    OwnerAuthor

    Released, behind the flag: gHashTag/trios#1698 is merged, and TRIOS_QUEEN_REVIEWER=actors is set on trios-agent-server

    The owner said yes in chat on 2026-10-09.

    One correction before release, in 1f738b9. A killed review keeps its row and its z.ai lane until the review really ends. A JS promise cannot be stopped, so without this the intake would have handed the same row to a second worker while the first review still ran. The rerun benchmark below is on the same seeded input and is deterministic.

    steady 45/h, 60 s reviews, faults (263 rows, horizon 7 h)

    runtime done open perHour waitP50 waitP95 doneP50 doneP95 recoveryP50 recoveryMax stallsStarted killed gaveUp
    loop 263 0 37.6 6 45 72 220 7 9 8 0 0
    actors 263 0 37.6 0 0 62 215 0 8 8 8 0
    actors-hb-only 263 0 37.6 5 10 66 220 0 20 8 8 0

    steady 45/h, 180 s reviews, faults (278 rows, horizon 7 h)

    runtime done open perHour waitP50 waitP95 doneP50 doneP95 recoveryP50 recoveryMax stallsStarted killed gaveUp
    loop 278 0 39.7 10 308 270 553 9 1181 6 0 0
    actors 278 0 39.7 0 152 210 384 0 27 6 6 0
    actors-hb-only 278 0 39.7 7 160 215 386 0 10 6 6 0

    steady 45/h, 180 s reviews, no faults (262 rows, horizon 7 h)

    runtime done open perHour waitP50 waitP95 doneP50 doneP95 recoveryP50 recoveryMax stallsStarted killed gaveUp
    loop 262 0 37.4 6 98 216 319 0 0 0 0 0
    actors 262 0 37.4 0 90 210 309 0 0 0 0 0
    actors-hb-only 262 0 37.4 6 98 214 315 0 0 0 0 0

    overload 120/h, 180 s reviews, faults (660 rows, horizon 7 h)

    runtime done open perHour waitP50 waitP95 doneP50 doneP95 recoveryP50 recoveryMax stallsStarted killed gaveUp
    loop 476 184 68 4669 8327 4828 8389 12259 24781 8 0 0
    actors 565 95 80.7 2824 6165 3054 6363 10 22298 10 7 0
    actors-hb-only 569 91 81.3 2848 6090 3060 6306 85 13056 10 7 0

    burst 120 in 10 min + 45/h, 180 s, faults (365 rows, horizon 7 h)

    runtime done open perHour waitP50 waitP95 doneP50 doneP95 recoveryP50 recoveryMax stallsStarted killed gaveUp
    loop 365 0 52.1 2416 6201 2686 6551 657 11504 6 0 0
    actors 365 0 52.1 1153 4863 1593 5122 0 7712 6 6 0
    actors-hb-only 365 0 52.1 1115 4879 1598 5141 0 7521 6 6 0

    What changed against the first table:

    • With faults (180 s reviews), done p95 is now 384 s against 553 s for the loop.
    • Overload throughput is now +18.7% (565 against 476).
    • Burst median wait is now 1153 s against 2416 s.
    • No faults: still a tie.
    • Overhead: 3.7 us per message.

    Production baseline (the loop) at the moment of the flip, from /queen/status:

    reviewed/h finished/h unreviewed active bees
    25 62 100 27/70

    Over the previous 8 h, reviewed/h stayed between 23 and 31.

    The 24 h production comparison will be posted here. If production is worse, the rollback is to unset the flag; the loop is still the default code path.

  3. gHashTag commented on Oct 8, 2026

    @gHashTag
    OwnerAuthor

    Production, first measurements of the actor reviewer (2026-10-08)

    19:00Z, the actor reviewer went live without a wait backoff (gHashTag/trios#1698).

    • It visited the 22 rows whose branch cannot be read about 184 times a minute. The loop had visited them about 20 times a minute.
    • The traffic stretched the round's tail to about 6 minutes between rounds (normally about 30 s), so fewer bees were dispatched: active bees fell to 22-27 of 70.
    • Real decisions came faster than with the loop: 13 in 6.5 minutes.

    19:44Z, the wait backoff went live (gHashTag/trios#1702: reviewer.t27 due_again, a worktree-only git lock, per-row timing). First 10.5 minutes:

    verdict rows p50 p95 worker-seconds
    sendBack 8 49 s 77 s 329
    escalate 3 94 s 99 s 250
    empty 7 0.1 s 0.5 s 2
    wait 117 0.0 s 0.1 s 3
    • Rounds are back to one every ~31 s.
    • wait visits are down to about 11 a minute, and falling as the backoff grows.
    • The reviewer is no longer the bottleneck. It used 584 of 2520 worker-seconds (23%). The ceiling at ~60 s per real review is about 240 reviews an hour, against an inflow of about 60.
    • Criteria measurement is cheap: 4-7 s each, 3 in 10 minutes. Moving it to other nodes would not change the numbers, so it stays on the Queen.
    • The rows whose branch is lost are Rows whose branch is lost wait 6 h for the frozen-wait valve; release them at once as a free attempt trios#1706.

    The one-hour comparison against the loop follows.

  4. gHashTag commented on Oct 9, 2026

    @gHashTag
    OwnerAuthor

    Adaptive reviewer concurrency: fixed 4 against adaptive on the same input (gHashTag/trios#1712, item 3)

    The benchmark is the second table in queen-actors-bench.test.ts. It uses the same seeded workload() draws and the actor reviewer both times.

    Lane model. A review asks for its lane 30% of the way in, for its model call. If no lane is free, it answers wait, with backoff.

    Hosts. 47 free lanes is /queen/status on 2026-10-09: 70 lanes, 23 busy. Memory is 8 GB at 512 MB a review.

    Columns.

    • done/rows counts reviewed rows out of all rows.
    • p95 wait is from arrival to a row's first start.
    • max calls is the most model calls at once.
    • lane-less counts visits that found no free lane.
    • drain is seconds from the last arrival until nothing is open.
    • burst done is seconds until all 120 burst rows are done.
    case fixed 4: done/rows, /h, p95 wait s, max calls, drain s adaptive: done/rows, /h, p95 wait s, max calls, drain s
    quiet 10/h 60/60, 8.6, 0, 4, 202 60/60, 8.6, 0, 4, 202 (0.7 workers on average)
    steady 45/h 278/278, 39.7, 152, 8, 239 278/278, 39.7, 0, 9, 239
    overload 120/h 565/660, 80.7, 6165, 9, never 660/660, 94.3, 0, 12, 177
    burst 120 + 45/h 365/365, 52.1, 4863, 7; burst done 11690 365/365, 52.1, 725, 16; burst done 2895
    overload, 3 lanes 429/660, 61.3, 834, 3; 427 lane-less 360/660, 51.4, 12350, 3; 3 lane-less
    burst, 3 lanes 365/365, 52.1, 3136, 3; 322 lane-less 352/365 (13 open), 50.3, 8477, 3; 4 lane-less
    overload, 1.5 GB free, review size unmeasured 565/660, 80.7 399/660, 57.0 (pool capped at 3)

    Where adaptive wins. With ample lanes it is better under overload and bursts, and ties when quiet. The price is more model calls at once: up to 16, against 7.

    Where it loses. It is worse in two cases:

    • Scarce lanes. A review holds its lane only for the model call. A pool capped at the lanes leaves each lane idle for that review's no-lane 30%. Fixed 4 overlaps that phase, at the cost of hundreds of lane-less wait visits. In production those visits also count as reviewer misses; the simulation does not model that cost.
    • A small container whose review size nobody measured. The assumed 512 MB per review caps the pool at 3.

    Before the flag goes on, the item-1 telemetry must show:

    1. Free review lanes over 24 h stay at or above about 5. That is slots_needed(58).
    2. The container's free memory, and memory per review measured and set as TRIOS_QUEEN_REVIEW_MB.
    3. What actually bounds the reviewer now: worker-seconds split into decided / wait / empty / killed / held-after-kill. If most visits are wait on unreadable rows, more workers add load, not reviews.
    4. The rate of 1302/429 on review calls and on bee calls. Bee dispatch does not count lanes held by reviews.
    5. Time waiting on the serialized git worktree and import steps. These are not in the simulation.
    6. Then 24 h with the flag on: reviews/h at or above finishes/h (about 58), the backlog shrinking, and crashes and kills per hour flat.
  5. 19 remaining items

  6. gHashTag commented on Oct 10, 2026

    @gHashTag
    OwnerAuthor

    Self-hosting slice 1: gHashTag/trios#1756, lane trios#1757, PR trios#1760 (merged into hosting-mvp), specs t27#8720 and t27#8732

    The mechanism: a corpus shard (t27c test-report on one spec at a pinned commit) runs on k = 2 hosts. The Queen believes the result when their normalized output hashes agree, and credits both hosts 1 mTRI off-chain. The current way is the labs running the spec once (lab.py reference_one).

    Input: the same for both paths, specs/queen/netlink.t27 (sha256 447de250..., 9 tests). This host is an M-series Mac. Its load average was 12 to 26 during the runs, from other sessions. Each row is the median of 3 rounds. tests/api/hosting-bench.test.ts, HOSTING_BENCH=on.

    path wall s CPU s (t27c+zig) Queen CPU ms / job messages / job
    before: one run, the lab's way 10.8 6.17 n/a n/a
    after: verified, k = 2, the replicas back to back on one Mac 14.3 10.87 32.9 8
    • Raw rounds:
      • before: wall 10.8, 7.1 and 17.4 s; CPU 6.17, 5.26 and 6.53 s.
      • after: wall 13.1, 21.3 and 14.3 s; host CPU 9.89, 12.48 and 10.87 s; Queen CPU 35.5, 32.9 and 23.2 ms; messages 8, 12 and 8.
    • Messages per job on a VirtualClock, for a 10 s run:
      • setup: 2 registrations (once per host) and 1 job;
      • from the first lease to the verdict: 12 messages (4 leases, of which 2 got work; 2 receipts; 6 beats);
      • two idle hosts send 8 in the same 10 s (4 beats, 4 empty leases).

    Honest reading

    • Verification costs k times the compute. Host CPU was 1.76x one run for k = 2. A tiebreak (a third host) makes it 3x.
    • The Queen's own share is about 33 ms of CPU per job, about 0.5% of one run: signatures, hashes, card calls and the memory store.
    • The output hash was the same in every run: 3 single runs, 6 replicas and one run through the CLI binary all gave 41d6e44c.... Normalizing the output (facts, not text) is what makes honest hosts agree byte for byte.
    • Wall times here say little. The same single run took 7.1 to 17.4 s under this host's load. Replicas on two machines overlap, and that was not measured.

    Not measured: the Postgres store (the bench uses the memory store), network latency, the linux-x64 binary running on Linux, and a public host on a real second network.

  7. added a commit that references this issue on Oct 10, 2026
  8. gHashTag commented on Oct 10, 2026

    @gHashTag
    OwnerAuthor

    Self-hosting slice 1b, lane B: the sybil benchmark (gHashTag/trios#1761)

    This is the actors rule's benchmark for the sybil defense in self-hosting.

    anchor and canaries ON (all five rules), defenses = 31

    M attacker mTRI per run honest mTRI per run wrong results agreed per run attacker keys suspended (8 runs) honest keys suspended (8 runs) jobs agreed (8 runs)
    1 0.000 157.250 0.000 8 of 8 0 of 64 1920 of 1920
    2 0.000 155.000 0.000 16 of 16 0 of 64 1920 of 1920
    5 0.000 147.375 0.000 40 of 40 0 of 64 1916 of 1920
    20 0.000 111.875 0.000 160 of 160 0 of 64 1897 of 1920

    anchor and canaries OFF (probation, cap, slash on), defenses = 28

    M attacker mTRI per run honest mTRI per run wrong results agreed per run attacker keys suspended (8 runs) honest keys suspended (8 runs) jobs agreed (8 runs)
    1 0.000 399.875 0.000 8 of 8 0 of 64 1920 of 1920
    2 0.000 396.625 1.625 16 of 16 0 of 64 1920 of 1920
    5 0.000 384.750 8.000 40 of 40 1 of 64 1920 of 1920
    20 260.125 0.000 229.125 2 of 160 64 of 64 1920 of 1920

    slice 1 as merged (no new rule), defenses = 0

    M attacker mTRI per run honest mTRI per run wrong results agreed per run attacker keys suspended (8 runs) honest keys suspended (8 runs) jobs agreed (8 runs)
    1 0.000 480.000 0.000 8 of 8 0 of 64 1920 of 1920
    2 3.250 476.750 1.625 16 of 16 0 of 64 1920 of 1920
    5 16.000 464.000 8.000 40 of 40 1 of 64 1920 of 1920
    20 458.250 21.750 229.125 2 of 160 64 of 64 1920 of 1920

    Reading the tables

    • With all five rules, the attacker earns 0 and no wrong result is agreed, for M = 1, 2, 5 and 20.
    • With the anchor and canaries off (the negative control), the attacker earns 260 mTRI per run at M = 20 and suspends all 64 honest hosts.
      • At M = 2 and 5 it earns 0, because probation and slice 1's strikes stop it first.
      • Even then, 1.6 and 8 wrong results per run are agreed. The anchor is what keeps results right.
    • Slice 1 as merged pays the attacker from two keys up.
    • M = 1 is 0 in every table: one key never forms a quorum.
    • The cost to honest public hosts: about two thirds of their credit, because the anchors run half the replicas and probation holds back each key's first 10 votes.

    The same input in the runtime (gHashTag/trios#1771, memory store)

    The actors are the owner, a trusted host, an honest public host and two colluding public keys.

    • With TRIOS_HOSTING_SYBIL=on:
      • the second public key is refused the anchor's slot and the tiebreak;
      • the trusted host breaks the tie and the true verdict is agreed;
      • the liar is struck and slashed.
    • With the rules off: both keys take the job, agree on the wrong verdict, and are paid 1 mTRI each.

    🤖 Generated with Claude Code

  9. gHashTag commented on Oct 10, 2026

    @gHashTag
    OwnerAuthor

    Self-hosting slice 1b, lane A (gHashTag/trios#1762, part of gHashTag/trios#1761): the full lab verdict per spec, and the sandbox's cost. Same inputs on both sides. The machine is the owner's MacBook Air (8 GB), load average 18 to 52 from other sessions. t27c and t27b are the release builds of master cf42584; zig is ziglang.org's pinned 0.16.0.

    1. A full-verdict row against the lab's per-spec cost

    ours: 24 specs at master cf42584, both halves, k = 2, 2 hosts, in the sandbox the t27b lab, run of 8e2e44b (no cache hits)
    work per spec 4 runs (2 halves x 2 replicas) 2 runs (each half once)
    measured 446.7 CPU-s for both hosts (/usr/bin/time -l: 217.6 + 229.0 s), 98 runs. That is 18.2 CPU-s per complete row (4 runs, with beats and 2 probes), or about 9.1 CPU-s per spec for one replica of each half reference: 324.3 s wall x 12 workers / 1810 specs = 2.15 worker-s per spec. t27b under qemu: 325.1 s wall x 24 jobs / 1812 files = 4.31 worker-s per spec. Together, 6.5 worker-s per spec, once
    wall 24 rows in about 15.5 min (20:00:13Z to at most 20:15:44Z) with 2 hosts, slots 1 each 324 s + 325 s for 1812 specs
    result 25 of 25 rows equal the lab's rows of cf42584 on every key the lab wrote (24 sample rows, plus 1 from the clean-install host). This covers 74 t27b and 110 reference per-test verdicts. 0 differences.

    How to read this:

    • The lab gives step totals only, not per-spec timings. Its unit is wall x workers on a 24-CPU x86_64 container, and ours is CPU seconds on an arm64 laptop, so the two columns are not one measure.
    • k = 2 doubles the work by construction.
    • The lab reuses its reference cache: 1816 of 1821 hits on cf42584. The hosts have no such cache yet, so every half runs.
    • The sample is weighted to the rare row shapes (blocked/fail, fail/blocked, codegen, frontend, vacuous), not to the corpus mix.

    2. The sandbox's overhead: the same half, bare and in sandbox-exec

    Medians of 3 alternating rounds after one unmeasured warm-up, with one warm zig cache. Wall and CPU are in ms, CPU including children, from createT27cRunner exactly as trios-host join runs it. Every run of a half gave one output hash.

    spec half bare wall sandbox wall bare CPU sandbox CPU
    specs/a/b_c.t27 reference 21902 12189 8456 7072
    specs/a/b_c.t27 t27b 31 109 16 73
    specs/queen/netlink.t27 reference 19578 14381 8667 7845
    specs/queen/netlink.t27 t27b 57 94 33 67
    specs/account/auth.t27 reference 4648 2348 3407 2485
    specs/account/auth.t27 t27b 727 1115 474 514
    specs/fpga/eth_reply.t27 reference 20440 20338 9508 8984
    specs/fpga/eth_reply.t27 t27b 85 235 45 105
    specs/lsp/language.t27 reference 10497 8562 6504 5818
    specs/lsp/language.t27 t27b 21 91 15 64
    specs/isa/ternary_shift.t27 reference 14288 13925 7342 6827
    specs/isa/ternary_shift.t27 t27b 172 293 104 156
    sum over the 6 specs bare sandbox
    reference half, wall / CPU 91.4 s / 43.9 s 71.7 s / 39.0 s
    t27b half, wall / CPU 1.09 s / 0.69 s 1.94 s / 0.98 s
    • Reference half: the sandbox measured faster here (-21% wall, -11% CPU). No sandbox denial was logged for t27c or zig during a run, so nothing was skipped. The machine's load noise and the order (bare always ran first in a round) are larger than whatever the sandbox costs, so read this as "no measurable cost", not as a speed-up.
    • t27b half: the sandbox adds about 0.14 s wall and 0.05 s CPU per run, mostly sandbox-exec's own start.
    • The planted job (trios-host probe) builds and runs a small C program with zig inside the sandbox once per join: 9.8 s inside and 7.4 s bare in the test run, cold cache.

    3. The Queen

    • Requests: during the rehearsal (25 rows = 50 jobs, up to 3 hosts, about 28 min), the Queen's /hosting route answered 478 beats, 213 lease requests, 100 receipts, 50 job submissions and 5 registrations.
    • CPU: 4.42 s in all, including 46 polls of GET /hosting/verdicts.

    🤖 Generated with Claude Code

  10. gHashTag commented on Oct 10, 2026

    @gHashTag
    OwnerAuthor

    The first production reading, with telemetry on (trios#1763, step A)

    Setup. Production Queen trios-agent-server at 82bcc934e (batch 3), with TRIOS_QUEEN_ACTORS_TELEMETRY=on since 19:29Z. Every other actor flag is off. The reading covers 12 summary windows of 300 s, 2026-10-10 19:34–20:33Z. It spans two containers: an unexplained same-commit redeploy happened at 20:15Z. Both logs show 0 [ERROR].

    actor kind live turns in the hour busy, median / max turn p95 bucket, max mailbox depth, max crashes / kills / restarts dead letters / dropped
    reviewer-worker 4 469 9.3% / 24.9% 262–524 s, in one window 1 0 / 1 / 1 0 / 0
    reviewer-intake 1 886 0.1% / 2.7% ≤ 128 ms 5 0 / 0 / 0 0 / 0
    • Real reviews: 14 Queen reviewer measured lines in the hour, all ok=true, typically 3.0–3.6 s each. Two model calls (nemotron-3-ultra, issue 6241) timed out, and the log reports nothing spent.
    • The one kill: one worker turn passed turnMaxSeconds=300. The supervisor killed and restarted it, and no storm followed.
      • With turnStop=false, the killed turn's model call is abandoned but not stopped, so it keeps its lane until its own timeout. That is exactly the gap TRIOS_QUEEN_TURN_STOP closes (turn_stop.t27).
    • The old question, "23% busy or 11 reviews an hour". Both old readings fit this hour:
      • 23% is a peak window; the peak here is 24.9%;
      • about 11 reviews an hour is close to today's 14;
      • the median worker is only 9.3% busy.
    • Not measured: the CPU cost of telemetry in production, which needs an on/off comparison under the same load. The load here is low.

    Next flag: TRIOS_QUEEN_TURN_STOP=on, on this evidence: one kill an hour whose call kept its lane. Its check: kills still followed by a restart, a killed call aborted within the grace, 0 new [ERROR], and reviews per hour not lower.

  11. gHashTag commented on Oct 10, 2026

    @gHashTag
    OwnerAuthor

    Vault bench (gHashTag/trios#1759, PR gHashTag/trios#1775, specs #8777)

    What is compared. Both sides start the same command with the same five dummy values in its environment, made at test time.

    • Today: the platform injects the environment, so the process is spawned with the variables already set. This is Bun.spawn(['true'], { env }).
    • The vault: trios-vault run --scope five -- true. This is a signed lease, the values sealed to a one-lease age key, opened in memory, then the spawn, then a revoke.

    How it was measured. The file is tests/api/vault-bench.test.ts, run with VAULT_BENCH_ROUNDS=15. Everything is in-process, through the vault's Hono app, with no network. The store is a file. The host is an 8 GB MacBook Air shared with other lanes, at load average 23.2 / 22.3 / 21.9 during the run, with bun 1.3.11 and age 1.3.2.

    measure n median ms p90 ms min ms max ms
    lease, 1 name 15 72.7 92.4 62.2 108.4
    revoke after a lease of 1 15 3.5 6.0 2.5 7.5
    lease, 5 names 15 123.9 146.9 104.9 198.2
    revoke after a lease of 5 15 2.5 5.1 2.2 5.5
    spawn true, env given directly (today) 15 4.3 8.4 3.5 10.1
    run --scope five -- true (vault) 15 121.5 153.1 111.1 213.1

    Reading it.

    • A lease costs 3 + N process starts of the reference age tool:
      • the one-lease key;
      • one decrypt per name, in the vault;
      • one encrypt to the one-lease key;
      • the client's decrypt.
    • So the vault adds roughly 120 ms to a process start with 5 names. It pays this once per start and once per renewal, every 450 s; a running process pays nothing per call.
    • The p90 of the 5-name lease (146.9 ms) and of run (153.1 ms) are close, because run is that lease plus a 4 ms spawn plus a 3 ms revoke.
    • An earlier 15-round run of the same file at load 18-22, before the store re-read on every request, gave: lease 1 name 96.8 ms median, lease 5 names 186.2 ms, run 176.8 ms, direct spawn 7.2 ms. On this host the load moves the absolute numbers by tens of ms. The ratio between the vault and the direct spawn is what each run shows.

    Not measured. I did not measure a lease over the network to a deployed Queen, or one with the Postgres store; the Postgres store is tested for correctness in pglive, not timed. The vault did not replace anything in this PR: it is off unless TRIOS_VAULT=on, and services move to it one at a time.

    🤖 Generated with Claude Code

  12. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    A correction and a real bug: turn stop's escalation could miss under load (gHashTag/trios#1766, fix #1811)

    The bug.

    • When a turn is stopped, the runtime arms a 3000 ms grace timer.
    • When the timer fires, the runtime asks the card whether the grace has passed, from Date.now() - at.
    • Bun's setTimeout(fn, 3000) sometimes fires when Date.now() shows only 2997–2999 ms. The card then answers "not yet" (stop_signal(2999, 3000) = X_NONE), and nothing asked again.
    • As a result, a process turn deaf to SIGTERM never got its group SIGKILL and held its lane.
    • Measured under an 8-thread CPU burner: in one run of 100 deaf turns, 27 escalations were lost. On CI, run 38094454978 had 2 survivors out of 5 in bench B.
    • Same defect elsewhere: the orderly-stop timeout in queen-actors-links.ts stop(), which keyed actors, bee actors and review lanes use.

    Production. TRIOS_QUEEN_TURN_STOP=on since 20:38Z.

    • Bounded by the path: the defect bites only a deaf process turn. A model call hears its abort within milliseconds.
    • Not observed: the production hour after the flip had 0 kills, so the defect was never exercised there.

    The fix (#1811).

    • The change: re-ask the card after the remaining time until it answers X_KILL; the decision stays the card's.
    • New tests: an EarlyClock that fires every timer 1 ms early. Three new tests fail on the old code.
    • Under the burner: 20/20 runs pass, 0 survivors, deaf turns freed 3003–3143 ms after the kill.

    Correction to earlier numbers. The turn-stop benchmark's "before" row posted earlier in this issue showed deaf-stall freeing at 9998–9999 ms. That was the bench's own cleanup kill, not the runtime. With the fixed bench it reads "> 10000".

  13. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    Self-hosting slice 2a: the feed (gHashTag/trios#1869, behind TRIOS_HOSTING_FEED, off), same input before and after.

    Rule: specs/hosting/feed.t27 (#8974). A half of a lab row is carried when its spec's use closure and its toolchain sources did not change; otherwise it is sharded to the hosts.

    input: t27b-lab runs d2570f4 -> 7cb53d6 (no toolchain source moved), 1856 specs half-runs for the commit rows equal to the lab's
    before: the lab, every spec, both halves, k = 1 3712 1856 (by definition)
    after: the feed (8 changed, 4 new, 6 lab timeouts, 36 audited halves), k = 2 108 1852 of 1852 settled; 4 testbenches open, which the lab timed out on

    Behind the rule: 21 consecutive lab-run pairs. Where a half's toolchain held, 0 of 16556 reference halves and 0 of 14742 t27b halves of unchanged closures changed. Carrying by closure alone would have carried 233 halves wrongly.

    Capacity, at 5.83 s per run per slot (measured in the production rehearsal):

    • spec churn plus the audit needs 1 slot;
    • a lab-equivalent receipt, re-running a half on every t27c or t27b move, needs 13 to 18 slots;
    • the hosts also need each commit's own toolchain, which is not built. Today's hosts are pinned to cf42584.

    The labs stay on.

    🤖 Generated with Claude Code

  14. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    #9021: the round's dispatch loop vs the batch chooser on one recorded backlog

    Mechanism: gHashTag/trios#1885, behind TRIOS_QUEEN_DISPATCH_BATCH and TRIOS_QUEEN_DERIVED_FREE, both off by default. It calls the cards:

    Input: the live data of 2026-10-11 10:57:58Z: 1239 open issues, 754 candidates, and the public board with 7 running and 46 in review. queend comes from trios origin/queen ddbff4a94, with 70 lanes. The holds fix (trios#1880) is applied in every arm.

    Virtual clock: 180 rounds of 60 s. A bee runs 35 min, then holds 30 min, then it is done. No new issues arrive.

    arm bees active, mean (max) idle lanes, mean dispatches/h duplicate claims concurrent same-file pairs: meeting / derived-only / disjoint regions
    loop (today) 7.1 (13) 62.9 11.3 0 6 / 0 / 0
    batch 6.7 (13) 63.3 10.7 0 0 / 0 / 0
    batch + derived 9.8 (27) 60.2 16.0 0 0 / 158 / 0
    batch + derived + regions (as the issues name fns today) 10.0 (27) 60.0 16.3 0 0 / 158 / 1
    same, whole-file compiler.rs editors last (upper bound) 18.3 (55) 51.7 30.7 0 0 / 156 / 493

    Where the old mechanism wins, or ties:

    • The batch alone is not faster than the loop. The loop already re-asks queend after every pick: in production, 56 of 56 dispatching rounds ended on "nothing to choose", not on a cap.
    • The loop admits 6 pairs that really share files. Swift's pathsOverlap reads .trinity/seals/ with its trailing slash as overlapping nothing but itself. The batch reads it as a directory.

    What moves the number:

    🤖 Generated with Claude Code

  15. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    Golden chain vs the GitHub loop, same 80 merged PRs

    Replay spec: specs/queen/replay/golden_chain_history.t27 in #9089 (issue #9088). Window: master's last 80 first-parent commits up to a9ad525, all PR merges, #8776 .. #9082 in merge order; start head master at #8775.

    GitHub loop, as it went golden chain (gc_land, t27c gen-c of golden_chain.t27)
    landed 80 80 of 80 in one pass
    landed with nothing rerun or recomputed - 71 (FORWARD 39, CARRIED 32)
    derived files recomputed 3 PRs were seal re-saves only (#9070, #9073, #9079) REGEN 9
    (a) merges of master into a PR branch 11 in 9 PRs 0
    (b) force pushes 2 (1 rebase, 1 amend that kept its base) 0
    (d) full reruns caused by base movement 13 by the literal count (a)+(b); 4 measured extra full CI rounds on a PR that had already run CI (24-35 workflow runs each) REJUDGE 0
    (c) conflict resolutions 5 update merges conflicted, in 5 PRs (#8647, #8954, #8993, #8999, #9009); 2 of them only in derived files CONFLICT 0

    Headline: 80 of 80 PRs would have landed with no rerun (71 untouched, 9 with only seals or the ledger recomputed); GitHub's loop took 11 update merges and 2 force pushes, 13 full reruns by the literal count and 4 by the Actions record, and 5 hand-resolved conflicts.

    Sensitivity (the same edit rows unless noted):

    parents from FORWARD CARRIED REGEN REJUDGE CONFLICT UNKNOWN_PARENT landed
    last merge-base (asserted) 39 32 9 0 0 0 80
    first fork point (asserted) 34 30 14 0 1 (#9009) 1 (#9054) 78
    first fork point and the diff as first proposed (lab scratch, not asserted) 31 21 12 0 2 (#8993, #9009) 14 64

    Method. Collected with gh and git only; transformed and run on the lab. One link per PR, every stage golden. Parent: git merge-base <merge>^1 <PR head>, mapped to the last window PR merged at or before it. Edits: git diff -U3 --no-renames <merge-base> <PR head>, each hunk EDIT_LINES over the base side (lo = old start, hi = old start + old count); added, deleted, binary or wholly rewritten files EDIT_WHOLE; gen/, .trinity/seals/, docs/reports/t27b_expectations.json and bootstrap/stage0/FROZEN_HASH GC_EDIT_DERIVED (2012 rows over 1332 files, 986 of them derived). Imports: the use targets of the touched .t27 files at the merge commit, resolved as use_resolve.rs does (372 of 372). The outcomes are what t27c gen-c specs/queen/golden_chain.t27 plus a one-file C main printed; the replay asserts the counts, all 80 outcomes, the fork-point run and a negative control (a real service.rs hunk of #8857 moved onto #8847's conflicts). test-report: 3 pass, 0 FAIL, 0 vacuous. GitHub side: (a) PR commits with two parents, one an ancestor of master (11 distinct; 3d322df counted once, under #8999, though #9009 inherits it); (b) head_ref_force_pushed timeline events, classified by the before/after parents from GraphQL; (c) git merge-tree --write-tree on each update merge's parents (the one rebase replays cleanly with --merge-base); (d) Actions runs per head sha. 6 of the 11 merges were made before the PR opened, so its first CI round ran on them; #8993 and #9009 opened in conflict, when GitHub runs no pull_request workflow, and their merges gave them their first round; #8647 (1) and #9055 (2 merges and the rebase) paid the 4 extra rounds. Two clean merges also carry hand edits (#8795, and #8919 recounting a ledger cap).

    What the model does not capture, and findings.

  16. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    Benchmark: no_bundle_is_empty (reviewer.t27; see #9069, spec PR #9095, runtime gHashTag/trios#1891)

    Mechanism. Today a runner bee that committed nothing leaves no bundle, so its row answers wait ("the diff of queen-N could not be read") until the six-hour frozen-wait valve. With TRIOS_QUEEN_NO_BUNDLE_EMPTY=on (off by default), the row is judged exactly as a container bee that committed nothing once the card's grace has passed: NO_BUNDLE_GRACE_MINUTES = 10 since the row finished. Its issue is then free after the 30-minute retry floor.

    Same input, both arms. The input is every reviewer visit whose diff could not be read, from production deploys 363860fd, fdbbd636 and 932a3c3a, 2026-10-11 07:10:27-10:31:08Z.

    before (flag unset) after (flag on)
    runner attempts with no bundle in the window 41 41
    judged empty 0 36 (36 issues)
    still waiting at their last recorded visit 41 5
    how long those had waited since their first visit, median / max 151.2 / 199.3 min 1.8 / 8.8 min
    first visit to verdict, median / max never in the window 16.0 / 17.2 min
    reviewer visits spent on them 694 228
    rows already waiting at 07:10Z: judged / attempts 0 / 24 24 / 24
    rows dispatched in the window: judged / attempts 0 / 17 12 / 17
    runner attempts whose bundle came late 15 15
    read on their branch 15 15
    misjudged empty 0 0

    Deploy 363860fd alone (07:10-09:43Z), with the same cut:

    Reading it:

    Issue #9069's success criterion said "35 rows judged empty 10 minutes after finishing". On the replay:

    • 36 attempts on 36 issues are judged empty, 16.0 minutes after their first visit (median);
    • 5 more are still inside the grace when the record ends;
    • 0 of the 15 late-bundle attempts are misjudged.

    The issue counted 16 late bundles by issue. #8770's import belongs to a re-dispatch with no failed diff before it, so it is not a late bundle here.

    🤖 Generated with Claude Code

  17. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    Benchmark: how often would a region contract on bootstrap/src/compiler.rs be honoured? (merge-side region check, see gHashTag/trios#1888 and #9097)

    Input: the last 30 merged PRs on master that edited bootstrap/src/compiler.rs, from #9028 back to #6003. All 30 are real merges, none synthetic.

    Method:

    • For each PR, take its diff against its base: the parent of a squash commit, or the merge base of a merge commit's two parents.
    • Read the diff with git diff -U0, through the same hunk reader the check uses (hunksOf).
    • Ask, for each hunk, which fn item it lies inside at that base. This uses the vendored cards, issue_shape.t27 item_span and edit_inside, with the spec(policy): a Boundary line may name one item of its file -- path :: fn NAME, resolved at the base commit (Closes #9059) #9062 insertion rule. Nothing is re-typed.
    • A hunk is "outside every fn" when no fn resolvable by item_span contains it. The reasons split three ways:
      • it adds a new fn (the inserted text has fn NAME);
      • its fn's name is defined more than once in compiler.rs, so item_span refuses it and the line would hold the whole file;
      • other: a doc comment or attribute above a fn (item_span starts at the fn line, so these are outside), a top-level item, or a hunk that spans two items.
    • Candidate fn NAME texts after // or inside an open string on their line are skipped as candidates. item_span makes the final decision on every candidate.
    • Two contracts are measured. In the first, the issue declares only the fn with the most changed lines. In the second, it declares every fn the PR's hunks lie in. The first is the upper bound on widening, the second the lower bound.
    PR base hunks fns its hunks lie in hunks outside every fn: new fn / ambiguous name / other class widened, main fn declared widened, every touched fn declared
    #9028 cc19d1f 19 6 5: 2 / 1 / 2 outside every fn yes yes
    #8993 eafe700 3 2 1: 1 / 0 / 0 outside every fn yes yes
    #8991 ada4e8b 14 4 2: 1 / 0 / 1 outside every fn yes yes
    #8986 1670645 1 1 0 inside one fn no no
    #8667 a6627d1 2 1 0 inside one fn no no
    #8636 b8100e8 2 1 0 inside one fn no no
    #8617 223c095 3 1 0 inside one fn no no
    #8566 0b5309d 1 1 0 inside one fn no no
    #8418 c9d36f3 1 1 0 inside one fn no no
    #8415 aca4bf7 2 2 0 crossed into another fn yes no
    #8154 409b153 14 7 3: 1 / 0 / 2 outside every fn yes yes
    #7495 8378792 2 1 0 inside one fn no no
    #7331 d4f74f8 6 4 1: 1 / 0 / 0 outside every fn yes yes
    #7469 cfc97d2 11 4 3: 1 / 1 / 1 outside every fn yes yes
    #7536 78c060f 3 1 1: 1 / 0 / 0 outside every fn yes yes
    #7468 bb24bf8 3 2 0 crossed into another fn yes no
    #7456 ec1c2bd 1 1 0 inside one fn no no
    #7387 05e633d 6 4 1: 0 / 0 / 1 outside every fn yes yes
    #6994 4f1dea7 2 2 0 crossed into another fn yes no
    #6939 322cc77 22 5 8: 4 / 3 / 1 outside every fn yes yes
    #6862 fa39f30 2 2 0 crossed into another fn yes no
    #6599 16f15c8 1 1 0 inside one fn no no
    #6596 1f9b370 15 3 3: 1 / 1 / 1 outside every fn yes yes
    #6585 7e49725 8 4 1: 1 / 0 / 0 outside every fn yes yes
    #6579 597216d 4 3 1: 1 / 0 / 0 outside every fn yes yes
    #6102 82d8795 11 5 1: 0 / 1 / 0 outside every fn yes no (that fn is whole from the start)
    #6187 e0f894f 3 2 1: 1 / 0 / 0 outside every fn yes yes
    #6103 c3a123d 5 2 2: 2 / 0 / 0 outside every fn yes yes
    #5947 5fc99ce 12 4 2: 2 / 0 / 0 outside every fn yes yes
    #6003 ec1d9a6 8 3 4: 1 / 1 / 2 outside every fn yes yes

    Totals over 30 PRs (187 hunks):

    • Stayed inside one fn: 9 of 30.
    • Crossed into another fn, with every hunk inside some fn: 4 of 30.
    • Touched code outside every fn: 17 of 30.
      • 15 PRs add a new fn. In 8 of them, that is their only outside hunk.
      • 6 PRs touch a fn whose name is defined more than once in the file (new, parse, default, calls).
      • 8 PRs touch something else outside every fn: a doc comment or attribute above a fn, a top-level item, or a hunk that spans two items.
    • The check would widen:
      • 21 of 30 when the issue declares only its main fn;
      • 16 of 30 even when it declares every fn the PR touched.
    • By hunk: 147 of 187 lie inside a resolvable fn. Of the 40 outside, 21 add a fn, 8 sit in an ambiguous fn, and 11 are other.

    What this says:

    • A one-fn region contract held for 30% of real compiler.rs PRs.
    • The main reason it fails is not that a bee wandered into a neighbour's function. It is a new helper fn added beside the declared one. The spec(policy): a Boundary line may name one item of its file -- path :: fn NAME, resolved at the base commit (Closes #9059) #9062 rule puts an insertion at a region's edge outside it, so every such PR widens.
    • A widened edit is sent back only if it meets a live edit of the same file. So the cost of a widening depends on how many bees hold compiler.rs at once. Today no issue declares a region, so it is zero.
    • Before port issues with :: fn NAME go out, two changes would raise the honoured share. Neither is in this PR, and each is a decision for issue_shape.t27:
      1. A form that also admits a new fn next to the declared one, for example :: fn NAME + or a second line :: new fn HELPER.
      2. Letting a region start at its doc comment.
    • Not measured: these PRs were written without a region contract, so this is the rate an author with no contract happens to keep one. It is not what a bee told its region would do.

    Reproduce: bench.ts, a scratch script that is not committed. It imports itemSpan from queen-batch-choose.ts, and hunksOf and editInside from queen-region-merge.ts, on trios branch queen-region-merge-check, against t27 master ea1bccd.

    🤖 Generated with Claude Code

  18. gHashTag commented on Oct 11, 2026

    @gHashTag
    OwnerAuthor

    Benchmark: claim rules (jam round, lane B), same production snapshot, before and after

    The rules are in specs/queen/holds.t27: send_back_claim, accept_claim and held_stage (see #9111, PR #9113). The trios wiring is gHashTag/trios#1894, base queen, not merged, with both flags off by default.

    The snapshot is the live Queen on 2026-10-11:

    • claims: 89 open issues in claimed at 12:55Z, read from /queen/public-tasks and each /queen/public-task drawer;
    • completed: 374 issues in completed at 12:47Z. The PR state of queen-<issue> was read at 13:25Z with a batched GraphQL query, 4 requests in all.
    rule (flag) input today: claimed until with the rule: freed must not touch (and did not)
    send_back_claim (TRIOS_QUEEN_RELEASE_DEAD_CLAIMS) 89 claims idle 1 h, then the files hold 1 h more 8 (8409 9000 9002 9011 9012 9023 9041 9084; ages 6-55 min) 81: 8939 at the ceiling, 16 running, 47 wait, 10 empty, 5 escalate, 1 registry
    held_stage (same flag) the 8 freed files held to 120 min, self-blocking from minute 60 files freed with the claim every row whose claim is live
    accept_claim (TRIOS_QUEEN_FREE_CLOSED_PR) 374 completed the 7-day window 2 (4374, 6414) 336: 121 open PR (incl. 4372: one closed, one open), 2 merged (5074, 6382), 213 no PR; plus 36 issues closed since
    accept_claim, after the planned close of stale bee PRs same same up to 121 (each issue whose every queen-<issue> PR ends closed unmerged) 2 merged, 213 no PR

    Why 213 accepts with no PR are not freed

    • Every one of them has a remote branch.
    • For 206 of them, the accepted head is not that branch's head and is not on GitHub (10 of 10 sampled; gh pr list --state all --head is empty).
    • queen-export-push.yml has failed on every run since 2026-10-08 06:15Z ("could not reach the export: The read operation timed out"), so the accepted work is still in the Queen's export.
    • Freeing these would throw that work away and spend a bee on each. The fix belongs in the export, not in the claim rules.

    Tests and checks

    • t27c test-report: 16 tests, 0 FAIL, 0 vacuous.
    • 6 negative controls, each FAILs by name.
    • trios tests/api/queen-claims-now.test.ts: flag off is unchanged, and each flag fires only in its own case.
    • api suite: queen 2080 pass / 6 fail; with the branch 2094 / 6, the same six failures.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions