Release to production: actors batch 3 and flags on by data, self-hosting, vault; then the actors blog post updated with production facts

The owner, 2026-10-10 (translated): "finish it to release, so everything is verified working, and in production." Then: "afterwards update the same blog post, it says a lot is not ready."

Ledger: #1729. Every step is merged only on green CI. Production means a merge into queen, which Railway deploys (about 1 min since the drain change). The flags are switched on one at a time, each with a check and a rollback.

A. Actors: close the post's "What this does not show" list

Gap the post names	Release step	How it is verified in production
"Nothing here is switched on in production yet"	Ship batch 3 (#1755, telemetry cost cut). Then switch the flags on by data, one at a time: telemetry first (`TRIOS_QUEEN_ACTORS_TELEMETRY=on`); then turn stop and the lane queue; waits as rows after their migration; adaptive concurrency only when telemetry shows ≥5 free review lanes over 24 h; the keyed dispatcher last	Deploy log flags; no new `[ERROR]`; per-flag readings in the Queen log
"reviewer, concurrency, turn-stop and dispatcher numbers are simulations"	Production readings after each flag: reviews/h, done p95, busy share, held lanes	Each reading is a comment on t27#7851 with its log window
"Telemetry's cost ... no production reading yet"	A production reading of CPU per message with telemetry on, plus the reviewer reading (23% busy against 11 reviews/h) answered from the telemetry itself	A t27#7851 comment
"simulation gate does not yet simulate keyed actors or the adaptive pool, and nothing runs it nightly"	A lane extends the sim gate with keyed actors and the adaptive pool, and runs it nightly	The gate's own runs
"FPGA figures are synthesis estimates ... nothing placed, timed or run on silicon"	Place and route `actors_hw.t27` on the Railway lab (never on the Mac). A run on the board needs the owner's `/bench approve`	PnR timing report; a bench receipt
"competitor figures are vendor defaults"	Stays: it is a statement of method, not a gap	n/a

B. Self-hosting (#1756, #1761)

hosting-mvp goes into queen after lanes A and B report production-ready. Then:

set TRIOS_HOSTING=on with an owner allowlist of host public keys;
run a rehearsal: two hosts join api.t27.ai, run shard jobs, and the distributed rows must equal the lab's rows;
the release trios-host-v0.1.0 goes from prerelease to full.

C. Vault (#1759)

vault-mvp goes into queen after the lane reports production-ready. Then three owner-only steps:

set the identity variable;
set TRIOS_VAULT=on;
import from Infisical and from the Railway age file, then dedup.

The smoke check proves injection without printing a value.

D. The blog post

After A–C, the same post (t27.ai/blog/actors-beat-the-loop-on-recovery-not-everywhere/) is updated in both languages:

what is now on in production, with the production readings;
what still is not, said plainly.

Nothing is claimed that has no reading behind it.

The owner's bench approval, 2026-10-10 ~19:10Z. In chat the owner gave the word /bench approve for running specs/queen/actors_hw.t27 on silicon, which is release step A, FPGA row.

What happens first. The approval is not posted on GitHub yet, because no bench-job issue exists. A lane first builds three things:

a JTAG top generated from .t27;
place and route plus timing on the Railway lab for xc7a200tfbg484-2, over 5 seeds;
the bench-job issue.

Then. The coordinator posts /bench approve on that issue, citing this chat word, and runs t27c bench-agent on die A (the owner's ALINX AX7203). The run uses SRAM only: no flash, eFUSE or BBRAM. One die is not a citable run, because independence needs three placements.

Batch 3, #1764. It is green except Biome and Typecheck, whose counts equal queen's (20 errors / 121 warnings, and 19 TS errors, all in files the PR does not touch). One api test failed: "a stopped process turn (real processes, real clock) > ignores its SIGTERM ...", at 14.2 s. Batch 3 does not touch turn stop, and the same test passed on #1754's CI. That failed job is being rerun once.

FPGA row update, 2026-10-10 ~20:00Z.

The first FPGA lane stopped on the compiler, not the hardware. No bench job exists yet, so the owner's /bench approve is still held for it.

t27c gen-verilog cannot instantiate device primitives or modules (t27#8660). Without a JTAG top, QueenActorsHw exposes 512 port bits, against 285 user IOs on xc7a200tfbg484-2.
A function brought in by use is called but never defined, and gen-verilog exits 0 anyway (t27#8769, filed by the lane).
The Railway lab cannot place anything either: it has the fbg484 chipdb but no yosys or nextpnr.

The owner's direction (translated): "use our t27b". A new lane gives t27b a Verilog lowering whose decisions are .t27 plans (specs/tri/t27b/), with no net growth of hand Rust in cli/t27b. Its steps:

the #8660 and #8769 repros accepted by yosys;
actors_hw.t27 lowered bit-exact;
a JTAG top generated from .t27;
place and route on fbg484 over 5 seeds;
the bench-job issue. Only then is /bench approve posted.

Telemetry has been on in production since 19:29Z (TRIOS_QUEEN_ACTORS_TELEMETRY=on; the deploy log shows telemetry=true and 0 [ERROR]). First 5-min window:

reviewer-worker: 4 live, 125 turns, busy 2.6%, turn p95 32–64 ms;
reviewer-intake: 163 turns, busy 0.2%, max mailbox depth 5;
0 crashes, kills, dead letters or storms.

A one-hour reading is being collected for t27#7851.

Release progress, 2026-10-10 20:45Z.

A. Actors in production

Batch 3 shipped: queen(actors): ship batch 3 - telemetry's own cost cut 27-33% (#1755), decisions byte-identical, flag still off #1764 merged at 19:25Z as 82bcc93. Its only red checks are Biome and Typecheck, with counts equal to queen's. One flaky real-process turn-stop test passed on rerun and is filed as flaky: 'a stopped process turn (real processes, real clock) > ignores its SIGTERM ... nothing survives' failed once on CI at 14.2 s #1766.
Flag 1: telemetry on, since 19:29Z. The first production reading is on t27#7851 (comment 6101940911):
- 14 real reviews an hour;
- worker busy median 9.3%, max 24.9%;
- 1 kill and 1 restart, from a 300 s model-call timeout;
- 0 storms, 0 dropped messages, 0 [ERROR].
Flag 2: turn stop on, since 20:38Z (TRIOS_QUEEN_TURN_STOP=on). The deploy log shows turnStop=true telemetry=true, with 0 [ERROR]. A one-hour check is running: reviews per hour, kills, and killed calls aborted within the grace.
Unexplained same-commit redeploy at 20:15Z (4890af61). No flag changed; the cause is unknown and is being watched.

B. Self-hosting

Lane B is merged into hosting-mvp (feat(hosting): slice 1b lane B - sybil-proof credit, pay-for-work epoch statements, live Postgres (Closes #1770) #1771): sybil rules, signed epoch statements and a live-PG store. Details are on Self-hosting slice 1b: one-command join, the full lab verdict, OS sandbox, sybil-proof credit, pay-for-work TRI statements, live Postgres #1761.
Lane A (release assets, full verdict, sandbox) is still running.

C. Vault: still running. It now has the live local PG for its pglive tests.

FPGA: the t27b Verilog lane is running. The bench approval waits for its bench-job issue.

Part B, self-hosting: in production and rehearsed, 2026-10-10 22:10Z.

Ship. #1776 merged into queen at 21:29Z as dfb6d97. Its CI shows two red checks:

Biome: 20 errors, the same as queen. Warnings are 122 against queen's 121; the +1 is a complexity warning in trios-host.ts:149, filed as trios-host: split the function at trios-host.ts:149 (Biome noExcessiveCognitiveComplexity, +1 warning vs queen) #1777.
Typecheck: 19 errors, the same as queen.

pglive passed 145 of 146 tests on the first run, including every hosting test. The one failure is the known race in the PgLink fence test, which also fails on queen (filed as #1778), and it passed on rerun.

Off means off, checked after the deploy and before the flag: /hosting/ledger and /hosting/ledger/epochs answered 503, the log had no hosting line, and there were 0 [ERROR].

On, at 21:44Z: TRIOS_HOSTING=on with TRIOS_HOSTING_ALLOWLIST naming three owner host ids (public ids only).

The rehearsal against https://api.t27.ai:

Two hosts ran on the owner's Mac from the release binary trios-host-v0.1.0, checked against its sha256. Each fetched and verified zig, t27c and t27b against the pins and ran the planted job inside sandbox-exec: connected false, wrote outside its directory false, isolation job-dir-only. Both registered as tier owner.
The owner key submitted 24 specs at t27 cf42584c9, as 48 jobs: the reference half and the t27b half of each spec.
Result: 24 of 24 rows equal the lab's. It took about 5 minutes.
Public ledger: 96 credit rows, 48 mTRI per host, 0 for the submitting owner key. off_chain: true, transferable: false, token_value_claimed: false, settlement disabled (threshold 1000 mTRI, chain of record TON).

Sybil rules on at 22:07Z (TRIOS_HOSTING_SYBIL=on): canary 10%, probation 10 jobs. The ledger survived the redeploy with 3 hosts and 96 rows, so the Postgres store holds in production.

Left for the owner: TRIOS_HOSTING_LEDGER_KEY, the 64-hex Ed25519 seed. Until it is set, epochs stay open and unsigned (ledger_key: missing). Agents never set keys.

Merge queen (with the vault) into hosting-mvp for the second hosting …

Release status, 2026-10-11 03:35Z.

Part	State
A. Actors	Live: batches 1–3, `TRIOS_QUEEN_ACTORS_TELEMETRY=on` and `TRIOS_QUEEN_TURN_STOP=on`. The escalation fix is live too (#1809, plus #1811 for the orderly stop). The remaining flags (lane queue, waits as rows, adaptive pool, keyed dispatcher, actor events) stay off until the data asks for them.
A. FPGA	`actors_hw` was lowered by t27b (t27#8820) under constraints generated from t27 (#8857), placed and routed over 5 seeds, and run on die A (t27#8858): verdict 0xA5A536FD, signature 0x3DED69B1, 77,794 decisions equal the card. It is not yet citable: dies B and C are still owed.
B. Self-hosting	Live since 21:29Z: `TRIOS_HOSTING=on`, `TRIOS_HOSTING_SYBIL=on`, with an owner allowlist. The production rehearsal produced 24 of 24 rows equal to the lab's. Ship 2 (#1796) added the network job card (`TRIOS_QUEEN_NETWORK_JOBS` off), `/queen/public-net-credits`, and the operator rule imported from `network/*.t27`. The owner still has to set `TRIOS_HOSTING_LEDGER_KEY`.
C. Vault	Deployed and off since #1780: `/vault/*` answers 503, and `age` 1.2.1 is in the image. The owner's steps are pending: the identity, the two public values, then the imports.
D. Blog	trinity#1603 is merged and auto-published, in English and Russian. It covers the production readings, the timer defect and its fix with the benchmark correction, the silicon run, and a "What this does not show" list that is still true.

Self-heals this round:

The turn-stop escalation defect. A Bun timer firing 1–3 ms early meant nothing asked the card again. It was found by CI under load, with up to 27 of 100 escalations lost, and fixed with EarlyClock tests.
The flaky pglive fence test (flaky pglive: 'PgLink at its edges > an acknowledgement the store refuses fences the node at once' races on CI #1778, fixed by test(queen): pglive PgLink edge test asserts the fence, not which racing write is logged (Closes #1778) #1808).
The Bun segfault in pglive (pglive: Bun 1.3.x panics (Segmentation fault at 0xFFFFFFFFFFFFFFF8) after queen-actors-pg-edges-live runs, failing the job #1813) is under investigation in its own lane.

Watching: Biome's error count on queen went from 20 to 22 with the zcode series (callback-server.ts); that is not from this release.

@dmitrii-f-t27

@dmitrii-f-t27, a proposal for merges into trios queen. queen is production: every merge deploys straight to Railway (Queen, api.t27.ai). Right now two flows merge there, this release loop and your PRs (#1808, #1809 and zcode/l3 #1815–#1818). Today #1809 and our #1811 fixed the same timer defect at the same time.

A simple order that would prevent that:

Before a merge into queen, get fresh CI on the current queen. Tests / server-api and server-pglive must be green, and Biome and Typecheck no worse than queen. Biome went from 20 to 22 errors after the zcode series, with one of them in callback-server.ts.
One merge at a time, then check the deploy: /health answers 200 and the log has 0 [ERROR]. Railway labels a normal redeploy CRASHED, because the server exits with code 3 on SIGTERM.
New behaviour goes behind an off flag.
A one-line note here in Release to production: actors batch 3 and flags on by data, self-hosting, vault; then the actors blog post updated with production facts #1763: what was merged, when, and which flag.

If that works for you, a reply here is enough.

Self-hosting ledger signed in production, 2026-10-11 05:00Z.

The key. The owner set TRIOS_HOSTING_LEDGER_KEY himself, through stdin; no agent saw it. /hosting/ledger/epochs now answers:

signed: true and ledger_key: set;
queen_key: 1d90f33e31d90296f6f05652b0e78e6d4c39d533dae62c91ba68b14d5d0a25e1;
chain_of_record: ton, settlement: shut, transferable: false.

Epoch 0 (2026-10-10) is closed: 4 leaves, 96 mTRI, root fa10ac07…, Ed25519 signature.

Host-side check, with the production code:

Host A 40ab49ee13569e82: executor_fee 20 mTRI and verifier_fee 28 mTRI, both ok. The total, 48, equals its ledger balance.
Host B: ok as well.
A wrong Queen key is refused with exit 1.

Release trios-host-v0.1.1 (from queen 8cd6df7, not marked latest) carries the statement command. Version 0.1.0 was built before lane B and lacked it. https://github.com/gHashTag/trios/releases/tag/trios-host-v0.1.1

The darwin-arm64 sha256 is 63dc5887…65cee7, and the download checks out.
No trios workflow triggers on a release or a tag; all 27 active ones were checked.
Both production hosts on the owner's Mac now run 0.1.1 and re-registered.

Seen on the way: across about 20 Queen redeploys overnight, each host re-registered 24 to 28 times, up to incarnation 28. Every deploy lapses their lease, and they recover as designed.

Waiting on the owner: the vault (TRIOS_VAULT_IDENTITY through stdin, plus the two public values).

spec(hosting): feed.t27 -- the network feeds itself: which master hea…

spec(queen): the simulation gate plays keyed actors, two bee dispatch…

Queen throughput (t27#9021), first fix live, 2026-10-11 12:10Z.

What shipped: #1880 (t27#9047, holds.t27) as efae3e5, deployed at 12:09:42Z. A running dispatch holds only its issue's CURRENT Boundary, never more than it stored at dispatch. It holds nothing once the issue is closed or its Boundary is removed. This is a pure defect fix: a closed issue's work is never reviewed or published.

Before, from the replay of the 10:57:58Z data:

Closed #8660's dispatch still held bootstrap/src/compiler.rs and blocked 125 candidates, 122 of them blocked by nothing else.
With the fix, fileConflict falls from 232 to 97 and 135 candidates become choosable. But 122 of those 135 name compiler.rs, so the first pick takes the whole file again.

Live ticks:

tick	allowed	fileConflict	claimed	running	idle
10:57Z	no	231	51	7	63
11:05Z	yes	237	61	23	47
12:10Z (after the fix)	yes	228	84	18	52
12:11Z	no	229	89	19	51

fileConflict stays at about 229 because a new bee now holds compiler.rs whole, as the replay predicted.

What moves that number: region-level ownership inside hot files (item e). Of the 125 compiler.rs-blocked candidates, 82 name at least one compiler.rs function, and a greedy disjoint batch admits 32. Derived-free paths through golden_chain.t27 gc_pair add about 13. That work is in progress, behind flags, with a replay benchmark.

test(queen): the simulation gate plays keyed actors, two bee dispatch…

Batch dispatch: on 12:42:52Z, rolled back 13:06:50Z

TRIOS_QUEEN_DISPATCH_BATCH=on and TRIOS_QUEEN_DERIVED_FREE=on went live at 12:42:52Z (deploy 21490f45). Batch was set back to off at 13:06:50Z (deploy 1c0b665d, SUCCESS). Derived-free stays on.

Readings (public /queen/status)

	before (12:39-12:41)	batch on (12:59-13:04)
running	15-16	18-24
idle lanes of 70	54-55	46-52
dispatchedLastHour	61-65	61-64 (window still mixed)
fileConflict	226	214-215
claimed	87	94-99
completed	373	341-342 (18 issues closed by hand, see below)

Batches per round, 31 rounds: 55 picks in total; 11 rounds picked more than one, and the largest was 11.

Why it was rolled back

The batch re-dispatched issues it had just dispatched:

Round 19 (12:58:44) queued 11 runner orders.
Round 20 (12:59:17) re-picked 10 of them.
t27#8396 was queued 6 times between 12:58:44 and 13:01:33, and 8398, 8404 and 8407 were queued 4 times each.
queen_dispatch is keyed by issue, so each re-dispatch overwrote the previous order (new conversation id, claimed_by cleared).

For the batch to pick an issue, queend's first answer must have called it not first or held by. So the next round's queend did not see the orders just written as live claims. The root cause is being traced. The fix comes with a two-round test: round 2 must see round 1's orders as live. Batch stays off until then.

Also done

18 open t27 issues whose closing PR had merged into master were closed by hand, each with a comment linking the PR. Each was re-verified through the API first: the PR is merged and lists the issue in its closingIssuesReferences. completed fell from 372 to 354 when they closed.

Navigation Menu

Repository navigation

Release to production: actors batch 3 and flags on by data, self-hosting, vault; then the actors blog post updated with production facts #1763

A. Actors: close the post's "What this does not show" list

B. Self-hosting (#1756, #1761)

C. Vault (#1759)

D. The blog post

Batch dispatch: on 12:42:52Z, rolled back 13:06:50Z

Readings (public /queen/status)

Why it was rolled back

Also done

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Release to production: actors batch 3 and flags on by data, self-hosting, vault; then the actors blog post updated with production facts #1763

Description

A. Actors: close the post's "What this does not show" list

B. Self-hosting (#1756, #1761)

C. Vault (#1759)

D. The blog post

Activity

gHashTag commented on Oct 10, 2026

gHashTag commented on Oct 10, 2026

gHashTag commented on Oct 10, 2026

gHashTag commented on Oct 10, 2026

gHashTag commented on Oct 11, 2026

gHashTag commented on Oct 11, 2026

gHashTag commented on Oct 11, 2026

gHashTag commented on Oct 11, 2026

gHashTag commented on Oct 11, 2026

Batch dispatch: on 12:42:52Z, rolled back 13:06:50Z

Readings (public /queen/status)

Why it was rolled back

Also done

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions