Skip to content

Queen actors night loop 2026-10-10: weak spots after the seven, continuous improvement (ledger) #1729

Open
Open
Queen actors night loop 2026-10-10: weak spots after the seven, continuous improvement (ledger)#1729

Description

@gHashTag
gHashTag
opened on Oct 9, 2026

The owner's night loop, started 2026-10-10 20:07Z. The cron fires every 15 min with the owner's standing instruction (translated):

  • find the weak spots of the actor-model work and study competitors;
  • make a decomposed plan and implement all of it;
  • end with a report and three options for cooperating on the next loop;
  • keep improving continuously, update skills with what was learned, and never let a new tick break earlier work;
  • be self-critical, look for anomalies and self-heal;
  • extend tri CLI with new commands;
  • track every task as an issue, push coverage to 100%, auto-fix, and merge when green.

Earlier epic: #1712. Its seven items are merged into actors-next; lane 5 (#1726) and the three-way decision replay (#1728, t27#8317) are in flight.

Tick protocol (so a tick never breaks earlier work)

  • This issue is the ledger. Every tick appends one comment: what it checked, what it started, merged or healed, and what it found wrong.
  • Ownership. One task, one issue, one branch, one worktree. A tick starts a task only if no open branch or PR for it exists (gh pr list, git ls-remote).
  • Concurrency. At most 4 agents at once, because the host load reached 40 with 7 and timing tests then flaked. Load is checked first.
  • Merge rule. Merge only green: the spec checks required on t27, and the full api suite equal to the recorded baseline on trios.
    • actors-next goes into queen (production) once, after lane 5 and the replay are in and the suite equals the baseline. All new behaviour stays behind flags that are off by default.
  • Anomaly checks every tick: a red PR, a stale agent (no file change in 45 min), actors-next suite drift, disk under 3 GB, load over 30, a cron that did not fire.
  • Healing: rebase, rerun, restart the lane from its worktree, or file an issue.

Decomposed plan (weak spots found by the seven lanes and the study)

  1. Ship. Merge actors-next into queen. Then turn flags on one at a time, by data:
    • telemetry first;
    • then waits=rows and turn stop;
    • adaptive concurrency only when the telemetry shows ≥5 free review lanes over 24 h.
  2. Deploy drain. Every deploy still freezes the swarm for 1800 s, and no lane shortened it. With durable jobs and waits and stoppable turns, a spec can bound the drain by in-flight turns and abort past a deadline.
  3. Reviews as process isolates. A step that ignores its abort still runs to its own end (lane 2 gap).
  4. A semaphore wait for review lanes. Adaptive concurrency loses with scarce lanes (lane 3: 3 lanes, 360 vs 429).
  5. Waits.
    • a GitHub workflow_run webhook into /queen/waits/resolve, because detection is 83 s, worst case 240 s;
    • move the reviewer backoff and supervisor restart timers to rows;
    • prune ended rows.
  6. PgLink.
    • piggyback the acknowledgement on the next read (6.2 back to ~4 SQL per round trip);
    • rejoin after a fence without a process restart.
  7. Keyed.
    • the claim holder is the actor pid (needed for a second dispatcher);
    • supervise the admission actor;
    • extend the call-cycle check beyond slots 0-63.
  8. Telemetry.
    • an overhead number on a quiet host (the night window);
    • fix the flaky "a process takes its messages in order" test (slice versus virtual clock);
    • make the 200 000-turn flood test independent of load.
  9. Coverage to 100%.
    • every queen spec with 0 vacuous tests, on both the reference and t27b;
    • line coverage of the actor runtime modules (queen-actors*.ts) measured and raised to 100%, or each uncovered line justified in a list.
  10. tri CLI. New commands, each with its decisions in a .t27 spec and only thin glue:
    • tri actors (metrics from /queen/actors/metrics);
    • tri lab receipt (t27b verdicts for a set of specs);
    • tri lanes (worktrees, branches and PRs of a loop).
  11. Skills and memory. Write down the traps learned:
    • queen/… branches are impossible next to queen;
    • the repo-local .githooks;
    • SIGPIPE from | head kills zig;
    • Closes does not fire on a non-default base;
    • load-sensitive timing tests;
    • shared node_modules through symlinks;
    • keep both lanes' PIN lines.
  12. Compiler defects found on the way, each its own t27 issue (see #5980):
    • gen-c 1 << 32 as a C int;
    • generic dividers for constants;
    • __mul_noop on narrow operands;
    • literal-bound loops;
    • modules without data ports.
  13. FPGA hardware profile (actors_hw.t27): in flight.
  14. Closing. The queen(actors): a killed turn stops - AbortSignal into model and git calls, process groups killed whole #1713-feat(queen): actor telemetry per kind and a replayable decision log - telemetry.t27, GET /queen/actors/metrics (Closes #1718) #1725 issues stay open because Closes only fires on main. Close each one by hand with its merge link once actors-next is in queen.
  15. Final report with three options for cooperating on the next loop.

Activity

  1. gHashTag commented on Oct 9, 2026

    @gHashTag
    gHashTag
    on Oct 9, 2026
    OwnerAuthor

    Tick 1, 2026-10-10 20:07Z

  2. gHashTag commented on Oct 9, 2026

    @gHashTag
    gHashTag
    on Oct 9, 2026
    OwnerAuthor

    Tick 2, 2026-10-10 ~20:45Z

  3. gHashTag commented on Oct 9, 2026

    @gHashTag
    gHashTag
    on Oct 9, 2026
    OwnerAuthor

    Tick 3, 2026-10-10 ~21:20Z: self-healing on the ship PR

  4. gHashTag commented on Oct 9, 2026

    @gHashTag
    gHashTag
    on Oct 9, 2026
    OwnerAuthor

    Tick 4, 2026-10-10 ~22:05Z

    Finished since tick 3:

    Changes to the protocol:

    Wave 2 started, 4 agents:

    • plan item 2: bound the deploy drain;
    • the per-message cost regression and 5 flaky tests;
    • plan item 10: tri lab receipt, tri actors, tri lanes;
    • plan item 4: a review lane semaphore.

    The blog draft (trinity#1583) is waiting for the ship merge.

  5. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 5, 2026-10-10 ~13:00Z: anomalies and self-healing on the ship PR #1730

    1. Security finding, fixed. /queen/actors (telemetry counts and the logged card decisions) was mounted with no guard. The route-guard audit in tests/api/routes/route-guard.test.ts caught it on CI. fix(queen): /queen/actors is guarded by the trusted-origin check, like needs-you #1735, merged into actors-next (6a68c93), puts it behind requireTrustedAppOrigin() like /queen/needs-you and re-measures the audit's pins: 62 mounts, 20 guarded sub-apps, 39 /queen mounts with 13 wrapper-guarded.
      • Local runs missed it because my sparse clone lacked trios/tools, so the test errored at import. Now included.
    2. History rewrite by another account. At 00:44Z dmitrii-f-t27 rewrote actors-next and actors-next-2 from 55fb49f to 281495d. The tree is identical (c7bedeebeca3); only the commit authors changed, to "Dmitriy Vasilev". Nothing was reverted. My fix branch was rebased onto the new history.
    3. CI-only test failures (ubuntu, bun 1.3.6):
      • Three tests in queen-bee-actors-round, one in queen-review-actors-start and one in queen-actors-pg-beat-live use jest.useFakeTimers() and hang or fail only on CI.
      • A real-process fixture writes no grandchild pid under dash.
      • The rest is a cascade (WaitForHelper reads Date.now() delta = 0; SIGTERM shutdown, Klavis, waitForReady) consistent with fake timers leaking after a timed-out test.
      • What does not reproduce: none of this fails locally on macOS with bun 1.3.6 or 1.3.11, even in CI's file order. Docker hangs here, so no Linux reproduction.
      • The fix is assigned to its own lane (branch queen-ci-linux-timers into actors-next): injected VirtualClocks instead of global fake timers, a dash-safe fixture, and a file-level afterEach restore.
    4. Biome and Typecheck at 6a68c93 are still below queen (20 against 29, and 19 against 75).
    5. Weekly limit. All four wave-2 agents died on the weekly limit. Three are resumed after the reset (drain bound, cost and flakes with the CI files excluded, review-lane semaphore). The tri CLI lane waits for a free slot.
  6. gHashTag
    added 2 commits that reference this issue on Oct 10, 2026

    spec(queen): review_lanes.t27 - a review with no lane free waits for …

    3c0ae4d

    spec(queen): drain.t27 - a draining Queen waits for her own bees, sto…

    a59676a

  7. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 6, 2026-10-10 13:40Z: SHIPPED. #1730 merged into queen as 56c5df7 at 13:38:37Z. Railway is building trios-agent-server, and the drain will hold up to 30 min. Every new behaviour is behind a flag that is off.

    Checks at merge: 18 pass, 2 fail.

    • The 2 failures are Biome and Typecheck. Both were red on queen before this PR and are now lower than there: 20 against 29, and 19 against 75.
    • Every Tests/* check passed, as did Actor simulation.

    How the CI-only failures were healed (#1741, issue #1738):

    • Fake timers. jest.useFakeTimers() tests stalled on ubuntu with bun 1.3.6. A timed-out test kept the fake clock installed for every later file, which caused the cascade. Those tests now drive an injected VirtualClock, through new clock parameters that default to the real clock: startBeeDispatcher, startReviewerActors, nodeBeat.
    • Logger mock. task-queue-service.test.ts restored a mock.module'd logger and pg with the mock itself. That is platform-independent and reproduced on macOS.
    • Left alone. chat-service.test.ts mocks four modules and never restores them. It breaks nothing today but is a latent hazard.

    Wave 2, merged into actors-next-2 (the next ship batch):

    In flight: cost regression plus flakes, tri CLI commands, keyed hardening.

  8. gHashTag
    added 2 commits that reference this issue on Oct 10, 2026

    tri lanes: a loop's lanes, each judged; red, conflicting and stale la…

    14c41a1

    tri lanes: a loop's lanes, each judged; red, conflicting and stale la…

    a162313

  9. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 7, 2026-10-10 14:05Z.

    Deploy. 56c5df7 is still DEPLOYING on trios-agent-server (deployment 9fcef88d, started 13:38:40Z). This is the bee drain: the old container waits up to 1800 s for its runner's bees. A watch is armed for the outcome.

    Closed by hand, because Closes fires only on main:

    #1737 and #1740 stay open until actors-next-2 ships.

    New: item 16, actor events for the kanban (#1744). The event shape is agreed on the issue with the session that builds gHashTag/t27#8615 (actor_view.t27):

    • Task-carrying deliver: offer, result, cancel, and the exit or DOWN that ends a task. These are unsampled.
    • Every other deliver: aggregated per (from, to) pair per window into count, dropped, max_depth, first_seq and last_seq.
    • Fields on every event: seq, at, kind, from_pid, to_pid, and task_ref. Never payloads.
    • Delivery: the events.t27 bus, served at GET /queen/public-actors?since=, behind TRIOS_QUEEN_ACTOR_EVENTS, which is off.

    A lane agent is on it, on actors-next-2.

    Anomaly: the disk filled. At 13:47Z the Mac had 116 MB free, and every Bash call failed with ENOSPC. Healed from the terminal:

    • pruned the zig cache entries older than 60 min: 2.3 GB down to 534 MB, since every card build adds to it;
    • removed 9 finished agent worktrees whose branches were merged and pushed (~300 MB each).

    Free space is now about 4.2 GB. The cache cleaner the notes describe exists only on the old machine; this Mac has none. Installing one is a standing config change, so it waits for the owner. Running lanes were told to rerun anything that failed in that window.

    Self-correction. My quiet-host telemetry comment on t27#7851 dated the run 2026-10-10. The run was on 2026-10-09; the comment is fixed.

    Blog. trinity#1583 now says the batch merged into production behind off flags. It also gives the quiet-host telemetry figures: +3.3% CPU per message, and the base rose from 3.7 to 4.5 µs, which #1743 is chasing. Its "Website checks" are red on the Queen viewport contract, the kanban card-slot geometry. That check also fails on an unrelated PR today (fix/lanes-never-count), so it is not the post; re-running.

    In flight (4): cost and flakes (#1743), tri CLI, keyed hardening, actor events (#1744).

  10. gHashTag
    added a commit that references this issue on Oct 10, 2026

    spec(queen): keyed_guard.t27 - the claim holder is a process boot and…

    b8100e8

  11. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 8, 2026-10-10 14:20Z.

    The deploy is live. 9fcef88d / 56c5df7 reached SUCCESS. The new container served from 14:13:04Z. Its log reads Queen reviewer starting as actors ... workers=4 turnStop=false telemetry=false, so every flag is off, as intended. It shows no new error class. 15 "could not read the diff of finished work" lines are fewer than the previous container's 47.

    Found in that log: an old defect, now fixed (#1747, #1748, merged into actors-next-2). The Queen's t27 root checkout cannot reach origin/master. About 60 untracked specs/port/*.t27 block checkout -B, so the Queen serves a local commit, c925f20804. The entrypoint's comment claimed untracked files never block. The fix moves exactly the untracked paths that the target tracks into .git/entrypoint-set-aside/<time>/ and retries. Nothing is deleted. A local reproduction failed before the fix and passes after it. The fix ships with the next batch.

    tri CLI lane: done. Three commands are merged into t27 master, each with its own spec and with negative controls:

    • tri lab receipt (#8601);
    • tri actors (#8614);
    • tri lanes (#8623).

    An orphan check that #8601 turned red was fixed in #8631. A compiler bug was filed as t27#8630: an item-level use pulls in names that appear only in strings, and same-named items vanish.

    • Gap: tri actors gets 403 from production. The /queen/actors* routes are guarded by requireTrustedAppOrigin() since fix(queen): /queen/actors is guarded by the trusted-origin check, like needs-you #1735, and the command reports that as "telemetry off or unreachable". This becomes item 18: give operators a real credential path and make the message name the HTTP status. Spoofing an Origin header is not a fix.
    • Not done: tool cards for the three commands in specs/tools/tri/.

    Self-correction: my disk cleanup broke zig's cache. I deleted old ~/.cache/zig/o archives and left their manifests in h/. After that, zig test failed with failed to parse archive: FileNotFound until the tri lane removed the stale manifests. Running lanes were told to rerun. The rule now reads: delete zig's cache whole or not at all.

    Disk, root cause. The disk was not the grower; swap was. Four agents on this 8 GB Mac swapped 4.4 GB, and swap shares the APFS container. The parallel-agent limit drops from 4 to 3, with at most one heavy job at a time (a cargo build or a full api run). 15 finished trios lane worktrees were removed after checking that their commits are on remote branches.

    Item 17: trinity#1588, the kanban card overflows its slot on live data. The Queen viewport contract runs against the live backend, so every site PR is a coin flip. That includes the blog post trinity#1583. The lane starts when a slot frees.

    In flight (3):

  12. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 9, 2026-10-10 14:40Z. Two lanes merged into actors-next-2.

    Cost and flakes, #1746 (issue #1743): a message costs 1.55 µs, not 4.47.

    • Where the cost went. No single lane merge stands above the noise. The cost predates the lanes:
      • 44% of the ring's time went to looking a card up on every call, by building and hashing a 100+ character key;
      • slot_of ran three times a message for a slot already known at spawn.
    • Fixes. Cards are found by file name and kept once per process, and the slot is kept from spawn. Card calls fall from 12 to 8 per message.
    • Result. 1.55 µs is 2.4× below the pre-lane base of 3.69 µs.
    • Correction to my t27#7851 note. Lanes 2, 6 and 7 were not the cause.
    • Flakes fixed, load-independent now:
      • "takes its messages in order", via a two-turn virtual-clock yield;
      • the 200k flood under coverage;
      • the sim gate next to a real timer: the seeded clock is passed in, and performance.now is no longer replaced process-wide.
    • Not verified: the pglive fence test was not run (see below).
    • Follow-up. Telemetry's absolute cost is unchanged at about 0.15–0.25 µs, so on the cheaper base it is +8–13%. That is at the edge of telemetry.t27's +10%. It becomes item 19, and a lane is on it.

    Keyed hardening, #1749 (issue #1745, spec t27#8634): all three lane-7 gaps closed.

    • Claim holder. It is now the process name, a boot nonce and the pid. Duplicate pickups by two dispatchers fall from 273 to 0. Pid alone still gave 16, because two replicas reuse pids.
    • Supervision. Admission runs under a one_for_one supervisor with a board that survives a crash. Three crashes in 8 h give 352 done; before, the dispatcher died after the first crash and finished 63.
    • Call cycle. The chain now carries pids, so an A→B→A cycle past slot 100 is refused at once instead of timing out at 5 s.
    • Cost. A call's round trip costs no more than before.
    • Verification. 32 spec mutations, all caught. The api suite fails 5 tests against the base's 10, a subset.

    Not verified: pglive, for both lanes. This Mac has no Postgres. Docker Desktop has not answered since 07:19Z, and pglive runs in CI only on PRs into queen. Both lanes' pglive tests therefore first run when actors-next-2 goes to queen. That ship PR must show them green before merge.

    New issue: item 18 is t27#8651. tri actors turns a 403 into "telemetry off"; it should name the status. Operator access to /queen/actors* needs an owner choice: an operator token, or the lab/ssh path. Spoofing Origin is not an option.

    Blog trinity#1583 now carries the profile's answer. It also drops my wrong guess about lanes 2, 6 and 7.

    Housekeeping. The tri lane's worktree was removed after its branches were checked on the remote; the new tri is saved to the scratchpad and is in use. tri lanes --base actors-next-2 reports no red, conflicting or stale lane. Disk is at 2.6 GB, swap is stable, and the agent limit stays at 3.

    In flight (3):

  13. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 10, 2026-10-10 15:20Z.

    Actor events, #1750 (#1744; specs t27#8642 and t27#8652): merged into actors-next-2, behind TRIOS_QUEEN_ACTOR_EVENTS (off).

    • Where the events go. They have their own actors stream on the same log, so actor traffic does not wake every follower of queen.
    • Kind codes follow actor_view.t27: spawn 0, restart 1, exit 2, down 3, deliver 4, pair count 5.
    • Pids are decimal strings in JSON.
    • Public route. GET /queen/public-actors?since= has a test that no title, body, args or payload ever reaches it.
    • Replay. ?replay=mvp serves a 24-row stand-in for now. The #8648 lane adds trace_row and trace_len to specs/network/mvp.t27 in this shape.
    • Cost. None measurable with the flag off, about +0.43 µs a message with it on. The first draft capped open pairs at 256 and cost +25%; t27#8652 raised the cap to 4096.
    • Mutants. 30 across the specs and the runtime, all caught.

    Self-heal: master's t27b ratchet was red because of us. #8634 added specs/queen/keyed_guard.t27 without its expectations row, which shows as UNLISTED. Fixed by t27#8665 (issue #8664), one row, with auto-merge on. The same run's MOVED entry for specs/compiler/lexer.t27 is not ours. The other actor specs (drain, review_lanes, actor_events, actor_view, actors_hw and the rest) all have rows.

    Host: disk 0.5 → 6.6 GB. The owner said "clean it yourself". The cleanup removed:

    • node_modules in repos idle 2–5 weeks: 999-multibots-telegraf, trinity, trinity-wt-explorers and leela; they come back with one install;
    • the npm cache and unavailable simulators;
    • 3 finished agent worktrees, all clean and pushed;
    • two shells left by the coverage lane that had been waiting 16 h for a log line that never comes.

    Personal data, VMs and Docker were left for the owner. The peer session also stopped its Mac-side nextpnr runs, which were part of the swap.

    Ship 2 under way. actors-next-2 → queen conflicts with #1741 and #1735 in four files:

    • queen-bee-actors-round.ts;
    • queen-bee-actors.ts;
    • queen-review-actors.ts;
    • the route-guard test.

    A ship lane merges queen into actors-next-2 on queen-ship-2, keeping both sides, and opens the ship PR. Its CI is the first-ever run of the keyed_guard and fence pglive tests, because this Mac has no Postgres. The coordinator merges into queen only on green.

    Queued: item 20, #1751, the job runner for the t27b ledger steward card (t27#8656). It waits for the owner's word on credential scopes and token rights.

    In flight (3):

    • telemetry overhead (item 19);
    • the kanban clamp (trinity#1588);
    • ship 2.

  14. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 11, 2026-10-10 16:10Z. SHIP 2 is live.

    #1754 merged into queen as 3654fdd at 16:03Z. It merges queen into actors-next-2 first (#1753). The four conflicts from #1741 and #1735 were resolved by keeping both sides, and the route-guard counts were recomputed (63 mounts, 20 guarded sub-apps, 17 public-read).

    The deploy took about 1 minute instead of 30–34. The owner applied the drain recipe at 15:27Z:

    • TRIOS_BEE_RUNNER_SECONDS was removed from trios-agent-server;
    • TRIOS_BEE_RUNNER_SLOTS was set on the runners. runner-1 logs slots=36; runner-2 is still redeploying.

    The new Queen container logs no Bee runner starting, so it runs no bees and has nothing to drain.

    #1747 is verified in production. The new container logged:

    [entrypoint] moved 30 untracked path(s) the target tracks into /workspace/t27/.git/entrypoint-set-aside/20261010T160410Z
[entrypoint] checkout ready: d1a3280f3 on master

    d1a3280f3 is the current origin/master, so the Queen no longer reads the stale local commit c925f20804.

    Flags, as logged: turnStop=false laneQueue=false telemetry=false actorEvents=false. The log has 0 [ERROR] lines. The 21 "committed-file diff failed" INFO lines are the old class (bee branches absent from the root checkout), not new.

    Closed by hand: #1737, #1740, #1743, #1744, #1745, #1747.

    Anomaly healed: the Mac's process limit. Monitors and shells died with fork failed: resource temporarily unavailable, because the user had 1342 processes against a per-user limit of 1333.

    • Cause: 955 of them were zombies of the Pioneer DJ firmware daemon FwUpdateManagerd, up 10 days.
    • Fix: killing the daemon reaped them all. 385 processes are left, and 0 zombies.

    The 30-minute deploy monitor had died silently from this; its output file showed the fork errors.

    New: disaster recovery, t27#8675. The owner reports that the Railway and GitHub accounts may be cut off, and chose an off-Railway bucket for backups.

    • Phase 0 lane: an inventory of all 14 projects with variable names only, the sources of the 10 services uploaded from the CLI with no git source, and a cost and duplication review.
    • Owner's own action: the variable values of project 999 were exported, encrypted with age under the owner's passphrase. The file is 100 KB and its values were never seen by an agent.
    • Next owner actions:
      • an R2 bucket, with its keys put into the shared variables;
      • a read-only GitHub token for nightly bundles of the 215 repos.

    In flight:

    Next: item 20, #1751. The owner approved it; it waits for the token scopes.

  15. gHashTag commented on Oct 10, 2026

    @gHashTag
    gHashTag
    on Oct 10, 2026
    OwnerAuthor

    Tick 12, 2026-10-10 18:30Z, after a session restart.

    The restart. The previous Claude Code process exited. Its running agents died: the vault lane, plus a few wait loops. The session-only cron 6a1aefdc is gone with it.

    Anomaly: load 38, swap 7.3 GB. Three causes:

    • Two orphaned hung test binaries (ppid 1): t27c-test-report-gf16_accel_tb/spec_tests had run 4 h 49 min and axi4_tb 2 h 52 min. Both were killed.
    • A peer session's runaway run. A gh ... --body "...t27c test-report..." executed the command, because backticks inside double quotes are command substitution in zsh. A whole-corpus run went on for an hour until clock_domain_tb hung. The peer stopped it. Rule recorded: bodies only through --body-file - <<'EOF'.
    • The GitHub API limit (5000/h, shared) was exhausted at 18:18Z by parallel lanes polling CI. The new lanes poll every 2 min or more from one loop.

    Shared host rules for every lane now live in the scratchpad's HOST-RULES.md:

    • one heavy job at a time, through an mkdir lock;
    • perl alarm timeouts on every test run;
    • the body rule above;
    • the API budget;
    • the zombie check.

    Lanes now (3):

    Waiting: the blog post trinity#1583 is merged with the latest main. A watch merges it only when checks, website gate and report are green.

  16. gHashTag commented on Oct 11, 2026

    @gHashTag
    gHashTag
    on Oct 11, 2026
    OwnerAuthor

    Tick, 2026-10-11 06:30Z. Self-heal chain closed; production is healthy.

    Four defects found by CI under load, each fixed with a reproducing test and a negative control, and each in production:

    1. flaky: 'a stopped process turn (real processes, real clock) > ignores its SIGTERM ... nothing survives' failed once on CI at 14.2 s #1766, turn-stop escalation. A Bun timer fires 1–3 ms early, the card answers X_NONE, and nothing re-asked, so a SIGTERM-deaf group survived. Fixed by fix(queen): a grace timer that comes back early asks the card again, so a deaf turn still loses its group (Closes #1766) #1809 (the other contributor, for turns) and fix(queen): an orderly stop whose timer fires early still kills at its timeout; the turn-stop bench waits for its kills, not a fixed 11 s #1811 (the orderly stop, an EarlyClock helper, and the stopping benchmark's cleanup artefact removed).
    2. flaky pglive: 'PgLink at its edges > an acknowledgement the store refuses fences the node at once' races on CI #1778, the pglive fence test pinned one racing write. Fixed by test(queen): pglive PgLink edge test asserts the fence, not which racing write is logged (Closes #1778) #1808.
    3. pglive: Bun 1.3.x panics (Segmentation fault at 0xFFFFFFFFFFFFFFF8) after queen-actors-pg-edges-live runs, failing the job #1813, a Bun 1.3.x segfault in pglive. It came from our own stop terminating the PgLink beat Worker mid-compile. fix(queen): a beat thread is ended only once it has loaded, so a node stopped at its start no longer crashes Bun 1.3.x (Closes #1813) #1837 waits for loaded, and the pglive suite then passed 20 runs in a row.
    4. queen-actors-isolate threadWork: terminating a thread turn while its Worker is still compiling can segfault Bun (production, turn stop on) #1865, the same mid-load terminate in threadWork's stop(). Not reachable in production today: no production caller, and turn stop abandons thread turns. Fixed ahead of time by fix(queen): a thread turn's worker is ended only once it has loaded, so a stop at its start no longer crashes Bun 1.3.x (Closes #1865) #1866. On 1.3.6 the old code crashed 5 of 20 runs and the fix 0 of 20.

    Production (queen 219813f): /health 200; telemetry and turn stop on; hosting on, with sybil rules and signed epochs; vault deployed but off; 0 [ERROR].

    Host: disk was down to 144 MB free once. Removing the whole zig cache (3.4 GB, no zig running) brought it to 9.75 GB free. Swap is about 6 GB.

    Owner pending: confirm the two TRINITY DEV proposals (die B, release order), and the vault keys.

  17. gHashTag
    added 2 commits that reference this issue on Oct 11, 2026

    spec(queen): the simulation gate plays keyed actors, two bee dispatch…

    cd55351

    spec(queen): the simulation gate plays keyed actors, two bee dispatch…

    49a4586
  18. dmitrii-f-t27
    added a commit that references this issue on Oct 11, 2026

    test(queen): the simulation gate plays keyed actors, two bee dispatch…

    7e0b244
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions