You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
t27b lab: stale deployed image (no ratchet step), 1800 s poll, reference cache misses imports and caches timeouts #6443
The lab gives slow feedback and partly stale numbers.
The running image is older than master's lab.py. latest.json (run 03362c1) has no steps.ratchet, no t27b_pass_vacuous_where_reference_passes and no timeout_retried. All three come from b1fd5f8 (t27b: per-spec ratchet ledger + vacuous-pass count, so the coverage number can only move visibly #6115) and c7dab23. So the per-spec ratchet is not enforced anywhere: no CI job runs it either, since loop-tools-gate.yml runs only Python unit tests. The lab never reports its own lab.py sha, so tri t27b doctor cannot see the drift.
Cadence. It polls every 1800 s, then a run takes about 9-10 min (corpus 200 s, reference 325 s). Feedback lags master by up to 40 min, which is the root of slip Q40.
The reference step starts from scratch every run.reference_all deletes /work/reference and runs all 1259 files on 12 workers, even when t27c and the spec did not change.
The reference-cache key is wrong (blockers.rs::key, binary_stamp): it leaves out the use import closure and the zig version, so an importer keeps a stale verdict. Reference::Timeout is cached, so one contended timeout is frozen.
Change
status.json and every run JSON carry lab_py_sha and the image build time. tri t27b doctor gets LAB-IMAGE-STALE when the deployed lab.py differs from master's.
Railway redeploys when contrib/railway/t27b-lab/** changes (watch path), and the run log names the image sha.
T27_POLL_SECONDS 1800 -> 120 (git ls-remote is cheap), plus tri t27b lab kick (POST /kick) for the steward.
A persistent reference cache on the lab volume, keyed by spec bytes + the sorted bytes of its import closure + the t27c binary hash + zig version. Timeouts and lab errors are never cached.
Done when
A lab run on master shows steps.ratchet and the vacuous summary key.
The median time from merge to lab JSON is under 12 min over a day, and a run with t27c unchanged spends under 60 s in the reference step.
A test in cli/t27b/tests/refcache.rs shows that editing an imported spec invalidates the importer's row.
Redeployed master's lab, no new code. Railway deployment b3382f29-43af-4dfb-acfd-e85666872f97 reached SUCCESS (the previous image was 7b0ee265 from 2026-10-04T17:55Z). The first run on it, commit 2d5cdfd finished at 15:43:15Z, has steps.ratchet. The ratchet is red with 0 UNEXPECTED FAILURE, 65 UNEXPECTED PASS, 5 UNLISTED and 24 MOVED; the ledger moves are owner-only.
The poll is still 1800 s (next_poll_in_s = 1800). lab.py already polls with git ls-remote and runs only when the sha changes. The interval itself is lab.py line POLL = int(os.environ.get("T27_POLL_SECONDS", "1800")).
No PR was opened. The owner's 15:45Z rule bans hand-written non-t27 code, and every part of this fix would need some:
the image-sha publishing in lab.py and the Dockerfile;
the LAB-IMAGE-STALE doctor wiring in scripts/tri_loop/t27b.py and t27b_rules.py;
the poll default in lab.py;
the reference cache: lab.py reference_all and blockers.rs RefRunner::key/run, which also caches Timeout.
Only the decision image_code in specs/tri/t27b/steward.t27 (with its 4 tests and the regenerated gen/c) is allowed, and even that would break the hand-written scripts/ci/test_the_t27b_steward_decides_in_t27.py, which hardcodes "111 tests passed".
The lab is deployed by railway up and has no git source, so a merge never redeploys it (S37).
Activity
gHashTag commentedon Oct 5, 2026
Steward status, 2026-10-05 15:45Z (Refs #6063)
Redeployed master's lab, no new code. Railway deployment
b3382f29-43af-4dfb-acfd-e85666872f97reached SUCCESS (the previous image was 7b0ee265 from 2026-10-04T17:55Z). The first run on it, commit 2d5cdfd finished at 15:43:15Z, hassteps.ratchet. The ratchet is red with 0 UNEXPECTED FAILURE, 65 UNEXPECTED PASS, 5 UNLISTED and 24 MOVED; the ledger moves are owner-only.The poll is still 1800 s (
next_poll_in_s= 1800). lab.py already polls withgit ls-remoteand runs only when the sha changes. The interval itself is lab.py linePOLL = int(os.environ.get("T27_POLL_SECONDS", "1800")).No PR was opened. The owner's 15:45Z rule bans hand-written non-t27 code, and every part of this fix would need some:
reference_alland blockers.rsRefRunner::key/run, which also caches Timeout.Only the decision
image_codein specs/tri/t27b/steward.t27 (with its 4 tests and the regenerated gen/c) is allowed, and even that would break the hand-written scripts/ci/test_the_t27b_steward_decides_in_t27.py, which hardcodes "111 tests passed".The lab is deployed by
railway upand has no git source, so a merge never redeploys it (S37).Merge pull request #6535 from gHashTag/claude/t27b-lab-image-sha
policy: owner-approved exception for #6554 (lab.py checkout retry)
Merge pull request #6554 from gHashTag/claude/t27b-lab-retry-checkout