Skip to content

t27b lab: stale deployed image (no ratchet step), 1800 s poll, reference cache misses imports and caches timeouts #6443

Closed

Description

@gHashTag

Refs #6434
Refs #6063

Problem

The lab gives slow feedback and partly stale numbers.

  1. The running image is older than master's lab.py. latest.json (run 03362c1) has no steps.ratchet, no t27b_pass_vacuous_where_reference_passes and no timeout_retried. All three come from b1fd5f8 (t27b: per-spec ratchet ledger + vacuous-pass count, so the coverage number can only move visibly #6115) and c7dab23. So the per-spec ratchet is not enforced anywhere: no CI job runs it either, since loop-tools-gate.yml runs only Python unit tests. The lab never reports its own lab.py sha, so tri t27b doctor cannot see the drift.
  2. Cadence. It polls every 1800 s, then a run takes about 9-10 min (corpus 200 s, reference 325 s). Feedback lags master by up to 40 min, which is the root of slip Q40.
  3. The reference step starts from scratch every run. reference_all deletes /work/reference and runs all 1259 files on 12 workers, even when t27c and the spec did not change.
  4. The reference-cache key is wrong (blockers.rs::key, binary_stamp): it leaves out the use import closure and the zig version, so an importer keeps a stale verdict. Reference::Timeout is cached, so one contended timeout is frozen.

Change

  • status.json and every run JSON carry lab_py_sha and the image build time. tri t27b doctor gets LAB-IMAGE-STALE when the deployed lab.py differs from master's.
  • Railway redeploys when contrib/railway/t27b-lab/** changes (watch path), and the run log names the image sha.
  • T27_POLL_SECONDS 1800 -> 120 (git ls-remote is cheap), plus tri t27b lab kick (POST /kick) for the steward.
  • A persistent reference cache on the lab volume, keyed by spec bytes + the sorted bytes of its import closure + the t27c binary hash + zig version. Timeouts and lab errors are never cached.

Done when

  • A lab run on master shows steps.ratchet and the vacuous summary key.
  • The median time from merge to lab JSON is under 12 min over a day, and a run with t27c unchanged spends under 60 s in the reference step.
  • A test in cli/t27b/tests/refcache.rs shows that editing an imported spec invalidates the importer's row.

🤖 Generated with Claude Code

Activity

  1. gHashTag commented on Oct 5, 2026

    @gHashTag
    OwnerAuthor

    Steward status, 2026-10-05 15:45Z (Refs #6063)

    • Redeployed master's lab, no new code. Railway deployment b3382f29-43af-4dfb-acfd-e85666872f97 reached SUCCESS (the previous image was 7b0ee265 from 2026-10-04T17:55Z). The first run on it, commit 2d5cdfd finished at 15:43:15Z, has steps.ratchet. The ratchet is red with 0 UNEXPECTED FAILURE, 65 UNEXPECTED PASS, 5 UNLISTED and 24 MOVED; the ledger moves are owner-only.

    • The poll is still 1800 s (next_poll_in_s = 1800). lab.py already polls with git ls-remote and runs only when the sha changes. The interval itself is lab.py line POLL = int(os.environ.get("T27_POLL_SECONDS", "1800")).

    • No PR was opened. The owner's 15:45Z rule bans hand-written non-t27 code, and every part of this fix would need some:

      • the image-sha publishing in lab.py and the Dockerfile;
      • the LAB-IMAGE-STALE doctor wiring in scripts/tri_loop/t27b.py and t27b_rules.py;
      • the poll default in lab.py;
      • the reference cache: lab.py reference_all and blockers.rs RefRunner::key/run, which also caches Timeout.

      Only the decision image_code in specs/tri/t27b/steward.t27 (with its 4 tests and the regenerated gen/c) is allowed, and even that would break the hand-written scripts/ci/test_the_t27b_steward_decides_in_t27.py, which hardcodes "111 tests passed".

    • The lab is deployed by railway up and has no git source, so a merge never redeploys it (S37).

  2. added 3 commits that reference this issue on Oct 5, 2026
    18a240e
    4f565b6
    4040407
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority/highMust fix before next milestone

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions