epic: t27 vs direct Verilog on spec-to-RTL -- close the model-matrix gap (Build-A-Chip-Bench proxy)

Close the gap between a model writing .t27 (compiled by t27b verilog) and the same model writing
Verilog directly, then measure whether t27's in-spec tests let it pull ahead.

Where we stand (2026-10-18)

Proxy for Build-A-Chip-Bench: the 10 smallest CVDP cid003 spec-to-RTL tasks, one shot, the same
prompts for every model. Gates: G1 cocotb/Icarus, G2 Yosys, G3 ORFS nangate45 with 0 DRC and
hold >= 0, G4 setup >= 0 at 2.5 ns. All runs were on the Railway lab.

Tasks passing all gates, of 10:

Model	A: direct Verilog	B: .t27 -> t27b	B - A	Writer cost A / B
Claude Sonnet 5.5	9	7	-2	~$0.41 / ~$0.77
Claude Opus 5.5	8	7	-1	~$0.99 / ~$1.53
GLM-4.7-flash (z.ai free)	5	2	-3	$0 / $0

Direct Verilog wins on every model. On the Claude models, B's losses are t27 tool gaps, not wrong
designs. GLM-flash could not write parseable t27 from a one-page guide (5 of 10 failed to parse).
Claude dollar figures are estimates at assumed list prices.
Sonnet beat Opus at about 40% of the cost: route spec-to-RTL work to Sonnet by default.
N = 10 and a single shot, so a one-task gap is noise. The robust parts are the direction (3 of 3
models) and GLM's -3.

Sub-tasks

Tool gaps, cheapest first:

t27b verilog: fold a constant rst_n/en tie-off out of the clocked process #10046 fold a constant rst_n/en tie-off out of the clocked process (PR fix(t27b): fold a constant rst_n/en tie-off out of the clocked process (Closes #10046) #10057, merged)
t27b verilog: a port cannot be named after a t27 keyword (in, for, ...) #10061 a port cannot be named after a t27 keyword (in). The shape gate closed the issue;
the fix is in progress.
odd-width uN as a fn param/return type and in as uN casts (issue to be filed by the
session that took it)
no test bench when the clock is a renamed net (clk = i_clk)
t27b test JIT runs only on arm64, but the lab is x86_64. Writers cannot run their own tests.
signed iN, arrays and Verilog parameter in t27b verilog

Learnability:

parser hints for foreign syntax: let, loop, as u3 and Rust-style returns get a "use X"
message
a worked-examples guide (10 small modules) generated from specs, replacing the one-page
cheat sheet

Measurement:

re-run the matrix after the tool gaps land, with a compile/check feedback loop (at most 3
iterations) for both arms
the full cid003 set, pass@3, one harness for every model, and tasks whose tests probe hidden
internal names filtered out
one paid non-Anthropic model, once a provider balance exists (all paid GLM tiers answer 1113)

EDA in t27:

turn specs/igla/race/eda.t27 from a data model into a real gate that reads ORFS reports
(DRC, WNS, area)

Boundary

specs/tri/t27b/verilog_emit.t27
specs/tri/t27b/verilog_plan.t27
specs/igla/race/eda.t27

User Scenarios

Given a CVDP task prompt when a model writes .t27 and t27b verilog compiles it then
the emitted module matches the task's interface and passes the same four gates as hand Verilog.

Requirements

FR-001: Every tool fix MUST land as a .t27 change with test blocks and a negative control.
FR-002: Every published number MUST come from a lab run with receipts (logs + sha256).
FR-003: The headline MUST say only what the model x arm x gate table supports.

Success Criteria

Sonnet 5.5 arm B reaches 9 of 10 tasks on the same 10-task set (tie with arm A).
With a feedback loop, arm B is at or above arm A on at least 2 of 3 models over at least 30 tasks.
GLM-flash arm B parses at least 8 of 10 specs.

Odd-width casts (as u3, as u12) for the proxy run: #10072, PR #10084.

feat: t27_check -- parse and typecheck .t27 source, with a fix hint f…

feat: t27_spec brief by default; field lists and failure reasons come…

spec(t27mcp): check_hints -- compiler message to a short fix hint for…

t27c: accept uN casts, 1 <= N <= 64, decided in int_type_names.t27 (C…

feat(t27mcp): spec_brief -- brief spec row fields and failure-reason …

feat(t27mcp): find.t27 -- t27_find, intent to ranked corpus specs or …

feat: t27_find -- intent to ranked corpus specs, or an honest miss

feat(t27mcp): dashboard.t27 -- grades and next step of the t27 MCP se…

feat(t27o): obligation ledger -- per-fn status, oracle outcomes, proo…

feat(t27mcp): telegram.t27 -- the owner's Telegram account as read-on…

feat: tg_* tools -- the owner's Telegram account, read-only, graded b…

spec(t27mcp): seal_verdict -- the seal receipt of t27_spec (#10066) (#…

spec(t27mcp): seal_verdict -- the seal receipt of t27_spec

feat(tg): tg_* tools read the owner's Telegram through the VIBEE app,…

specs(t27mcp): telegram.t27 names the owner's app TRI CLAW

docs(tg): the owner's app is TRI CLAW in the tg_* texts, README and b…

specs(t27mcp): telegram.t27 reads Telegram through the VIBEE app (refs …

@1

t27-mcp measurement: does the MCP server help cheap models write correct .t27? (10 tasks x 2 models x 2 arms)

Short answer. On these 10 tasks the MCP server made Sonnet 5.5 about 2x cheaper (same 10/10 correctness). It did not measurably help GLM-4.7-flash on correctness (4/10 with MCP vs 5/10 without, within noise at N=10). Sonnet hit a ceiling, so this set cannot show a correctness effect for it.

Design

10 small, well-specified tasks (counter, saturating ALU, FIFO pointers, Gray code, bit ops, LFSR, debounce, edge detect, up/down counter, min/max/clamp/absdiff). Same prompt text, same gate, same tools-except-MCP in both arms.
Arms. mcp: the t27-mcp tools (t27_check, t27_examples, t27_spec, t27_grep, t27_find) are available. plain: no mcp__t27__* tools; the same read-only corpus (specs, docs, CLAUDE.md, AGENTS.md at a18d7ee) is readable by file tools in both arms, so the baseline is strong.
Gate (not t27_check, because ok:true is not correctness, t27#8087). The model's file is run through t27c test-report on the Railway lab. A pass needs: the model's own tests pass, at least 2 tests, 0 vacuous, AND a hidden oracle test block (appended by the gate, never shown to the model) compiles and passes. Reference solutions pass the gate; 4 negative controls (wrong behaviour that still compiles) fail it as intended.
At most 3 gate attempts per task (the model runs gate.sh, which stops after 3). Pass = gate passes within 3 attempts.
Models: Sonnet 5.5 (Agent subagents; claude -p cannot run Anthropic models here) and GLM-4.7-flash (claude -p on z.ai, free tier).

Results (pass = hidden-oracle gate, 0 vacuous, <= 3 attempts)

Model	Arm	Pass	Pass @1 / @2 / @3	Tool calls / task	Tokens / task (in+cache-write+out)	Est. cost per passing task
Sonnet 5.5	plain	10/10	10 / 0 / 0	12.7	81.0k	$0.52
Sonnet 5.5	mcp	10/10	10 / 0 / 0	7.6	39.6k	$0.26
GLM-4.7-flash	plain	5/10	4 / 1 / 0	25.6	108k in+out (see below)	$0 (free tier)
GLM-4.7-flash	mcp	4/10	2 / 1 / 1	25.3	194k in+out (see below)	$0 (free tier)

Details behind the columns:

Sonnet totals over 10 tasks (mcp / plain): tool calls 76 / 127 (after subtracting the one extra Read that delivered the prompt file; the mcp arm's count includes one ToolSearch per task that loads the MCP tool schemas); uncached input 124 / 184; cache write 380,524 / 786,904; cache read 3,093,880 / 6,409,004; output 15,600 / 22,963 (output = max(reported, chars/4); streaming records under-count).
Assumed Sonnet 5.5 prices (per Mtok, unverified list-price assumption): $3 input, $15 output, $0.30 cache read, $3.75 cache write. At these prices: mcp $2.59 / 10 passes, plain $5.22 / 10 passes. The cost ratio is driven mostly by cache write, i.e. by how much corpus the model had to read.
GLM totals over 10 tasks (mcp / plain): tool calls 253 / 256; input 1,647,817 / 963,191; output 297,112 / 120,466; cache read 3,409,705 / 3,992,595. Tokens per passing task (input+output): mcp 486k (4 passes), plain 217k (5 passes). z.ai reports total_cost_usd as notional (flash tier is free), so cost per passing task is $0 actual and the notional figure is not reported. The MCP arm's GLM runs made 51 MCP calls in total, but only 7 were t27_check; the model's retry loop is driven by gate.sh in both arms.
GLM per-task outcome (P@n = passed at attempt n, F = failed 3/3). plain: 01 F, 02 P@2, 03 P@1, 04 P@1, 05 F, 06 F, 07 F, 08 P@1, 09 F, 10 P@1. mcp: 01 F, 02 P@2, 03 F, 04 P@3, 05 F, 06 F, 07 F, 08 F, 09 P@1, 10 P@1. Tasks 01, 05, 06, 07 failed in both arms. Failure causes were not classified.

What this supports and what it does not

Supported: for Sonnet, MCP cut tool calls by about 40% and billed tokens/cost by about 50% at equal correctness, per-task and consistently (the mcp arm is lower on every one of the 10 tasks in cache write).
Not supported: any claim that MCP improves correctness. Sonnet is at 20/20 on attempt 1 (ceiling effect; these tasks are too easy for it). For GLM the 5 vs 4 difference is one task out of 10, and the passing sets overlap heavily.
GLM tokens are not comparable with Sonnet tokens (different system-prompt overhead); compare arms within a model only.
N = 10 per cell, one run per cell. No variance estimate. Treat 5/10 vs 4/10 as a tie.
A harder tier (arrays, structs, enums, signed arithmetic, multi-function modules) would be needed to test a correctness effect for Sonnet. Not run.

Harness differences and deviations (disclosed)

Sonnet received its prompt through one extra Read of the prompt file (subtracted from tool counts above).
In the Sonnet plain arm the MCP tools were excluded by instruction only; I audited the subagent transcripts: zero mcp__t27__* calls. The GLM plain arm has no MCP server configured (--strict-mcp-config), also zero calls.
10-sonnet-mcp wrote its solution through a Bash heredoc instead of the Write tool (still a single file, still gated).
z.ai returned 429 (rate limit, code 1302) and 529 (overload, code 1305) under load. Every run whose gate outcome was not decided and that ended in an API error was discarded and rerun from scratch, at concurrency 1 after the first rounds; discarded and superseded transcripts are kept in the receipts (glm-infra-failed/). A run counts only if the gate reached a pass or three failed attempts, or the model ended normally. 01-glm-mcp was re-run once more because its first run had been orphaned by a killed runner (superseded transcript kept).
The harness scripts (gate.sh, run/prompt builders, usage summarizers) are scratch shell and Python in /tmp, never committed to any repository; they were written before the owner's 2026-10-18 only-t27 scratch-code rule reached this session and are shipped inside the receipts as data only. They should be replaced by a t27 harness if this experiment is repeated.

Receipts

mcp_bench_receipts.tgz, 12.6 MB, sha256 7f92eee49537df22624ce2817209679b1c1a945ad1f76a29511fe782371239bb. Contents: task statements, hidden oracles and reference solutions, all prompts, every gate attempt file and output per run (runs/<id>/attempt-N.t27|out, log), Sonnet usage JSON plus the 20 subagent transcripts, the 20 GLM stream-json transcripts plus discarded API-error runs, the harness scripts, and the runner log. Scanned for secrets before packing (key values from the environment and common token patterns): none found. The archive is in the side-session notes folder of the starting session (not uploaded to GitHub; ask for a release asset if wanted).

Follow-ups worth a separate issue: (1) tier-B tasks to get past the Sonnet ceiling, (2) classify why 01, 05, 06, 07 fail for GLM in both arms (probably a language-feature gap the MCP examples do not cover), (3) a t27_check-in-the-loop variant where the gate command is withheld so MCP is the only compile feedback.

Navigation Menu

Repository navigation

epic: t27 vs direct Verilog on spec-to-RTL -- close the model-matrix gap (Build-A-Chip-Bench proxy) #10066

Where we stand (2026-10-18)

Sub-tasks

Boundary

User Scenarios

Requirements

Success Criteria

t27-mcp measurement: does the MCP server help cheap models write correct .t27? (10 tasks x 2 models x 2 arms)

Design

Results (pass = hidden-oracle gate, 0 vacuous, <= 3 attempts)

What this supports and what it does not

Harness differences and deviations (disclosed)

Receipts

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

epic: t27 vs direct Verilog on spec-to-RTL -- close the model-matrix gap (Build-A-Chip-Bench proxy) #10066

Description

Where we stand (2026-10-18)

Sub-tasks

Boundary

User Scenarios

Requirements

Success Criteria

Activity

gHashTag commented on Oct 17, 2026

gHashTag commented on Oct 18, 2026

t27-mcp measurement: does the MCP server help cheap models write correct .t27? (10 tasks x 2 models x 2 arms)

Design

Results (pass = hidden-oracle gate, 0 vacuous, <= 3 attempts)

What this supports and what it does not

Harness differences and deviations (disclosed)

Receipts

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions