Repository navigation
[IGLA-Coder] P5 Multi-language evaluation harness #1038
Description
Part of epic #1032 (IGLA-Coder). Plan: t27_coder_model_plan.md P5.
Goal
Stand up the multi-language evaluation that gates all scale-up spend.
Sub-tasks
- P5.1 MultiPL-E (18 langs) via the BigCode evaluation harness as the primary multi-language gate.
- P5.2 t27-native eval: generate code from held-out .t27 specs, run the emitted test/invariant/bench blocks as executable oracles, report per-language pass rate. Unique in-house benchmark.
- P5.3 Optional McEval (40 langs) once the model is larger.
Done when
Automated, reproducible per-language pass@1 dashboard committed.
References: MultiPL-E (nuprl), BigCode evaluation harness, McEval. See research_code_llm_sota.md.
Anchor: phi^2 + phi^-2 = 3
User Scenarios
- Given a held-out set of .t27 specs with test/invariant/bench blocks, When the t27-native evaluation harness runs, Then it generates code for each target language and executes the emitted blocks as executable oracles, reporting per-language pass rates.
- Given the BigCode evaluation harness configured for MultiPL-E, When the multi-language gate runs, Then it evaluates the model across 18 languages and produces a pass@1 dashboard.
- Given both evaluation pipelines are configured, When the automated dashboard job runs, Then it commits a reproducible per-language pass@1 dashboard to the repository.
Requirements
- FR-001: The system MUST integrate the BigCode evaluation harness to run MultiPL-E across 18 languages as the primary multi-language gate.
- FR-002: The system MUST implement t27-native evaluation that generates code from held-out .t27 specs and executes emitted test/invariant/bench blocks as executable oracles.
- FR-003: The system MUST report per-language pass rates for both MultiPL-E and t27-native evaluations.
- FR-004: The system MUST produce an automated, reproducible per-language pass@1 dashboard committed to the repository.
Success Criteria
python -m scripts.tri_loop.harness --multilingual --bench multilingual-evalexits 0 and produces per-language pass@1 results for 18 languages via MultiPL-Epython -m scripts.tri_loop.harness --t27-native --specs-dir specs/heldoutexits 0 and reports per-language pass rate from t27-native evaluation- A committed dashboard file
results/multilingual-pass-at-1.jsonexists with per-language pass@1 metrics for all evaluated languages - 18 languages evaluated via MultiPL-E with pass@1 recorded
- t27-native evaluation runs on at least 5 held-out .t27 specs across at least 3 target languages
Boundary
- specs/tools/tri/harness.t27
- scripts/tri_loop/harness.py
- specs/lsp/language.t27
Activity
gHashTag commentedon Jun 7, 2026
Loop+10 -- P5: first resumable C-only curriculum, compile@1 still 0 (commit
4bbd5a9)Negative-first: at hidden=128 (493K params), 12200 steps, C-only, the model reaches a record-low val BPB of 1.038 but cannot synthesise a single compiling C function -- compile@1 = 0/4, pass@1 = 0/4.
Trainer change. New
--resume <ckpt>flag on thegeneratesubcommand (train_from_model): warm-starts WEIGHTS across short CPU sessions. Honest about its limits -- the checkpoint stores weights only, not the Adam moment buffers, so it is a warm-start-weights / fresh-optimizer-state continuation, NOT a bit-exact optimizer-trajectory resume; the code does not claim otherwise. A shape guard rejects a checkpoint whose (d, heads, layers) mismatch the args. Combined with--save, a long run accumulates across many short sessions.Curriculum.
coder_ablation/run_cB_curriculum.sh: hidden=128 (493K params, vs the 296K hidden=64 ceiling that pinned compile@1=0 in Loop+7/8), lang_id=1 (C), fim_loss=middle (infill), standard optimizer (phi is neutral per Loop+9, so the capacity probe is unhandicapped). 12000 steps in 3000-step resumable chunks on top of a 200-step warmup.Resume chain ran end-to-end; each chunk's
pre_resume_val_bpbmatched the prior chunk's saved BPB exactly (weight load is bit-exact):This 1.038 is the lowest BPB the coder track has produced and the doubled capacity is clearly doing real work -- but the execution eval is unambiguous:
A sample greedy completion of
uint32_t add_u32(uint32_t a, uint32_t b) {is token-salad: it stays in the C lexical neighbourhood (int32_t, plausible identifiers, operators) but emits degenerate repetition (_srex_srex_copop...) and never a valid statement orreturn. The model learned local byte statistics, not C grammar.Verdict [Efit, negative]: compile@1 = 0 is a capacity-ceiling result, not a tuning result -- consistent with Loop+7/8 (296K ceiling), now pushed to a confirmed-still-failing 493K. The
--resumeeval-harness machinery itself is VERIFIED working (monotonic BPB descent across bit-exact resumed chunks); the bottleneck is parameter count, not the training loop. Clean lower bound for the next capacity scan: 493K is not enough.Saved
coder_ablation/eval_pass1_loop10_cB.csv. Checkpointckpt_cB_h128.bin(1.9 MB) is gitignored/regenerable. Report:coder_ablation/LOOP10_REPORT.md.Next for P5: capacity scan (hidden=192/256, layers=3) with the resume machinery to locate the first non-zero compile@1 budget, or document the CPU ceiling as a real negative bound.
Anchor:
phi^2 + phi^-2 = 3.gHashTag commentedon Jun 14, 2026
Power-analysis front-end for the P5.1 MultiPL-E gate (n=32 eval bundle + Wilson harness)
Info-drop (ASCII-only). Before P5.1 makes MultiPL-E pass@1 the primary gate that gates all scale-up spend (P6, budget-gated), it is worth deciding up front how many samples a "pass" or "fail" actually needs, so a 0/n at P5 is not over-read. This note offers a ready-made power front-end for that decision -- numbers only, no capability claim.
Two pieces, both already built and self-tested in-sandbox:
n=32 stratified held-out eval bundle (
eval_bundle32.json, schema v2). 32 self-contained C functions across tiers (base=16, arith=6, bit=6, control=4), name- and body-disjoint from the train corpus, bidirectional Lee-2022 k=50 decontam CLEAN, all 32 compile undergcc -std=c11 -O2 -Wall -Werrorand pass a property assertion. capability fields are the literalTRAIN_BOX_PENDING-- the bundle is INPUT only; it does not contain or imply any measured rate.floor_oracleWilson harness. Given a future per-functionresult.jsonit reports per-tier and pooled Wilson 95% intervals on compile@1/pass@1 plus an interval-overlap verdict; given NO result file it prints the 0-success ceiling per n so the gate's sample budget is honest before any run. Reference ceilings: n=4 -> a true rate up to ~0.49 is still consistent with 0/4; n=16 -> ~0.19; n=32 -> ~0.11.Why this matters for P5 spend. If the pilot model returns 0/32 on MultiPL-E, the honest read is "true rate is below ~0.11", a far stronger basis for a go/no-go on P6 than 0/4 (which rules out almost nothing). A thin per-tier 0-result is weak; the harness shows the pooled read alongside, so a single sparse tier cannot drive the decision.
Honest scope. This is a measurement-power tool, not a capability result. It does not move compile@1/pass@1; those stay
TRAIN_BOX_PENDINGuntil a real run. The floor numbers (param/token gaps, Chinchilla deficit) are typed from the papers and carry their arXiv ids; the self-test (20/20, every gate failure-reachable) checks the math/determinism, not the literature. Sub-1M-param configs are far below the phi-1-small 350M floor, so a null at the pilot scale is the expected outcome, not a defect -- the harness just makes the null quantitative.Anchor: phi^2 + phi^-2 = 3.
gHashTag commentedon Jun 15, 2026
Methods note: report the F2 accuracy verdict with a small spec-curve, not one config
Filing a short methods data point for the P5 eval protocol. It concerns how a
significance verdict is reported, not a capability result.
The F2 breadth-as-moat harness (
src/bin/f2_harness.rs->multi_seed.rs+format_ladder.rs) reports a 3-way accuracy verdict (Welch t-test on the proxyloss
proxy_bits) from ONE hardcoded config (8 seeds, 40 steps, warmup 8, dim 128,alpha 0.05). A single-config significance verdict on a small sample is the classic
multiverse / forking-paths failure mode (Steegen 2016; Simonsohn spec-curve;
Gelman and Loken). So I ran a 12-cell one-factor-at-a-time sweep around the anchor,
varying one knob at a time: n_seeds {6,12,16}, seed_start {1,100}, steps {20,80},
warmup {4,16}, dim {64,256}, calling the live
run_multi_seedengine unchanged.Result (12 cells, all rc=0):
to-thousands), so the conversion-count signature is also config-stable.
Honest reading: the stable verdict is ZooWins, NOT PhiWins. On the proxy accuracy
axis the format-zoo arm reconstructs better. This is consistent with the harness's
own disclaimer that the proxy axis is "NOT a Verdict" on the breadth moat (FL-002
stays Open conjecture).
proxy_bitsis an internal MSE-reconstruction proxy, notBPB and not an optimality measure.
Limits: one-factor-at-a-time around one anchor (not a full cross-product), alpha not
swept, single engine, single machine. A far interior config could still differ.
Proposal for P5: when the protocol reports any significance verdict, report it with a
small multiverse spec-curve (a handful of one-factor variations) rather than a single
hardcoded config, and state which axis is a proxy. The runnable harness
(
race_oracle, 16/16 self-test, deterministic) and the 12-cell report are availableif useful. No ask; an optional 1-page sanity read is welcome.
gHashTag commentedon Jun 16, 2026
P5 Status Update (2026-06-16)
Progress since the last comment (Loop+10, 2026-06-07):
Blocked: first non-zero compile@1 requires parameter count above the 493K CPU-only ceiling; capacity scan needed before P5.1 runtime gate.
Anchor: phi^2 + phi^-2 = 3.
fix(ci): Coq admitted check covers all .v files; add allow-list mecha…
gHashTag commentedon Oct 7, 2026
Reopened 2026-10-07: the bulk close of issues without a
## Boundarysection was reverted by the owner. Nothing to do here; this comment replaces the earlier close notice.