Skip to content

igla coder: pre-flight and kill rule as .t27 specs (Closes #10363) - #10364

Open
gHashTag wants to merge 2 commits into
masterfrom
claude/funny-bell-6c5efe
Open

gHashTag wants to merge 2 commits into
masterfrom
claude/funny-bell-6c5efe

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Closes #10363

What

Two decision specs for the igla-coder-gpu training pipeline. The PyTorch kernels stay Python (t27c has no GPU backend); the decisions around a run are plain logic and live here.

  • specs/igla/coder/preflight.t27 -- GO or REFUSE with a reason code, before a run is launched. Order: unusable input (6), batch not a multiple of micro x seq_len (3), d_model % n_head (4), n_head % n_kv_head (5), planned hours >= queue timeout (1), planned cost > cap (2). The speed comes from the 60-step smoke run, derated by 0.95 (the real 4090 run took 19.74 h against 19.49 h predicted from the smoke speed).
  • specs/igla/coder/kill_rule.t27 -- CONTINUE, WARN or KILL at an eval point. A non-finite loss, grad_norm or val_bpb kills at any eval. A grad_norm above 100, or a val_bpb above 2.5x the reference curve, warns on the first eval and kills from the second. Nothing but non-finite is judged below 500M tokens. The curve is 0.4722 bpb at 10B tokens (the 100m ternary run) scaled by (10B/T)^(5/64), which needs only square roots; it gives 0.5262 at 2.5B against the real 0.5248.
  • Both are sealed (.trinity/seals/coder_igla-coder-*.json).

Evidence (Railway lab, never the Mac)

  • t27c test-report: preflight 13/13, kill_rule 12/12, 0 vacuous passes.
  • t27c gen-c of both specs compiles with 0 errors and 0 warnings (cc -shared -fPIC ... -lm), and the generated test runner (-DT27_TEST_MAIN) prints "All 13 tests passed." and "All 12 tests passed.".
  • Healthy runs give CONTINUE: the real finals of the 10B pair (0.47219 tern, 0.42166 fp), the ladder endpoints 13m..100m, the healthy 25m LR-sweep runs, the five eval records of the real train.py fine-tune log, the 60-step smoke run (loss 10.38), and a ten-eval trace with a counted zero kills.
  • Runs that must stop give KILL: the 25m LR x4 run that diverged (1.6257 bpb at 0.5B tokens: WARN at the first eval, KILL at the second), a run stuck at the initial loss (10.4 / 3.85 bpb at 2B tokens), a NaN loss (KILL even at the first eval), a grad_norm of 400.
  • Preflight on real numbers: tern100m on an H100 (457,172 tok/s, 3.49 USD/h) is GO at 6.40 h / 22.3 USD; the 4090 plan is GO at 20.5 h; a 500M-class plan (scaled, not measured, rate) is REFUSE timeout at about 32 h, and REFUSE cap at about 112 USD under a 36 h timeout.
  • Changing the key constants (slack, gate, derate, comparison direction) makes tests fail, so the tests are not vacuous.

Honest limits

  • The per-eval trace of the 10B pair does not survive (runs/ is not committed). The kill rule is calibrated on the ladder and sweep endpoints, the HF final summaries and the fine-tune log. healthy_trace_has_no_kills uses an ASSUMED plateau factor of 1.3 over the curve and says so. Once a real per-eval 100m log exists, add it as a test.
  • EXP-025 (72 of 72 runs killed by tight thresholds) is why the rule is lax; the thresholds should be tightened only against a measured trace.
  • t27c gen-python / gen-js lower declarations, not bodies (by design, codegen_python.rs), so they emit only the constants of these specs. gen-c lowers bodies, so the igla-coder-gpu glue builds the vendored C with cc and loads it with ctypes. gen-c does not lower the @abs builtin (the # t27c gen-c output has a stray @); the specs avoid it.

Glue and vendored output are in the matching gHashTag/igla-coder-gpu PR (gHashTag/igla-coder-gpu#44, which resolves igla-coder-gpu issue 43 there). The running tern500m2 pod and the epic's LEDGER entry are untouched.

The PR carries no-auto-merge: the specs should be read before they gate real GPU spend.

🤖 Generated with Claude Code

specs/igla/coder/preflight.t27 decides GO or REFUSE (reason code) before a
run is launched: shape checks (batch, heads, kv heads), planned hours against
the queue timeout, planned cost against the cap. specs/igla/coder/kill_rule.t27
decides CONTINUE, WARN or KILL at an eval point: non-finite numbers, a grad_norm
spike, or val_bpb above 2.5x a curve scaled from the 100m ternary run.

Tests use real igla-coder-gpu numbers. 13/13 and 12/12 pass on the Railway lab,
0 vacuous; both are sealed; the gen-c output compiles clean and its own test
runner passes.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@gHashTag gHashTag added the no-auto-merge Opt out of the issue gate's default squash auto-merge (issue-gate.yml) label Oct 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-18 09:35:08 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 50
PRs with All Checks Green 0
READY 0
FAILING 50
PENDING 0
NO CHECKS YET 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fad12492937a != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

This was referenced Oct 18, 2026
This was referenced Oct 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-18 16:46:28 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 43
PRs with All Checks Green 7
READY 2
FAILING 43
PENDING 0
NO CHECKS YET 0

These columns do not partition: 2 + 43 + 0 + 0 = 45, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fad12492937a != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-auto-merge Opt out of the issue gate's default squash auto-merge (issue-gate.yml)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

igla coder: pre-flight and kill rule as .t27 specs (preflight.t27, kill_rule.t27)

1 participant