Skip to content

t27t: profile the t27c CPU trainer -- 94.2% of a train_hslm step is matmul, 42.4% outer_add; four levers bounded (Closes #10850) - #10857

Merged
gHashTag merged 4 commits into
masterfrom
claude/peaceful-kepler-585ad9
Oct 19, 2026
Merged

gHashTag merged 4 commits into
masterfrom
claude/peaceful-kepler-585ad9

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Closes #10850

What

  • specs/ml/train_hslm.t27 gets a profile mode. It is off by default.
    • profile=1 reads the monotonic clock; profile=2 reads the thread's CPU clock.
    • Every interval between two clock reads is charged to one of 13 parts of a training step.
    • The four local kernels mark their own entry and exit. The gaps between kernel calls are attributed by the call order of nn/hslm.t27: the gap before each O matmul is the attention core.
    • Two new tests:
      • one k=27 step with the profile on and off gives the same parameter digest, so the profile moves no bit;
      • a two-window step's marks match the call order (1026 calls per kernel, 162 attention gaps each way).
  • specs/t27t/t27t.t27 gets a second runner, the t27c --backend ml binary of train_hslm.
    • It holds the measured ns, the mark counts and the cost of one clock read.
    • Tests recompute every share and every lever bound from those figures.
    • What was not measured is listed in NOT_MEASURED.
  • .trinity/seals/ml_train_hslm.json is re-sealed on the lab: 22/22 tests, 0 vacuous.

Measured

Railway lab, 48 cores, shared host at load 56–86, 2026-10-19. 20 steps at speed 2, loss_positions 27 (1782 scored positions per step). CPU clock, net of the clock reads:

part share
backward outer_add (weight gradients) 42.4%
forward matmul 27.4%
backward t_add (input gradients) 24.5%
three matmul kernels together 94.2%
attention cores, fwd + bwd 1.7%
serial tail (zero, gather+clip, LAMB, requantize) 0.6%
  • The wall clock (profile=1) gives the same matmul share, 94.1%, and the same biggest part. Its total closes on train_s (133.79 s).
  • profile=2 costs 0.05% of s_per_step: 7.6970 s against 7.6935 s with the profile off. The losses are identical.

Lever bounds (Amdahl on the split; ceilings, not measurements)

lever bound new t27c --backend ml support?
(a) 66 windows on N threads, fixed-order gradient sum 7.05x at N=8, 27.5x at N=48, 46.8x at N=66; 158.8x limit (99.4% parallel) yes: spawn/join, per-thread state
(b) f32 instead of f64 (2x on the matmuls) 1.89x no
(c) one GEMM over 27 positions instead of a GEMV per position (27x on the matmuls) 10.75x; outer_add alone 1.69x no
(d) int8 activations against packed trits, forward (8x) 1.31x; 1.38x if the forward matmul were free yes: packed trits, int8 SIMD dot
  • Biggest share: the backward matmuls at 66.8%; outer_add alone is 42.4%.
  • Biggest lever without new backend support: (c).
  • Biggest lever overall: (a), but it needs a thread primitive.

Checks (lab)

  • t27c test-report on specs/t27t/t27t.t27: 17/17 pass, 5 invariants, 0 vacuous.
  • t27b test on specs/t27t/t27t.t27 (arm64 under qemu): 17/17 pass, 5 invariants held.
  • t27c test-report on specs/ml/train_hslm.t27: 22/22 pass, 0 vacuous.
  • Mutation: one forward-matmul ns changed by 1 stops t27t from compiling at its sum invariant.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-19 05:21:29 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 44
PRs with All Checks Green 6
READY 2
FAILING 44
PENDING 0
NO CHECKS YET 0

These columns do not partition: 2 + 44 + 0 + 0 = 46, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=88b8cb73ceda != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

gHashTag and others added 3 commits October 19, 2026 12:38
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nds (#10850)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…now std.os.linux.timespec (#10850)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag
gHashTag force-pushed the claude/peaceful-kepler-585ad9 branch from bb72d04 to 06968d7 Compare October 19, 2026 05:52
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-19 06:32:25 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 45
PRs with All Checks Green 5
READY 2
FAILING 45
PENDING 0
NO CHECKS YET 0

These columns do not partition: 2 + 45 + 0 + 0 = 47, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=88b8cb73ceda != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

t27t: profile the t27c CPU trainer (train_hslm) and bound the four levers from measured shares

1 participant