Repository navigation
spec(ml): train_hslm speed=3 -- one matrix over the 27 positions, bit-identical to speed=2 (Closes #10849) - #10866
Merged
Merged
Conversation
…-identical to speed=2 (Closes #10849) speed=3 computes the speed 2 window layer by layer over all 27 positions: each ternary matrix and each f64 gradient row is read once per window instead of once per position. Every sum keeps speed 2's order, so logits, loss and all 2130138 gradients match speed 2 to the bit (tests at k = 27 and k = 1). Lab: 478.1 tok/s against 323.5 at speed 2, side by side, same seed, byte-equal checkpoints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
This was referenced Oct 19, 2026
gHashTag
added a commit
that referenced
this pull request
Oct 19, 2026
…ys, unblocking it (Closes #10906) (#10907) PR #10866 added tr3_* and the b3_* module-level arrays to train_hslm.t27. use_resolve splices no var, so eval_hslm, which reaches tr3_forward through tr_window_forward, was BLOCKED with: use of undeclared identifier 'b3_q'. Declare the same 31 arrays with the same names, types and sizes. Tests 10/10, 0 vacuous, seal saved and verified on the Railway lab. Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
7 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10849
What
specs/ml/train_hslm.t27getsspeed=3: the speed 2 window computed layer by layer over all 27 positions, so each ternary matrix and each f64 gradient row is read once per window instead of once per position.tr3_forward,tr3_block_forward,tr3_attend): the embedding, the norms, Q/K/V, O, the MLP and the head over a [27 x n] block, one pass over each matrix (tr3_matmul, column tiles of 81, so 27 rows of a tile stay in L1). Attention stays causal. The last block at unscored positions makes only its key and value, astr_kv_onlydoes. VSA runs per block in position order.tr3_window_grad,tr3_block_backward,tr3_attend_backward): every scored position at once.tr3_outerholds one gradient row while every position adds to it; at speed 2 the whole 17 MB gradient was streamed once per scored position.tr3_t_addconverts each weight row once. Under the K-POSITION objective, earlier keys and values are still constants of a position's backward.tr_window_grad_k,tr_window_loss_kandtr_window_forwardbranch onspeed_level == 3.cfg_speed % 3becomes% 4. The default stays 2.Evidence (Railway lab)
t27c test-report specs/ml/train_hslm.t27: 27 tests, 27 pass, 0 vacuous (after rebasing onto master, which added the trit-count test of #10840). New tests:speed_three_gives_the_reference_logits_to_the_bit(speed 0 vs 3)speed_three_gives_the_logits_of_every_position_of_speed_two_to_the_bit(all 27 x 729 logits)speed_three_gives_every_gradient_of_speed_two_to_the_bit_at_k_27and..._at_k_1: all 2130138 gradients, the input gradient, the logits and the loss give 0 differing bits. The comparison is not one of zeros: more than 10^6 (k = 27) and 10^5 (k = 1) gradients are nonzero.speed_three_gives_the_k_position_loss_of_speed_two_to_the_bit(k = 27 and k = 1)speed_three_lab_tok_s_and_its_speed_up_follow_from_the_measured_timesre-derives the recorded tok/s and the speed-up from the measured train_s.Negative control: swapping the order of the four partial sums in
tr3_t_addmakes both gradient bit tests fail, and the rest pass.Throughput: ReleaseFast,
loss_positions=27 steps=10 seed=1729, speed 2 and speed 3 started at the same moment on the 48-core lab (load average 59-66 from other sessions). The run was done twice.Both speeds printed the same losses (first 6.6079, last 6.6012, valid 6.5947) and wrote byte-equal
ckpt_10.binandckpt_10.state. The release figure was 307 tok/s at speed 2.Seal:
t27c seal specs/ml/train_hslm.t27 --save, then--verify, exit 0.Note for the threads session
The new
b3_*buffers are per-window scratch. A thread running windows in parallel needs its own copy of them, as it does ofact_*andkv_*.🤖 Generated with Claude Code