Repository navigation
spec(ml): hslm_f32_study -- f32 vs f64 HSLM trainer, 2000 steps same seed (Closes #10844) - #10946
Merged
Merged
Conversation
…ps same seed: 1.15x faster, 0.64x peak RSS, largest held-out gap 0.0127 nats, verdict REJECT by the stated 0.01 threshold (Closes #10844) Two runs on the Railway lab, side by side: the f64 control (reproduces the release log at steps 500/1000/1500/2000 to every printed digit) and a mechanical f64->f32 copy of train_hslm, nn/hslm and nn/ternary_ste with everything f32, LAMB moments and loss accumulators included (no destabilization). The spec holds both curves, tok/s and VmHWM as constants; its tests re-derive the speed-up, the largest gap and tie the verdict to the threshold. train_hslm.t27 and nn/hslm.t27 are not touched. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10844
What this lands
specs/ml/hslm_f32_study.t27(+ its seal): the measurement the owner asked for on 2026-10-19 -- f32 instead of f64 in the HSLM trainer -- as constants and tests. It does not change the trainer:specs/ml/train_hslm.t27andspecs/nn/hslm.t27have 0 lines changed here.Method
Two runs on the Railway lab, side by side, same seed 1729, same TinyStories-V2 HTOK token files, release schedule (total 8000, warmup 2000), 2000 steps, held-out eval every 100 steps on the same 243 windows.
train_hslm.t27unmodified except that its file paths point to a private directory. It reproduces the release log (playra/hslm-t27-2m) at steps 500/1000/1500/2000 to every printed digit (12 of 12 numbers; tested).train_hslm,nn/hslmandnn/ternary_stewith thef64type token replaced byf32(mechanical sed) plus two hand fixes that keep the state file's f64 byte layout. Everything is f32, LAMB moments and loss accumulators included -- no destabilization (no NaN, gnorm tracks f64).Result
Every one of the 20 intervals is faster in f32 (min 1.046x). The curves coincide to 1.5e-3 nats until step 1400; the gap opens inside the lr peak (steps 1600-1900).
Verdict: REJECT by the rule stated before the numbers were read: ADOPT iff largest valid_full gap <= 0.01 nats AND speed-up >= 1.10. Speed passes, the gap is over by 0.0027 nats. This is "not yet", not "f32 trains worse": one seed, one run each; a second seed would show whether the 0.0059 final gap is precision or seed. No trainer change is proposed from this file; that decision is the owner's.
The lab was loaded (load average ~90 on 48 cores) so absolute tok/s is noisy; both runs shared the conditions, so the ratio is the fair figure.
Checks
🤖 Generated with Claude Code