Skip to content

spec(hslm): nn/hslm_tokenizer and ml/hslm_corpus -- the 729-token BPE and a pinned TinyStories token file (Refs #10372) - #10380

Draft
gHashTag wants to merge 5 commits into
masterfrom
feat/hslm-tokenizer
Draft

gHashTag wants to merge 5 commits into
masterfrom
feat/hslm-tokenizer

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Refs #10372 (stays open for the training run). Part of #10024, cell 25.

What this adds

Two specs, nothing else (no foreign-language code, no generated files):

  • specs/nn/hslm_tokenizer.t27 -- the HSLM byte-level BPE from gHashTag/trinity-training @ 3e9648c1817a97c5f9b8c727b0f8e2291928eead (src/hslm/tokenizer.zig, bpe_merges.zig). Vocab 729 = 4 specials + 256 bytes (offset 4) + 469 merges (ids 260..728). The merge table is const data converted from bpe_merges.zig with sed/awk, nothing retyped. encode and decode keep the Zig priority order (one full-sequence scan per rule, in index order) and its edge semantics (short output truncation, UNK -> '?', ids >= 729 ignored).
  • specs/ml/hslm_corpus.t27 -- main() reads /tmp/hslm/valid.txt and /tmp/hslm/train.txt and writes /tmp/hslm/valid.tok and /tmp/hslm/train.tok. No heap, fixed buffers, streamed 64 KiB at a time. Read-back verification inside main().

Token file contract

bytes 0..3   "HTOK"
bytes 4..7   u32 LE, version 1
bytes 8..11  u32 LE, N tokens
bytes 12..   N x u16 LE (every id < 729)

File size is exactly 12 + 2N.

Story boundaries: a story is the text between two lines that are exactly <|endoftext|>; each story is its tokens plus one EOS (id 2), no BOS; the newline before the marker is dropped; newlines inside a story are kept. This is deliberately NOT the Zig loader's cut (src/hslm/data.zig Dataset.addText / loadTextFile: one line = one story, 2000-byte pieces each wrapped in BOS/EOS, newlines lost, marker tokenized as text). Documented in the spec header.

Corpus (lab, Railway)

file bytes sha256
valid.txt (TinyStoriesV2-GPT4-valid.txt, whole) 22502601 6874bae9a4c1a4e7edcf0e53b86c17817e9cf881fc75ff2368da457b80c0585d
train.txt (first 25000000 bytes of TinyStoriesV2-GPT4-train.txt, cut to the last complete marker line) 24999431 11dd493e85414802061dbf9d94840839c9fbc8e7c71e8833a4c82713a4093e24

Token files (also in /data/hslm/ on the lab):

file tokens N bytes stories (EOS) sha256
valid.tok 7689354 15378720 27630 d076d9e8fd8a4fd6ab1ee67d0d57cf770fb0597bdf0ccee87c69a21e0c284d91
train.tok 8557893 17115798 30496 9631d31544f1733e752a7d7c92f67467fa5781cc4d917628d9641987521127d5

Wall time of one main() run (both files, ReleaseFast, 48-core lab, single thread): about 7.3 s.

Tests

  • t27c test-report specs/nn/hslm_tokenizer.t27: 29 tests, 29 pass, 0 fail, 0 vacuous.
  • t27c test-report specs/ml/hslm_corpus.t27: 25 tests, 25 pass, 0 fail, 0 vacuous.
  • The tokenizer tests pin ids that were produced by running the Zig Tokenizer.encode/decode itself on the lab (17 strings including "Once upon a time", a newline string, a non-ASCII UTF-8 string, the empty string, <|endoftext|>, control and high bytes; truncation and decode-into-short-buffer edges), plus structural checks over the whole table (layout, running sum, operands earlier, byte spelling, every token decodes, all 256 byte values round-trip) and a decode(encode(s)) == s round trip.
  • Extra check on the lab, not in the repo: a Zig program that splits valid.txt into stories by the same rule, runs the Zig tokenizer on each, and compares with valid.tok: 27630 stories, 0 mismatches, all 7689354 tokens accounted for.

Exact lab commands

cd /root/t27 && git fetch origin feat/hslm-tokenizer && git worktree add -f /root/t27-hslm-tok origin/feat/hslm-tokenizer
head -c 24999431 /tmp/hslm/train_raw.txt > /tmp/hslm/train.txt     # train_raw.txt = first 25000000 bytes of the train file; 24999431 = end of the last marker line
cd /root/t27-hslm-tok
PATH=/opt/zig:$PATH /root/t27/target/release/t27c test-report specs/nn/hslm_tokenizer.t27
PATH=/opt/zig:$PATH /root/t27/target/release/t27c test-report specs/ml/hslm_corpus.t27
cd specs/ml && /root/t27/target/release/t27c compile hslm_corpus.t27 --backend ml -o /tmp/hslm-build/hslm_corpus.zig
cd /tmp/hslm-build && /opt/zig/zig build-exe -O ReleaseFast hslm_corpus.zig -femit-bin=/tmp/hslm-build/hslm_corpus_exe
./hslm_corpus_exe
sha256sum /tmp/hslm/*.txt /tmp/hslm/*.tok

Not done / caveats

🤖 Generated with Claude Code

gHashTag and others added 3 commits October 18, 2026 17:18
)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
#10372)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…efs #10372)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-18 10:27:01 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 49
PRs with All Checks Green 1
READY 0
FAILING 49
PENDING 0
NO CHECKS YET 0

These columns do not partition: 0 + 49 + 0 + 0 = 49, and there are 50 open PRs. A PR is being counted twice or not at all.

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fad12492937a != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

…d ternary_mac discard fix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-10-18 10:37:58 UTC

Summary

Status Count
Total Open PRs 50
PRs with Failing Checks 50
PRs with All Checks Green 0
READY 0
FAILING 50
PENDING 0
NO CHECKS YET 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fad12492937a != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

This was referenced Oct 18, 2026
This was referenced Oct 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant