Repository navigation
Conversation
Contributor
…d ternary_mac discard fix Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
PR DashboardGenerated at: 2026-10-18 10:37:58 UTC
Summary
Seal Status
|
This was referenced Oct 18, 2026
Merged
Merged
This was referenced Oct 18, 2026
Merged
Merged
Merged
Merged
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #10372 (stays open for the training run). Part of #10024, cell 25.
What this adds
Two specs, nothing else (no foreign-language code, no generated files):
specs/nn/hslm_tokenizer.t27-- the HSLM byte-level BPE from gHashTag/trinity-training @ 3e9648c1817a97c5f9b8c727b0f8e2291928eead (src/hslm/tokenizer.zig,bpe_merges.zig). Vocab 729 = 4 specials + 256 bytes (offset 4) + 469 merges (ids 260..728). The merge table is const data converted frombpe_merges.zigwith sed/awk, nothing retyped.encodeanddecodekeep the Zig priority order (one full-sequence scan per rule, in index order) and its edge semantics (short output truncation, UNK -> '?', ids >= 729 ignored).specs/ml/hslm_corpus.t27--main()reads/tmp/hslm/valid.txtand/tmp/hslm/train.txtand writes/tmp/hslm/valid.tokand/tmp/hslm/train.tok. No heap, fixed buffers, streamed 64 KiB at a time. Read-back verification insidemain().Token file contract
File size is exactly 12 + 2N.
Story boundaries: a story is the text between two lines that are exactly
<|endoftext|>; each story is its tokens plus one EOS (id 2), no BOS; the newline before the marker is dropped; newlines inside a story are kept. This is deliberately NOT the Zig loader's cut (src/hslm/data.zigDataset.addText/loadTextFile: one line = one story, 2000-byte pieces each wrapped in BOS/EOS, newlines lost, marker tokenized as text). Documented in the spec header.Corpus (lab, Railway)
Token files (also in
/data/hslm/on the lab):Wall time of one
main()run (both files, ReleaseFast, 48-core lab, single thread): about 7.3 s.Tests
t27c test-report specs/nn/hslm_tokenizer.t27: 29 tests, 29 pass, 0 fail, 0 vacuous.t27c test-report specs/ml/hslm_corpus.t27: 25 tests, 25 pass, 0 fail, 0 vacuous.Tokenizer.encode/decodeitself on the lab (17 strings including "Once upon a time", a newline string, a non-ASCII UTF-8 string, the empty string,<|endoftext|>, control and high bytes; truncation and decode-into-short-buffer edges), plus structural checks over the whole table (layout, running sum, operands earlier, byte spelling, every token decodes, all 256 byte values round-trip) and a decode(encode(s)) == s round trip.valid.tok: 27630 stories, 0 mismatches, all 7689354 tokens accounted for.Exact lab commands
Not done / caveats
main()cannot take argv in the ml backend (host_openneeds a literal path), so the paths are fixed..tok.bodytemporary files are left behind empty (no unlink host call).tokenToTrits,tritsToToken) of the Zig tokenizer are not ported; encode and decode never use the former.🤖 Generated with Claude Code