spec(port): port specs/port/trios/download_fineweb.t27 (Closes #9730)

[dmitrii-f-t27]

Summary

Port gHashTag/trios:download_fineweb.rs (at 8b229e9489ee, 130 lines) as specs/port/trios/download_fineweb.t27 — Roadmap Stage 6 (goal #4548), target t27c gen-rust. Closes #9730.

The original is the FineWeb downloader and preparer for Trinity 3k: it writes the binary format the trainer expects — a 256-i32 header followed by uint16 tokens.

Per the issue's rule ("Port the decision, not the plumbing"), all three listed functions are ported under their names as pure functions (no transports — the filesystem and the byte writes are the callers' plumbing) with 7 tests (232 runtime asserts, 0 vacuous, 100% function coverage), including the original's own shard-size test:

• main — the dataset and tokenizer directories plan (original lines 12-28);
• create_test_dataset — the two-shard plan: 1M training tokens under seed 42, 100K validation under seed 123 (lines 30-53);
• create_fineweb_shard — the one-shard plan: the 256-word header (magic 20240520, version 1, the token count at index 2, the rest reserved) and the size formula 256*4 + tokens*2 — the 1000-token shard prices at 3024 bytes, the original's own test's assertion (lines 55-110);
• plus the deterministic token machinery: the LCG-style mix (seed*1e6 + i)*1664525 + 1013904223 masked to 32 bits, the rand value, the four-band distribution (10% space, 70% lowercase letters, 15% punctuation, 5% rarer tokens) with the band boundaries and token bases, and the mod-4096 vocabulary fold — asserted across 201 consecutive tokens, every one inside the SP4096 vocabulary.

Verification (all run live from the worktree root, zig 0.16.0)

• t27c parse: OK. t27c spec-status: IMPLEMENTED. t27c typecheck: 0 errors, 0 warnings.
• t27c coverage: 26/26 tested (100%) — no transports in this port.
• t27c test-report: 7 tests / 7 pass / 0 FAIL / 0 BLOCKED, 0 vacuous passes.
• t27c gen | grep -c 'not yet implemented': 0; gen | wc -l: 157 (> 12).
• grep -cE '^\s*(pub )?fn (main|create_test_dataset|create_fineweb_shard)\(': 3. grep -cE '^\s*test\s+(["|[A-Za-z_])': 7 (≥ 3).
• Boundary: git status --short — exactly this one file. The repo's lefthook pre-commit passed.
• dupe_scan: all 23 introduced names unique across 2040 specs, and the ShardPlan struct verified absent from the corpus type set by grep before the commit; the bare body scan adds no duplicate group (752 bodies / 221 groups on base, identical with this spec present).

Dmitrii Fedorov (@dmitrii-f-t27)

[dmitrii-f-t27]

@gHashTag Please merge this PR into master via the golden chain — rights on gHashTag/t27 are read-only for me (403), so the merge itself needs your action.

What this is: the #9730 port — download_fineweb.rs (gHashTag/trios, at 8b229e9489ee, 130 lines) re-authored as specs/port/trios/download_fineweb.t27. One file, one commit, head 5e2420826, Closes #9730.

What is verified:

• t27c parse OK, spec-status IMPLEMENTED, typecheck 0 errors / 0 warnings, coverage 26/26 (100%) — no transports in this port, t27c test-report 7 tests / 7 pass / 0 FAIL / 0 BLOCKED, 0 vacuous — the 256-word header (magic 20240520, version 1, the token count at index 2), the size formula 256*4 + tokens*2 (the 1000-token shard at 3024 bytes — the original's own test's assertion), the two-shard test plan (1M training under seed 42, 100K validation under seed 123), the deterministic LCG-style mix, the four-band token distribution with its boundaries and bases, and the mod-4096 fold — asserted across 201 consecutive tokens, every one inside the SP4096 vocabulary.
• gen → 157 lines, 0 not yet implemented; the 3 required names count exactly 3; 7 test blocks; boundary exactly the one file.
• dupe_scan: all 23 introduced names unique across 2040 specs, and the ShardPlan struct verified absent from the corpus type set by grep before the commit; the bare body scan adds no duplicate group (752/221 unchanged).
• The t27b-native-run artifact of this head (commit 5e24208, run 39033283279): download_fineweb.t27 row — reference pass, t27b pass, tests 7, invariants 0, asserts 233, blockers [], detail empty, reference_disagree: [].

CI (head 5e24208): required trio green — validate, check-linked-issue, parse-ratchet — Check L1 TRACEABILITY green and Corpus ratchet green; 26 checks pass overall, incl. emit-bitexact, t27b-native-linux, t27b-native-macos.

The 3 reds are the base-red trio, not this change: cli-tri (the #3324 include_str!), duplicate-bodies (only the known base names), t27b-native-ratchet (the same verdict red the Queen's port PRs carry; this spec's own row is reference pass + t27b pass — the registration debt is the UNLISTED class).

After the merge I will: verify the spec file live on the merge SHA, confirm #9730 auto-closed, and file the 14-step report.

Dmitrii Fedorov (@dmitrii-f-t27)