pre-pre-training

PPT — the thing to pre-train new funky model architectures before burning a hole in your wallet.

A minimal, forkable template for transformer / foundation-model pretraining architecture research: tokenize FineWeb-Edu-Dedup, train a clean decoder on one GPU node (or a laptop), measure FLOPs and tokens, report FineWeb NLL + bits-per-byte, and run six few-shot evals via the LM Evaluation Harness.

The base model is a vanilla Llama / SmolLM decoder — the "Vanilla" baseline from the Mixture-of-Recursions paper. Fork ppt/model.py and change the architecture; everything else (data, optimizer, measurement, eval) stays fixed so your comparison is clean.

Forked from karpathy/nanochat at commit dc54a1a3077cab11d68fac4c5d1cd5c51f5d8c7a, then stripped to the pretraining + measurement core. The full one-page spec is in SPEC.md.

Quickstart

# 1. Install for your backend (pick one): cpu (also Apple Silicon/MPS), gpu (CUDA), rocm (AMD)
uv sync --extra gpu

# 2. Tokenize FineWeb-Edu-Dedup into uint16 shards (writes to ~/.cache/ppt).
#    The 20B-token reproduction uses about 200 default 100M-token train shards.
python -m scripts.prepare_fineweb --shards 200

# 3. Train the SmolLM-360M vanilla baseline (the MoR paper's row, 20B tokens).
bash scripts/train.sh                      # python -m scripts.base_train --config smollm-360m --target-tokens 20e9
torchrun --nproc_per_node=8 -m scripts.base_train --config smollm-360m --target-tokens 20e9   # multi-GPU
#   smaller/quicker:  --config smollm-135m   |   generic scaling-law dial:  --depth 12

# 4. Evaluate: FineWeb NLL + bits-per-byte, and the lm-eval benchmark suite
uv sync --extra eval                       # enables the benchmarks (the only place transformers enters)
bash scripts/eval.sh --model-tag smollm-360m

No GPU? bash scripts/run_cpu.sh does a shrunk prepare → train → eval end to end on CPU or MPS.

Reproduction result

PPT's SmolLM-360M run reproduces the Mixture-of-Recursions "Vanilla" row with margin. It used the same 315M non-embedding parameter scale, 20B-token budget, cosmo2 tokenizer, FineWeb-Edu-Dedup data, and one H100.

Metric	MoR Vanilla	PPT run	Delta
FineWeb NLL ↓	2.7824	2.6031	-0.1793
LAMBADA	32.0	38.7	+6.7
HellaSwag	37.8	45.1	+7.3
PIQA	65.6	67.6	+2.0
WinoGrande	50.5	54.5	+4.0
ARC	39.6	49.7	+10.1
MMLU	28.0	25.1	-2.9
Avg	42.3	46.8	+4.5

Run details: SmolLM-360M, sequence length 2048, batch 524,288 tokens, 38,147 steps, bf16 on a single NVIDIA H100 80GB. The eval protocol follows the repo defaults: LAMBADA/HellaSwag/PIQA 0-shot, WinoGrande 5-shot, ARC-Easy/ARC-Challenge 25-shot, MMLU 5-shot; ARC is the mean of ARC-Easy and ARC-Challenge.

Backend matrix

Backend	Install	Compute dtype	torch.compile
CUDA (SM80+, incl. H100)	uv sync --extra gpu	bf16	on by default
ROCm (AMD)	uv sync --extra rocm	bf16	on by default
Apple Silicon (MPS)	uv sync --extra cpu	fp32	off
CPU	uv sync --extra cpu	fp32	off

COMPUTE_DTYPE is autodetected (override with PPT_DTYPE). torch.compile is probed before use and falls back to eager if the compile toolchain is unavailable, so a run never crashes on it (compiled is ~2× eager on an H100).

Repo layout

ppt/            the package — fork model.py
  model.py        vanilla Llama/SmolLM decoder (RoPE, GQA, RMSNorm, SwiGLU, tied) + SmolLM CONFIGS
  optim.py        uniform AdamW (default) + Muon (--optimizer muon)
  data.py         uint16 mmap shard reader + writer
  tokenizer.py    SmolLM cosmo2 wrapper via tokenizers lib (vocab 49152, EOT 0)
  flops.py        analytical FLOPs, cross-checked vs FlopCounterMode
  loss_eval.py    NLL + bits-per-byte
  evals.py        lm-eval loglikelihood adapter (behind [eval])
  checkpoint.py   save / load / resume
  report.py       per-run JSON manifest
  common.py       COMPUTE_DTYPE autodetect, DDP/device setup
scripts/        prepare_fineweb.py, base_train.py, base_eval.py, train.sh, eval.sh, run_cpu.sh
tests/          CPU-only: tokenizer, data, flops, smoke, evals

Tips

• Reproducibility. Each run writes a JSON manifest (config + git commit + hardware + dataset sha + results) under <cache>/reports/ (and the checkpoint dir for training).
• wandb is a no-op unless you pass a real --run name (or set WANDB_MODE=disabled).
• HPC clusters: if torch.compile falls back to eager, your /tmp is likely full — point TORCHINDUCTOR_CACHE_DIR / TMPDIR at a filesystem with space to get the compiled speedup.

Non-goals

transformers for modeling; SFT / RL / agentic training; inference serving / KV-cache / generation-based evals; multi-node parallelism; an architecture registry or plugin system; tokenizer training; a chat UI.