recurrent-bench

Ablation benchmark of recurrent and linear-attention language model architectures trained on identical data and token budgets.

Features

• Same-size training for RWKV-7, Mamba-3, GLA/DeltaNet, xLSTM, and a transformer baseline
• Identical datasets and token budgets across architectures
• State-tracking evaluation on parity and MQAR tasks
• Integration with lm-evaluation-harness
• Reproducible experiment configurations

Stack

• PyTorch
• flash-linear-attention
• HuggingFace datasets
• lm-evaluation-harness

Quick start

pip install -e .
python train.py --config src/rbench/configs/example.yaml
python eval.py --checkpoint checkpoints/model.pt

Project structure

recurrent-bench/
├── src/
│   └── rbench/
│       ├── __init__.py
│       └── configs/
└── tests/
    ├── __init__.py
    └── test_smoke.py

License

MIT