Skip to content

Add BigBang-Proton results for matbench_mp_e_form - #368

Open
supersymmetry-technologies wants to merge 1 commit into
materialsproject:mainfrom
supersymmetry-technologies:main
Open

supersymmetry-technologies wants to merge 1 commit into
materialsproject:mainfrom
supersymmetry-technologies:main

Conversation

@supersymmetry-technologies

Copy link
Copy Markdown

Summary

This PR adds BigBang-Proton results for the matbench_mp_e_form task
(formation energy per atom, eV/atom) to the matbench_v0.1 leaderboard.

Files:

  • results.json.gz — predictions for all 5 official test folds (132,752 entries)
  • info.json — algorithm metadata (authors, description, references, requirements)
  • benchmark_bigbang_proton.ipynb — source notebook with the matbench record flow

Algorithm

BigBang-Proton (arXiv:2510.00129) is a 1.5B-parameter unified sequence-based
architecture for autoregressive language modeling, pretrained from scratch
on cross-scale, cross-structure, cross-discipline real-world scientific tasks
to construct a scientific multi-task learner. It features three innovations:

  1. Theory-Experiment Learning — aligns large-scale numerical experimental
    data with theoretical text corpora;
  2. Binary Patch Encoding — replaces BPE tokenization (259-symbol byte vocabulary);
  3. Monte Carlo Attention — substitutes traditional transformer attention.

The base model is pretrained end-to-end by next-word prediction on
multidisciplinary datasets (quark jet tagging, inter-atomic potential
simulations, genome/protein sequences, spatio-temporal sensor data,
up-to-50-digit arithmetic) mixed with general text (SlimPajama).

Benchmark setup (standard matbench protocol)

  1. SFT of the base model on MPTRJ (Materials Project + JARVIS) → BigBang-Proton-MPTRJ;
  2. Crystal structures serialized into compact text with the prompt
    "The formation energy per atom of the lattice is:";
  3. Autoregressive generation of the value (eV/atom) as a byte string;
  4. One model fine-tuned per fold on the official train/val split;
  5. Test-time greedy decoding; best-loss + best-acc checkpoints averaged per fold.

Results (official test folds, MAE eV/atom)

fold MAE
0 0.0914
1 0.0896
2 0.0887
3 0.0934
4 0.0871
mean 0.0900

Resources

Label: new_benchmark

## Summary

   This PR adds BigBang-Proton results for the **matbench_mp_e_form** task
   (formation energy per atom, eV/atom) to the matbench_v0.1 leaderboard.

   **Files:**
   - `results.json.gz` — predictions for all 5 official test folds (132,752 entries)
   - `info.json` — algorithm metadata (authors, description, references, requirements)
   - `benchmark_bigbang_proton.ipynb` — source notebook with the matbench record flow

   ## Algorithm

   BigBang-Proton (arXiv:2510.00129) is a 1.5B-parameter unified sequence-based
   architecture for autoregressive language modeling, pretrained **from scratch**
   on cross-scale, cross-structure, cross-discipline real-world scientific tasks
   to construct a scientific multi-task learner. It features three innovations:

   1. **Theory-Experiment Learning** — aligns large-scale numerical experimental
      data with theoretical text corpora;
   2. **Binary Patch Encoding** — replaces BPE tokenization (259-symbol byte vocabulary);
   3. **Monte Carlo Attention** — substitutes traditional transformer attention.

   The base model is pretrained end-to-end by next-word prediction on
   multidisciplinary datasets (quark jet tagging, inter-atomic potential
   simulations, genome/protein sequences, spatio-temporal sensor data,
   up-to-50-digit arithmetic) mixed with general text (SlimPajama).

   ## Benchmark setup (standard matbench protocol)

   1. SFT of the base model on MPTRJ (Materials Project + JARVIS) → BigBang-Proton-MPTRJ;
   2. Crystal structures serialized into compact text with the prompt
      *"The formation energy per atom of the lattice is:"*;
   3. Autoregressive generation of the value (eV/atom) as a byte string;
   4. One model fine-tuned per fold on the official train/val split;
   5. Test-time greedy decoding; best-loss + best-acc checkpoints averaged per fold.

   ## Results (official test folds, MAE eV/atom)

   | fold | MAE |
   |---|---|
   | 0 | 0.0914 |
   | 1 | 0.0896 |
   | 2 | 0.0887 |
   | 3 | 0.0934 |
   | 4 | 0.0871 |
   | **mean** | **0.0900** |

   ## Resources

   - Paper: https://arxiv.org/abs/2510.00129
   - Code: https://github.com/supersymmetry-technologies/BigBang-Proton
   - Model: https://huggingface.co/SuperSymmetryTechnologies/BigBang-Proton

   Label: `new_benchmark
@supersymmetry-technologies

Copy link
Copy Markdown
Author

Hi maintainers — the CI failures on this PR are unrelated to the
submitted files. Both jobs abort during GitHub Actions setup with:

Error: This request has been automatically failed because it uses a deprecated version of actions/upload-artifact: v2

The workflow still references actions/upload-artifact@v2, which GitHub
now rejects automatically; this affects any PR against this repository.
Could the workflow be updated to actions/upload-artifact@v4 (and any
other deprecated actions)? The submission itself
(results.json.gz + info.json + source) is ready for review. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant