Add Jacobian Symmetry physicality benchmark - #713
Conversation
ElliottKasoar
added
the
new benchmark
There was a problem hiding this comment.
it would be great to have some form of interactivity here, do you have any ideas what you'd like to do? one idea would be a bar chart (bar for each molecule), like in the water slab dipoles, which you can click on to see the structures
There was a problem hiding this comment.
Went with the bar chart idea: click a model's cell → bar chart of λ per structure, then click a bar → a heatmap of that structure's antisymmetric Jacobian showing where the symmetry breaks. Also added N/A markers for structures a model can't run.
There was a problem hiding this comment.
This looks nice to me! I'm a little confused by the μ appearing in a few places though?
|
Thanks for the PR! just left a few comments for now, mostly minor |
2d02a5e to
eb6572e
Compare
ElliottKasoar
left a comment
ElliottKasoar
left a comment
There was a problem hiding this comment.
A few more minor suggestions/questions, but this is looking great!
| write_dir = OUT_PATH / model_name | ||
| write_dir.mkdir(parents=True, exist_ok=True) | ||
|
|
||
| for struct_name, struct in STRUCTURES.items(): |
There was a problem hiding this comment.
It might be nice to have tqdm here since it's fast enough to run locally, but could still be a few minutes for larger models.
There was a problem hiding this comment.
Added, thanks!
There was a problem hiding this comment.
This looks nice to me! I'm a little confused by the μ appearing in a few places though?
|
@ElliottKasoar There's no μ in the code, plotly just formats small ticks with SI prefixes, so 2e-6 renders as "2μ". Will change it to proper scientific notation. |
Add tqdm over structures per model. Use scientific notation ticks instead of SI prefixes on bar charts and heatmap colorbars (2e-6 rendered as 2μ). Models failing any structure now receive no mean/max lambda score instead of a score over the surviving subset. Replace fcc Al with cubic diamond so both periodic systems are carbon, within every model's element coverage. Lower the bad threshold to 0.004, the smallest lambda reported for a direct-force model in Bigi et al.
…ry-benchmark # Conflicts: # docs/source/user_guide/benchmarks/physicality.rst
|
thanks @ellalbolland i think this is looking really nice now. One thing i would add is maybe a few words to the description which shows on the app in more layman's terms. e.g. "Fraction of the force Jacobian's antisymmetric component, across 10 diverse structures which shows us...", imagine someone does not know what this means and what the implications are |
3 participants
Pre-review checklist for PR author
PR author must check the checkboxes below when creating the PR.
Summary
Adds a "Jacobian Symmetry" physicality benchmark that tests whether a model's predicted forces are conservative (derivatives of a single potential energy). The force Jacobian is built by central finite differences and the antisymmetric fraction of its Frobenius norm, lambda = ||J_anti|| / ||J||, is reported (mean and max over 10 diverse structures). lambda = 0 for perfectly conservative forces. Based on Bigi, Langer & Ceriotti, arXiv:2412.11569.
Linked issue
Resolves #712
Progress
Testing
Tested on all 21 registered models
New decorators/callbacks
None
AI usage disclosure
I developed this benchmark with some assistance from Claude Code. I have personally reviewed, tested, and understand all of the code.