[zeniheisser]

Builds on #314 and #317

Full addition of MadtRex, including a fork of Rex 1.1.0. MadtRex provides automatic reweighting for anything that can be generated and compiled with MadMatrix, as well as PDF reweighting through the MadSpace PDF interpolator.

Aside from all the MadtRex-specific materials and the parts from the PRs mentioned above:
Minor functionality modifications and additions to MadSpace I/O routines
Unified reweighting interface that automatically switches between "madtrex" and "classic" modes depending on the use case
MadtRex options in run cards and gridpacks
New reweighting tests
Minor fixes to existing tests

[oliviermattelaer]

Hi @zeniheisser,

This is very close to being ready to merge. Thanks a lot: the backend selection, with one reweight-card vocabulary for both backends, is really nice.

I asked an AI (Claude) to review the PR, with a focus on integration and user-friendliness. Below are the points it found; I removed the more minor ones. I already pushed one trivial fix to your branch (e560f32: help install listed madspace's build options under madtrex). If you prefer, I can use my AI to fix all of the points below. Just tell me.

On the merge order: this PR also carries the commits of #314 (single-helicity UMAMI entry point) and #317 (UMAMI setters/getters and model/fptype-tagged libraries), which are still open. It is better to merge those two first; this PR then shrinks to the MadtRex part only and is easier to review. Note that blocker 2 below comes from the library rename in #317.

Blockers

1. install madtrex cannot succeed (reproduced by building on macOS)

• madtrex/Rex/pyproject.toml:11: license = { file = "lgpl-3.0.txt" }, but that file is not in the PR, so pip install ./Rex fails at the metadata step on every platform. Related: there is no LGPL text anywhere under madtrex/, and the README points to Rex/FORKED_FROM.md, which does not exist.
• madtrex/Rex/include/Rex.h:1669: std::bad_any_cast has no string constructor, so clang refuses to compile it. GCC probably accepts it only because the template is never instantiated.
• madtrex/Rex/CMakeLists.txt:36,69: INSTALL_RPATH "$ORIGIN" is Linux-only. On macOS import pyrex then fails with Library not loaded: @rpath/libtearex.dylib; it needs @loader_path. madtrex's own CMakeLists already handles this.

2. The library rename (libmadmatrix_<model>_<proc>_<backend>_<fptype>.so) breaks existing tests

• tests/acceptance_tests/test_cmd.py:1475 and :1701 still glob the old libmadmatrix_P* names, so acceptancetest_uncovered currently fails. The second glob is also used by the GPU acceptance job (test_standalone_umami_first_call_gpu).
• docs/source/madspace/examples/{matrix-element,integrator,integrator-madnis,flow-training}.rst hardcode libmadmatrix_P0_gg_ttxg_scalar.so. The docs_examples job triggers on madspace/**, which this PR touches, so it will fail as well.

3. No CI job installs madtrex, so none of the MadtRex code is tested

• test_madtrex_standalone.py and test_reweight_backends.py are not referenced in any workflow, and both skip when madtrex is missing.
• test_reweight_backend.py is pure Python, but it is not in unittest.yml either.
• This is how blocker 1 got through. Suggestion: add a job that installs madtrex and runs these tests by name, so a missing madtrex fails instead of skipping.

Other points

4. Inline ! comments are not stripped. The card parsers and reweight_backend.classify_card only strip #, but the default card's own examples use !:

• set MT scan:[170, 180] ! one weight per value fails with "invalid scan definition".
• set MT 173 ! named is parsed as block MT, id 173, value ! named.
• classify_card does not recognise set PDF_SET 331700 ! x as a MadtRex-only line, so the backend chooser can pick the wrong backend.

5. Multi-index blocks are mis-parsed. _SET_RE (param_reweight_card.py:127) reads set NMIX 1 2 0.5 as id 1 with value 2 0.5. The classic module supports multi-index blocks (MSSM, CKM, ...), so these should be parsed, or at least rejected with a clear error.

6. TupperWare / WareHouse object lifetimes in the Python bindings.

• TupperWare and ParamHandler.add_api keep raw MatrixElementApi* pointers (tupperware.hpp:373,378), and the bindings (src/python/madtrex.cpp:113-158) do not keep the owner alive. Deleting the context or the api before build() means using freed memory, because the library is already unloaded. py::keep_alive<1,2> would fix it.
• WareHouse.add_process() and process(i) (madtrex.cpp:286-297) return references into a std::vector<TupperWare>. The next add_process() can reallocate that vector, which leaves earlier references dangling.
• The chaining methods return copies under the default return-value policy, so tw.add_api(a).set_normaliser(b) silently configures a temporary copy.

7. A reweighting failure stops the rest of the run. run_reweighting runs inside run_single (launch.py), after generation and before the post-processing tools. If it raises (for example on SPINUP=9 events), MadSpin, the shower and the analyses are all skipped. It would be friendlier to log the error, continue, and tell the user how to re-run the reweighting on the existing events. Related: there is no reweight run_01 command at the MG7 prompt to reweight an existing run; the only route is the standalone driver.

8. Version coupling between madtrex and madspace.

• madtrex/install.py:95 only checks that import madspace works. madtrex needs API that this PR adds: MatrixElementApi::device(), supports_set_parameter, EventFile::layout() and the newly installed umami.h. A madspace installed before this PR, or install madspace --bin, passes the check and then fails with compile errors deep inside CMake.
• Reinstalling madspace after madtrex leaves a silent binary mismatch: the class layouts change and EventFile::read/write are inline. Suggestion: record madspace's version or hash at build time and check it when madtrex is imported.

9. The default PDF for PDF reweighting. madtrex_driver.py:319-347 and 354-387 take the baseline PDF, and the normaliser precision, from the current Cards/run_card.toml, not from the run that produced the events. Example: generate run_01, change pdf1, then PDF-reweight run_01 with a card that sets no PDF_SET. The baseline is the wrong PDF, so every weight is scaled by the wrong ratio. It should come from the run's own banner or header.lhe (the LHAPDF ids in <init>).

10. include_madtrex_source does nothing useful. It copies the sources into the gridpack, but nothing builds them there, and bin/reweight is only written when an installed madtrex is included (include_madtrex). Also, when gridpack.py rebuilds madspace on another machine, the prebuilt madtrex is not rebuilt, which brings back the mismatch in point 8.

Thanks again, this is a great addition!

[zeniheisser]

@oliviermattelaer As far as I'm concerned, you can do it with Claude since it's all detail semantics; since you already have everything set up, it's probably easier for you to do it from your end than for me doing it by hand or having my own agent do it :)

[oliviermattelaer]

Hi @zeniheisser,

As agreed, I (with Claude) pushed fixes for all the points of my review to your branch, and merged main into it (two trivial conflicts, in unittest.yml and UpdateNotes.txt).

What changed

1. install madtrex builds again. The bad_any_cast in Rex.h that clang rejected is fixed, and Rex/CMakeLists.txt now uses @loader_path on macOS. madtrex now builds, installs and imports on macOS.
2. Library rename. The test_cmd.py globs and the four madspace docs examples now use the new library names (libmadmatrix_<model>_<Pdir>_<backend>_<fptype>.so).
3. CI.
   • The pure-Python tests (test_reweight_backend and the MadtRex card readers) are added to unittest.yml.
   • A new job, acceptancetest_mg7_madtrex, installs madtrex on top of the cached madspace and runs test_madtrex_standalone and test_reweight_backends. A skipped test fails the step, so a broken madtrex build can no longer pass CI quietly.
4. ! comments are stripped like # in all the MadtRex card readers and in the backend chooser.
5. Multi-index parameters. set NMIX 1 2 0.5 keeps all its indices, also when the parameter is given by its param_card name. This needed a small change in Rex: rwgt_block::multi_params, slha get/set with indices known at run time (also used by the file-based param_rwgt), and ParamHandler setting the UMAMI location "NMIX 1 2".
6. Bindings.
   • Every method that stores an api keeps it alive (keep_alive<1, 2>).
   • The builder methods return *this (reference_internal) instead of a copy.
   • WareHouse stores its processes in a std::deque, so the references handed out by add_process()/process(i) stay valid.
7. A failing reweighting no longer stops the run. The error is logged with a traceback file, and MadSpin, the shower and the analyses still run. A new MG7 command, reweight RUN [--reweight=CARD] [--reweight-backend=...] (after launch PROC -i), reweights the events of an earlier run.
8. madtrex/madspace coupling.
   • CMake records a digest of the madspace headers madtrex is built against (_build_info.json). import madtrex refuses any other madspace and gives the command to rebuild, instead of a silent binary mismatch.
   • The backend chooser reads the same check (madtrex/_compat.py, pure Python), so auto then falls back to the classic module.
   • install.py rejects a madspace that is too old before building.
   • install madspace/install madtrex at the MadGraph prompt now report a failing installer.
9. Default PDF. The baseline PDF set and the normaliser precision come from the run card stored in the events' own LHE header (<MG7RunCard>), not from the current Cards/run_card.toml.
10. include_madtrex_source now also gives the gridpack bin/reweight. On first use, bin/reweight builds madtrex from the shipped sources when there is no install, or rebuilds it when it was compiled against another madspace. The MadGraph VERSION file is now shipped with the sources (source builds in a gridpack needed it).

Licence

The LGPL text is not shipped in the repository: as part of MadGraph7 it would conflict with the MadGraph7 licence, which is more permissive. The vendored Rex is treated as dual licensed instead: LGPL-3.0-or-later as upstream, or the MadGraph7 licence as part of MadGraph7. Rex/pyproject.toml now states this as text (it pointed at the missing file, which broke pip install), and Rex/README.md and madtrex/README.md say so.

Tested locally (macOS, after the merge)

• All passing:
  • the unit tests: test_mg7_launch, test_reweight_backend, the madmatrix density/writer tests, and the 36 test_madtrex_standalone tests against a real madtrex install;
  • testIO_FDgauge_madmatrix;
  • the two test_cmd.py acceptance tests that were failing;
  • test_reweight_backends (classic and MadtRex agree event by event);
  • the matrix-element.rst docs example.
• Multi-index reweighting was checked in the Python parser and with a small C++ check of Rex's card parser and writer, but not end to end on an MSSM process.

The individual commits are on the branch if you want to look at any of them. Let me know if anything does not match what you intended.

[zeniheisser]

Summary of additions/fixes besides fast-forwarding to the last main version:

1. Removed the old MadtRex implementation (CUDACPP); the MadtRex shipped with MadGraph7 is completely independent from the old implementation, and utilises UMAMI to avoid having to rewrite and recompile code by just loading the amplitude codes as libraries
2. Bugfixes where the original weight was not evaluated against the original physics model for a couple of different reasons
3. Ensure binary interfacing between MadtRex and event generation; when MadtRex is activated, the output will always be binary and the "real" output format is only evaluated by MadtRex itself rather than at event generation
4. Small optimisations: When MadtRex is activated, the values of the matrix elements squared are written to disk; additionally, MadtRex is standardised to reweight across as many CPU cores as event generation (including splitting individual parton configurations into several sub-jobs to maximise utilisation)
5. Fixed output from MLM to work with MadtRex

[oliviermattelaer]

Merged!! Thanks a lot Zenny