Skip to content

Add NinjaRS reduction and bare-loop UV/IR pole selection to MadLoop - #315

Open
huashengshao wants to merge 113 commits into
mainfrom
madloop_ninjars
Open

huashengshao wants to merge 113 commits into
mainfrom
madloop_ninjars

Conversation

@huashengshao

@huashengshao huashengshao commented Oct 22, 2026 •

Copy link
Copy Markdown

MadLoop can now select NinjaRS with MLReductionLib=8, with native oneloopmaster scalar integrals and a genuine higher-precision rescue path. A usable installation becomes the default for ordinary Born processes; explicit backend choices and the existing loop-induced TIR defaults remain respected.

Implementation

  • Add process-prefixed Fortran wrappers and a versioned, panic-contained Rust C ABI with retained reduction workspaces. The DP path uses NinjaRS’s fast tensor engine; rescue recomputes the Fortran numerator/kinematics in quadruple precision and transports the binary128 inputs to the 256-bit engine without passing through binary64.
  • Add repository-owned install ninjars, including --update, pinned dependencies, local source overrides, compiler/runtime linking metadata and a compiled Fortran installation check.
  • Integrate standalone and FKS/NLO exports, dependency-copying modes, parameter-card validation, availability detection, rescue/status reporting and backend usage counters.
  • Improve the existing Fortran coefficient handling and symmetric tensor code, and repair CLI profiling/stability output-directory, two-to-one card lookup and diagnostic-report edge cases.

Bare loop UV/IR selection

LoopPoleMode provides a common selector for NinjaRS and COLLIER: -1 preserves legacy behavior, 0 returns combined poles, 1 UV only, 2 IR only, and 3 no poles. Finite coefficients and model counterterms remain unchanged. The usage guide explains card and Fortran use, rescue restrictions, and loop-induced interpretation.

  • Use the public pole-selection API now published on upstream main: oneloopmaster 5b71043 owns the scalar/tensor pole residues, and NinjaRS 41a4c00 exposes selection through its master providers. MadGraph pins both revisions and forwards the selector without duplicating the numerical residue formulas. UV/IR diagnostic modes use the existing generic NinjaRS engine in DP and 256-bit rescue, retaining scaleless-bubble coefficients. Normal combined-pole DP keeps the fast engine.
  • Carry the choice through the Fortran interface, retained contexts, COLLIER tensor caches, and rescue. Explicit diagnostic modes bypass learned loop/helicity filters so cancelling UV/IR contributions cannot be discarded before reduction. Default filtering stays unchanged.
  • Require only NinjaRS/COLLIER in a selected-mode rescue chain. Bridge ABI 4 is required; existing installations receive actionable install ninjars --update guidance. The installer verifies selected DP/MP poles and binary128 precision in a compiled Fortran smoke test.

Validation and measured scope

CI: all 106 active checks passed on 9228fc58e, across 11 workflows; 85 guarded checks were skipped as intended. No failures, cancellations, or reruns (checks).

  • 128 Python/compiled-Fortran regressions pass on the merged review revision, covering installation/update, backend defaults and option ownership, dispatch, warnings, pole selection, runtime state, statistics producer/reader round trips, export boundaries, profiling, and report acquisition. These modules are now explicitly run in CI. All nine affected exporter IO methods pass; generated references were refreshed from actual generation with existing assertions retained.
  • Upstream qualification: oneloopmaster passed six new native pole tests, five default-feature tests and all nine existing tensor regressions. NinjaRS passed 80 provider/core tests and a full locked workspace compile check.
  • New UV/IR qualification: 455 independent COLLIER tensor components through rank four, with 1,820 UV/IR-only DP checks and 3,640 higher-precision checks across repeated selector changes. All pass; all 7,280 DP/MP mode-cycle records also reproduce the previous ABI4 implementation exactly, including statuses and rational terms. Ordinary fast-DP corner-case diagnostics remain explicit and reproduce the pre-feature library. Four actual CLI-generated processes (u u~ > z, g g > t t~, e+ e- > mu+ mu-, and loop-induced g g > h) agree with COLLIER at three fixed points each across 74 configurations, including forced DP/QP, automatic QCD stability checks, and warm mode/filter changes. An actual two-subprocess shared-library check additionally passes 40 amplitude evaluations with both backends, including setter calls before the first evaluation and global runtime changes. Counterterm slots are cleared only in disposable validation outputs to isolate the bare loop. Reproduction and evidence.
  • Numerical qualification covers eight SM QCD/EW processes at 100 fixed points each, including two-to-four processes, plus mixed orders, complex masses, direction/rotation/state changes and forced rescue. Additional loop-induced controls include gg > HHH and gg > HHHH.
  • The report contains 72 actual CLI profiles across twelve processes, separating the stock CLI reduction estimate, numerator stage and total time, with numerical-stability distributions and captured preparation costs. Ten process comparisons retain matching timing scopes; the two on-shell cases retain unavailable aggregate ratios where helicity/filter selection changed.
  • All six declared normal-rescue timing comparisons have upper 95% ratio limits below 1.05 relative to original MadLoop with Ninja. This is a scoped result, not a universal speedup claim. CLI FULL − SKIP includes live tensor callbacks, masters and interface work; it is not the standalone NinjaRS pure-kernel timer.
  • The ordinary Rust suite has 31 passes and seven pre-existing ignored tests, including the new UV/IR provider and ABI tests. Precision/provider edge cases and the two failing opt-in diagnostics remain explicitly documented rather than being presented as passes. Linux x86_64 is the validated platform.

Report and repository cleanup

The measured benchmark PDF, Typst source, compact measurements, figures, font license and reproduction scripts live under doc/benchmarkfolder. Optional font binaries and the raw CLI capture are fetched into ignored local paths from a checksum-pinned immutable archive:

python -B doc/benchmarkfolder/fetch_artifacts.py
python -B doc/benchmarkfolder/final-report-v1/reproduce.py --output .benchmark-work/reproduced-benchmark.pdf

The first command performs one-time acquisition with size/SHA256 verification. Afterward the unchanged reproduction script runs offline using Typst 0.15.1 and the archived fonts. A fresh download into an isolated report copy reproduced the retained PDF byte-for-byte. About 1.9 GB of intermediate experiments, campaign harnesses, generated evidence and session notes were removed from the final tree. The complete measurement archive remains available at immutable commit 19d67c8605337c176680878a521b3a014176a7fb, with links and original hashes in the report provenance.

The generated-output reference updates preserve the existing assertions, and optional NinjaRS imports remain usable by the older Python CI collector without requiring TOML parsing during import.

Robustness and source size

Generated TIR interfaces reject unsupported backends before accessing cached coefficients. Explicit-card warnings use normalized parameter names; default selection shares one policy without mutating caller options. Legacy and NinjaRS statistics use one decoder, covered by compiled producer-to-reader tests. Partial cards print the effective reduction list instead of an uninitialized string. The focused regressions run in CI.

Removed about 4.73 MB of tracked optional binary/capture content while retaining the requested PDF and every numerical regression component. Optional fonts/raw materialization are acquired from an immutable revision with checksums; fresh acquisition still reproduces the PDF byte-for-byte. Both NinjaRS and the CutTools/IREGI installer remain available; generated-output references match the current exporters.

Keep explicit selections and loop-induced defaults unchanged. Validate standalone, NLO, unavailable-backend and compiled default selection; retain complete timing inputs and support the genuine NinjaRS stability rescue path.

Validation: fresh-source software suite 292 passed with one existing conditional generated-process skip; installed default 218 agrees with explicit Ninja 216. Preserve exact software and smoke evidence.
Preserve eight-process legacy and NinjaRS correctness evidence, focused precision/state controls, eight-case native-DP stability data and the completed six-process paired timing study. Keep exact sources, inputs, custody, uncertainty and cancellation records; exclude native build products.

Independent numerical, raw-custody, arithmetic and publication-closure audits pass for the declared finite samples. The later twelve-process CLI profile study remains outstanding.
Use the configured output directory consistently with timing and standalone stability checks. Retain the private CLI debug log on failed benchmark attempts.

Validation: four real driver-path regression cases and 110 affected software tests pass. Timing, RNG and numerical algorithms are unchanged; retain the incomplete first CLI pilot separately.
Propagate LoopPoleMode through DP/QP reduction, cached state and supported rescue chains. Preserve scaleless bubble residues at the oneloopmaster provider boundary and bypass origin-dependent learned filters in diagnostic modes. Require bridge ABI 4 with actionable installation updates.

Validate against COLLIER tensor references and four CLI-generated QCD/EW/loop-induced processes. Retain portable checks and inputs, refresh affected IO references, and add NinjaRS/pole regression tests.
@oliviermattelaer

Copy link
Copy Markdown
Contributor

Silent no-op when reduction ID 8 is selected without NinjaRS compiled in (madgraph/iolibs/template_files/loop_optimized/TIR_interface.inc, SELECT CASE(MLReductionLib(I_LIB)), ~line 206)

The SELECT CASE handles only 2, 3 and 7, with no CASE DEFAULT. In helas_loop_amplitude.inc the MLReductionLib == 8 branch is wrapped in ## if(ninjars_available), so in a build without NinjaRS an ID-8 entry falls through to TIRLOOP. Nothing in the exporter or the Fortran reader removes 8 from the list in that case, so the TIR coefficients are never filled and the result is silently wrong. Suggest either a CASE DEFAULT that stops with an explicit message, or having the exporter reject/strip 8 when ninjars_available is false. Verified by reading the template at commit d816e1b; not reproduced at runtime.

@oliviermattelaer

Copy link
Copy Markdown
Contributor

'MLReductionLib' in user_set never matches (madgraph/interface/madevent_interface.py, lines 3518 and 3526)

ConfigFile.__setitem__ stores parameter names lowercased (self.user_set.add(lower_name), madgraph/various/banner.py ~line 1264), so the membership test with the capitalised key is always False. The OPP/TIR slowdown warnings in both branches therefore never fire. This PR edits both lines to add '8' but keeps the broken key. The fix is to test 'mlreductionlib' in ...user_set, which is what loop_exporters.py:set_default_reduction_library already does.

@oliviermattelaer

Copy link
Copy Markdown
Contributor

NinjaRS usage is backed out of legacy unit buckets (madgraph/interface/amcatnlo_run_interface.py ~lines 3455-3456; madgraph/madevent/sum_html.py ~lines 135-136)

NinjaRS DP is tallied under legacy unit 8 and NinjaRS QP under unit 2 (BinothLHA.f, matrix_loop_induced_madevent.inc), then subtracted again in Python from Ninja_QP and PJFry. Any mismatch between the Fortran tallies and that subtraction misattributes PJFry or Ninja usage. A dedicated return-code slot for NinjaRS in U_RETURN_CODES and the RetUnit parser would remove the need for the subtraction on both sides. Maintainability suggestion rather than a bug.

@oliviermattelaer

Copy link
Copy Markdown
Contributor

Default NinjaRS selection is duplicated, and one site mutates the caller's options

The rule [8] if ninjars_install.is_available(...) else [6] appears in madgraph/interface/madgraph_interface.py (~line 5676), madgraph/various/process_checks.py (~lines 2444 and 2591), and the has_born variants around them. Also, process_checks.py ~line 2437 does MLoptions = MLOptions and then assigns into it, so a caller's dict picks up the chosen backend. Suggest one helper in ninjars_install (for example default_reduction_library(has_born, libdir)) that returns a new list, called from every site.

@oliviermattelaer

Copy link
Copy Markdown
Contributor

Bulky generated and binary artifacts committed to the source tree

Examples: doc/benchmarkfolder/benchmark.pdf (3.8 MB PDF), doc/benchmarkfolder/final-report-v1/cli-report.json.gz (1.3 MB), font binaries under final-report-v1/fonts/, report-data.json and expansion-results.json (53k lines), and about 50k lines of vendor/ninjars/tests/data. These go stale when the benchmark is rerun and are carried by every checkout. Consider keeping them as release assets or in a separate artifacts location. The new .gitignore entries for /references/ and .mpl-cache suggest the same treatment for the rest.

Store pending pole requests once in the shared parameter reader so the
latest setter survives initialization by any subprocess. Keep process-prefixed
entry points as wrappers and document the global choice.

Add a compiled two-subprocess regression and refresh the affected export
references. Validate generated two-flavor shared libraries against COLLIER,
including preinitialization ordering and subsequent global mode changes.
@huashengshao huashengshao changed the title Add NinjaRS reduction backend and automatic installation to MadLoop Add NinjaRS reduction and bare-loop UV/IR pole selection to MadLoop Oct 23, 2026
huashengshao and others added 4 commits October 23, 2026 12:47
Pin NinjaRS 41a4c00 and oneloopmaster 5b71043, whose main branches now expose
pole selection through native and generic master providers. Remove the local
numerical residue adapter while retaining MadLoop ABI4, context rebinding,
rescue policy and finite contributions.

Validate the real CLI update, 31 ordinary Rust tests, and fresh COLLIER tensor
and generated-process campaigns. All 7,280 tensor mode-cycle records and all
74 process configuration results exactly reproduce the previous implementation.
Retain compact provenance and distinguish current tests from historical
benchmark/export qualifications.
@ValentinHirschi

Copy link
Copy Markdown

@oliviermattelaer — responding to your comment:

Addressed in 94cbe712a. The current initialization already removes unavailable libraries from the requested list, preserves available fallbacks, and emits an actionable error for NinjaRS-only selection. I added a defensive CASE DEFAULT at TIR entry as well, before any cache lookup or coefficient access. An invalid ID now stops with a nonzero status; ID 8 explains how to install NinjaRS and regenerate/rebuild the process.

Compiled regressions exercise both Born and loop-induced dispatch, enabled/disabled caches, cold/warm calls, valid COLLIER dispatch, invalid IDs 6/8, and the existing unavailable-library fallback behavior. Generated-output fixtures were refreshed from actual generation.

@ValentinHirschi

Copy link
Copy Markdown

@oliviermattelaer — responding to your comment:

Fixed in 94cbe712a: both reduction warnings now check mlreductionlib in user_set. I also corrected the same case mismatch in the adjacent filter, rotation, and stability-threshold diagnostics.

New tests call the actual MadEvent card-treatment method with real partial cards: inappropriate explicit choices warn, appropriate choices/defaults do not, and explicit values remain unchanged. Running the broader pole tests also exposed a partial-card diagnostic that printed an uninitialized reduction-list string; that now prints the effective configured list and has its own compiled regression.

@ValentinHirschi

Copy link
Copy Markdown

@oliviermattelaer — responding to your comment:

Addressed the maintenance concern in 94cbe712a while retaining the existing return-code format. I did not allocate a new units-digit slot: all ten digits already have meanings, and doing so would break the public three-digit return codes and existing readers.

The XML and NLO log readers now share one documented contextual decoder and backend-label map, so the NinjaRS alias subtraction has one implementation. Added compiled producer-to-reader regressions using the actual grouped/ungrouped LO and fixed-order/matched NLO tally/writer blocks. They mix every legacy backend with ordinary and initialization NinjaRS DP/QP codes. Old files, missing/partial optional counters, repeated reads, aggregation, and non-mutation of input histograms are covered. The Fortran producers and serialized format are unchanged.

@ValentinHirschi

Copy link
Copy Markdown

@oliviermattelaer — responding to your comment:

Fixed in 94cbe712a. Added ninjars_install.default_reduction_library(has_born, libdir), returning a fresh list, and reused it in CLI timing/profile selection, direct check helpers, and the exporter. Explicit backend selections, export fallback order, and loop-induced policy are preserved.

Timing/profile/stability setup now copies caller-supplied MLOptions, including per-backend stability setup. Seven new regression tests cover default and explicit selections, reused processes, loop-induced/nonoptimized behavior, failure returns, and caller-option ownership across repeated calls.

@ValentinHirschi

Copy link
Copy Markdown

@oliviermattelaer — responding to your comment:

Addressed in e5a4a6544 by removing about 4.73 MB from the current tracked payload. Seven font binaries and the compressed raw CLI materialization (3.51 MB) are now fetched on demand from an immutable archived revision, with size/SHA256 verification, atomic installation, and ignored local paths. The README documents the one-time acquisition step; the subsequent PDF reproduction remains offline.

I retained the requested final PDF (currently 346 KB), Typst source, compact numerical summaries, vector figures, and font license. These are the reviewable report deliverable. The large callback capture was reduced from 1,350,447 to 131,790 bytes by retaining exactly the inputs and reference values consumed by its regression test: all 30 callbacks and all 480 numerical components are unchanged. The remaining small native fixtures are active regression oracles and require no downloads.

Validation includes download failure/retry tests, the existing 480-component Rust regression, and a fresh archived download followed by byte-identical PDF reproduction. Git history was not rewritten.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants