Repository navigation
AmpliCol matrix elements in mg7 ([amplicol] processes) - #291
Draft
theoheimel wants to merge 96 commits into
Draft
theoheimel wants to merge 96 commits into
theoheimel wants to merge 96 commits into
Conversation
…_max_weight gridpacks New madspace EventStream: a coordinator thread keeps generation jobs running, buffers each channel's unweighted events in launch order and picks the channel of every event from the current cross-section estimates. Reproducible from the seed, nothing written to disk. Shared helpers factored out of EventGenerator (event_utils) and gridpack.py (gridpack_setup.py).
…(valid .pyi stub), gridpack section in launch test run card
…t-streaming-event-gen # Conflicts: # madgraph/interface/tutorials/mg7.py # madgraph/iolibs/template_files/mg7/run_card.toml
# Conflicts: # madgraph/iolibs/template_files/mg7/gridpack.py
…and the EventStream API reference
…eat-multi-step-unweighting The bias of the custom functions is now masked in combined_event_layout and divided out in copy_channel_event, so EventStream handles it as well. Custom functions of a gridpack are loaded in gridpack_setup.load_context_data.
The Integrand looks up the matrix-element outputs by name. Only |M|^2 is required: diagram_amp2 only for diagram-based channel weights, and missing color, helicity and diagram indices are written as -1. With return_matrix_element, DifferentialCrossSection also returns |M|^2 itself, which the Integrand passes on as the matrix_element output.
With return_matrix_element, the Integrand also returns the flavor index passed to the matrix element and, with several differential cross sections, which one was evaluated, so that a later step can evaluate its own matrix element on the same events.
Groups of channels can be given a sequence of WeightingSteps, each with its own matrix element, for example a fast surrogate in the integrand followed by the full matrix element. A step multiplies the weights by the ratio of its matrix element to the previous one. An unweighting step then unweights the ratios since the last unweighting step, times the overweight carried from it, with its own maximum weight, truncation and freeze. A reweighting step keeps the ratio in the weight. The events that pass the first unweighting stay on the device and are collected per group and context until a batch is full, or until the end of a round. The decisions are made per channel in job order, with random numbers drawn per event when the events are first unweighted, so the results do not depend on the batching or the number of threads. Integrals, event counts and the sharing of the events between channels refer to the last step, the importance sampling to the first. The color, helicity and diagram indices are taken from the first step that returns them, or replaced with override_outputs. Without groups the generation is unchanged.
Like survey and generate: with matrix elements or custom functions in Python, the worker threads otherwise wait for the GIL held by the calling thread.
weighting_groups_to_json/from_json store the groups with their matrix elements by index, like the channel functions store theirs, so that a gridpack can restore them. With include_estimates, the channel JSON also holds the maximum weights of the steps, which stay fixed when the channel is loaded again. Tested with fix_max_weights() and a reload into a fresh context.
Destroying the stream waits for its running jobs, which can need the GIL, for example with custom functions or matrix elements in Python. Destroyed from Python with the GIL held, this deadlocked.
The EventStream takes the weighting groups like the EventGenerator and runs the events of their channels through the same pipeline, which now hands the events after the last step to a sink: the EventGenerator writes them to the channel files, the stream buffers them as finished jobs. All maxima have to be fixed, so every decision only depends on the event itself. As for the first step, the integral after the steps only includes the jobs whose events were requested, so the stream stays a pure function of the seed. Also passes the biases of the jobs on to their integration, which the stream dropped before.
The default mode of the WeightingStep constructor made pybind11-stubgen fail the wheel build.
A surrogate of the matrix element replaces it in the integrand, and the full
matrix element unweights or reweights the events in a second pass, as one
weighting step per subprocess. The surrogate is a .so library with the inputs
of the madmatrix libraries, loaded per subprocess with {subprocess} and
{device} in its path, or a Python file defining
custom_surrogate(momenta, pdg_ids, alpha_s). In unweight mode, the
first_pass_* settings apply to the unweighting with the surrogate and those of
[generation] to the second pass. Needs the denominators sde_strategy; no
gridpacks yet.
With weighting steps, the run summary shows the efficiency of the first unweighting, before and after cuts, and of each step relative to the one before, next to the overall one, their product. The channels keep showing the overall efficiency. The accepted counts of the steps are now rescaled like the final count when a maximum of an earlier step rises, and reweighting steps count their events too.
UMAMI 1.2 adds the output UMAMI_OUT_MATRIX_ELEMENT_RATIO, the ratio of the matrix element to a reference defined by the caller. A weighting step whose matrix elements return it (MatrixElement::matrix_element_ratio_out) uses the ratio directly, and the ratio times the previous matrix element is the reference of the next step. Integrands reject such matrix elements.
A non-negative channel passed to MatrixElement is given to the UMAMI channel input as a constant, which is then not an external input. Integrands and weighting steps can so evaluate one channel of a matrix element that treats its channels as independent processes.
The discrete two-solution choices of a color-ordered mapping were never passed to it by the integrand. They are now sampled uniformly.
A subprocess whose matrix element has no diagrams, like a color-ordered one, has no propagators to insert. Its events carry the diagram index -1.
launch.py stays the entry point and re-exports what the launcher modules define. The code is moved without changes: common (madspace import, logging, helpers), observables, subprocess_madmatrix, madgraph_process, options, model_widths, postprocess, driver and cli. model_widths does not import madspace, so the run interface loads the model without it.
The subprocess returns its weighting steps, so the weighting groups do not depend on how a subprocess builds them. A [surrogate] subprocess returns its second pass with the full matrix element, as before.
'generate p p > j j j [amplicol]' keeps the process definition for the mg7 output, which computes its matrix elements with AmpliCol, instead of generating MG diagrams. Such processes can be mixed with ordinary ones, and are refused by the other outputs and for decay chains, coupling orders, s-channel or particle restrictions and polarizations.
Each [amplicol] process gets a SubProcesses/A<n>_<name> directory with the leading- and full-colour pyamplicol artifacts, the sources of their ratio library and a makefile building both libraries into lib/. Every colour ordering becomes its own entry of subprocesses.json, with backend 'amplicol', and is never merged with other subprocesses.
A fixed channel is not an external input of a matrix element. The check that the random inputs of a step are passed in now only looks at them.
An output without MG diagrams skips their export. The [amplicol] processes stand in for them where the output needs MG processes: the run card defaults, proc_characteristics and the model reference.
The colour accuracy of the [amplicol] events, lc or full, whether the leading-colour events are re- or unweighted to it, and the settings of the leading-colour unweighting.
The matrix elements of a subprocess are loaded by load_matrix_elements, and those of its integrand built by integrand_matrix_elements, which a subprocess of another backend overrides. A flavor can carry its own factor.
Every colour ordering of an [amplicol] process is integrated at leading colour with the colour-ordered phase-space mapping of its UMAMI channel, and its events are re- or unweighted to full colour with the ratio library. The parameters of the param card are passed to the AmpliCol libraries, which the launcher builds when they are missing. GPUs, gridpacks and dummy matrix elements are refused.
MG replaces the light quarks and leptons of a process definition by flavour groups, which pyamplicol does not know: the export passes the original flavours of a leg, or the members of its groups. The directory is named with the multiparticle labels of the definition.
…mbolica after an hour
…[amplicol] libraries, accept system installs
… first '-' of its name
The freeze compared count_accepted, which is scaled by the first maximum at the time of the step decision and so depends on the thread timing. Count the expected accepted events of each step on its own, like _freeze_count.
… configuration The resolved physics lists every s-channel resonance a tree current can carry whose mass and width are not zero in the model, without the threshold and width filters of the default parameters: the launch decides with the parameter card which go on shell. The phase-space channels follow every pairwise nested or disjoint set of colour-singlet resonances, the non-resonant one first and the larger ones next, enumerated once per process and lazily up to the caps. A coloured resonance is listed in channels[].colored_resonances for the launch to check; a singlet decaying to coloured and colourless particles, or never to one complete colour line, is not followed and is warned about.
A non-finite ratio, or one with an earlier matrix element of 0, could become the maximum of a step and then reject every event, or abort a reweighting step. Such events now get the weight 0, are counted per channel in GeneratorStatus.count_invalid_step_values and a warning is logged after the survey and generate().
The phase-space channels of a subprocess follow the resonances found at output that go on shell with the param card: a positive width and a mass above the masses of the decay legs. The others are dropped, the channels that then agree merged and the flavours' channels and LHE resonances renumbered. A coloured resonance is an error only when it goes on shell.
… maxima set_target_count() only re-applied the budget to the first maximum, so a step maximum chosen under generate()'s generous round-one target stayed too low for a channel with a small target. EventGenerator now re-applies it to the steps at the end of each round, with the steps drained so that it does not depend on the thread timing.
The fallback put all of the weight on the generating channel, so the weights were not a function of the event alone, and it compared a final channel index with a mapping index. Use uniform weights over the masked channels instead, which makes the channel_index argument of density_channel_weights and MappingChannelWeights unused, so it is removed.
WeightingGroup takes optional first_pass_max_overweight_truncation and first_pass_freeze_max_weight_after that replace the GeneratorConfig values for the first unweighting of the channels of the group only, so that looser first-pass settings do not leak into the channels without steps. They are stored in the group JSON if set.
Channels without fixed maximum weights return no events, but their integral was still added to the cross section of status().
…pace-amplicol # Conflicts: # docs/source/madspace/cpp-api.rst # docs/source/madspace/python-api.rst # madgraph/iolibs/template_files/mg7/launch.py # madspace/generate_pyi.py # madspace/include/madspace/driver/io.hpp # madspace/include/madspace/phasespace/integrand.hpp # madspace/src/driver/channel_generator.cpp # madspace/src/driver/event_generator.cpp # madspace/src/phasespace/integrand.cpp # madspace/src/python/madspace.cpp # pyamplicol/umami_example/test_integrate.py
…es batch_size <= 0
…nd report its message
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AmpliCol as a second matrix-element backend of the mg7 output:
generate p p > j j j [amplicol], alone or mixed with madmatrix processes in one output. Based on the pyamplicol imported intopyamplicol/(merge-pyamplicol), which the output installs topyamplicol/installif needed.This PR also contains #253 (in-memory event streaming) and #285 (multi-step re- and unweighting), which are closed in favour of it. Their changes are summarized in the sections below.
How it works
Output: each
[amplicol]process definition goes toSubProcesses/A<n>_<name>/. pyamplicol generates a leading-colour artifact (lc), a full-colour one (full), and the sources of a UMAMI library returning their ratio (reweight). No MG diagrams are generated.Flavour classes: subprocesses related by a flavour relabelling that is a symmetry of the allowed vertices are generated once (pyamplicol's strict flavour-blind grouping) and evaluated through their representative. The rows of a UMAMI selector become the options of one flavour, written to the LHE file in proportion to their factors.
Subprocesses and phase space: every UMAMI channel becomes a madspace subprocess with the LC matrix element at a fixed channel. Each of its colour orders is integrated over several colour-ordered phase-space channels:
The channels are weighted by the densities of their mappings, through madspace's multichannel path, so the channel weight network and MadNIS apply. The resonances in the event record come from a channel drawn in proportion to its weight, with the charge of the decay products (W+ or W-).
Colour accuracy: a weighting step with the ratio library re- or unweights the events to full colour, using the ratio-returning weighting steps (see below).
Run card: a new
[amplicol]section withcolor_accuracy(full,lc),mode(reweight,unweight) and the first-pass settings.Gridpacks
[amplicol]processes support gridpacks, including thefix_max_weightmode andEventStream.MadgraphProcess.load_matrix_element, which records each load's path, init path (param card or AmpliCol artifact) and parameters. The gridpack loads the same list in the same order, so the matrix-element indices match the run.data/groups.json.lcandfullartifacts are copied toSubProcesses/A*. The libraries are self-contained, with no Symbolica dependency at run time.--max_overweight_truncationand--freeze_max_weight_afterset the final pass. The new--first_pass_max_overweight_truncationand--first_pass_freeze_max_weight_afterset the leading-colour unweighting; their defaults come from[amplicol]ingrid_run_card.toml. Colour accuracy and mode are fixed when the gridpack is made.include_source = truefor[amplicol]processes.madspace
MatrixElementwith a fixed channel index (no channel input). The JSON of the weighting groups now keeps this channel; before, a reloaded ratio step lost it.MappingChannelWeights: multichannel weights from the densities of the channel mappings, per flavour. It also draws the diagram that the event record takes its resonances from.Integranduses it in place ofPropagatorChannelWeights.SubprocArgs.pdg_id_weightsdraws each flavour's PDG-id options in proportion to their weights. The weights are saved with the completer.pt^2, notm_T^2, now bounds the momentum transfer to the beam in the 2->3 step off the opposite beam. The momentum transfers belowm_t^2were never sampled, so theg g > t t~ gchannels with the top pair on one side came out 25% low.Launcher (
launch.py)launch.pyis split into modules (common,observables,subprocess_madmatrix,madgraph_process,options,model_widths,postprocess,driver,cli);launch.pyre-exports their names. The e+e- j reference cross section is reproduced exactly.AmplicolSubprocess:Interface and output
[amplicol]syntax: process definitions are kept without diagrams. Decay chains, explicit orders, s-channel and particle restrictions, polarisations, non 2->n processes and outputs other than mg7 are refused.madgraph/iolibs/amplicol_worker.py). Symbolica holds a per-user lock until its process exits, so MadGraph sessions no longer block each other; a worker waits while another one runs.pyamplicol/install, and the makefiles use itsrusticol-config.umami.his madspace's.[amplicol]definition cover every particle of each final-state leg, so thatp p > e+ e- j jgets jet histograms, not gluon ones.[amplicol]:include_source;pyamplicol
process.flavor_blind_strict): a member is routed through a representative only if their relation passes the vertex-symmetry check.payloads.pacbin. Forp p > j j j [amplicol], the output takes 85 s instead of 346 s and writes 73 files instead of 2033; a runtime load opens 4 files instead of 207.Event streaming (from #253)
Gridpacks made with
[gridpack] run_mode = "fix_max_weight"shipbin/event_stream.py, a Python API that generates events in memory, with no survey, no combine pass and nothing written to disk (e.g. to pass events directly to Pythia).device,seed, batch sizes,thread_pool_size, cut repetition);Nonefalls back togrid_run_card.toml.batches()yields anEventBatchof LHE fields as numpy arrays or CPU torch tensors: event fields(n,), particle fields(n, max_particle_count), plusparticle_countandparticle_mask. Weights are in units of the channel's maximum weight.madspace::EventStream(driver/event_stream.{hpp,cpp}): a coordinator thread keepsChannelEventGeneratorjobs running and fills per-channel buffers of unweighted events. Each event's channel is drawn ∝ abs-integral estimate ÷ mean unweighted weight, which stays correct with overweight events.ChannelEventGeneratorworks without event/weight files (fill_event_bufferssplit out); shared LHE/integral helpers moved todriver/event_utils.{hpp,cpp}; the gridpack setup moved togridpack_setup.py(always inbin/, importable from any directory).Multi-step re- and unweighting (from #285)
Groups of channels can get a sequence of re- or unweighting steps, each with its own matrix element, e.g. for NN surrogates or leading-to-full colour reweighting (as used by
[amplicol]above).diagram_amp2only for diagram-based channel weights. Colour, helicity and diagram indices come from the first step that returns them, or later steps withoverride_outputs.fix_max_weightsandEventStreamsupport steps.copy_channel_eventand inEventStream; GIL deadlocks with Python matrix elements/functions inoptimize/fix_max_weightsand onEventStreamdestruction.Tests
tests/unit_tests/interface/test_amplicol_syntax.py: syntax, export (flavour groups and classes, multiparticles, coupling orders, mirrored beams, resonances).tests/acceptance_tests/test_mg7_amplicol.py: cross sections against madmatrix within errors forg g > g g g(reweight and unweight; LC = 9/8 x full);g g > t t~ g;p p > j j j;p p > e+ e- j, with the Z in the event record;p p > e+ ve j, with the W+ and never the W-;[amplicol]+ madmatrix output;p p > e+ e- j [amplicol]+ madmatrixp p > mu+ mu- j(unweight mode), which reproduces the run's cross section and writes the Z.test_mapping_channel_weights.py,test_color_ordered_massive_side.py, the weighted LHE flavour options intest_lhe.py, and the channel round trip intest_weighting_steps.py.test_event_stream.py(identical events across threads and batch splits, seed, weights, channel fractions, no files written) andtest_weighting_steps.py(surrogate→true-ME chains reproduce cross sections and distributions for unweight, reweight and reweight→unweight; thread independence;fix_max_weights+ JSON reload;EventStream),test_integrand_me_outputs.py.All pass.
Known limitations
p p > j j jatoutput.[amplicol]generations cannot run at the same time while Symbolica takes its restricted thread at import.