diff --git a/docs/data_structures/platform_specs.rst b/docs/data_structures/platform_specs.rst index bfc410405..e6d701430 100644 --- a/docs/data_structures/platform_specs.rst +++ b/docs/data_structures/platform_specs.rst @@ -90,6 +90,13 @@ E.g., in the command line or batch submission script: export LIBE_PLATFORM="perlmutter_g" +.. note:: + + The ``flux`` platform configures the :doc:`MPI Executor <../executor/ex_mpi>` + to launch applications by subprocessing ``flux run``. This is independent + of the :doc:`Flux Executor <../executor/ex_flux>`, which instead submits + jobs directly to Flux via its Python API bindings. + .. _known-platforms: Known Platforms List diff --git a/docs/executor/ex_base.rst b/docs/executor/ex_base.rst index 1a4d3cf31..ee3dd3269 100644 --- a/docs/executor/ex_base.rst +++ b/docs/executor/ex_base.rst @@ -1,7 +1,7 @@ Base Executor ============= -`Overview `__ \|\| **Base Executor** \|\| `MPI Executor `__ +`Overview `__ \|\| **Base Executor** \|\| `MPI Executor `__ \|\| `Flux Executor `__ .. automodule:: executor :no-undoc-members: diff --git a/docs/executor/ex_flux.rst b/docs/executor/ex_flux.rst new file mode 100644 index 000000000..01509c074 --- /dev/null +++ b/docs/executor/ex_flux.rst @@ -0,0 +1,99 @@ +Flux Executor +============== + +`Overview `__ \|\| `Base Executor `__ \|\| `MPI Executor `__ \|\| **Flux Executor** + +.. automodule:: flux_executor + :no-undoc-members: + +.. note:: + + The ``FluxExecutor`` requires the ``flux-core`` Python bindings to be + installed and importable (e.g., via ``conda install -c conda-forge flux-core`` + or Spack), and must either be run inside a Flux instance or be given a + Flux URI to connect to. If these bindings aren't available, ``FluxExecutor`` + won't be importable, but the standard :doc:`MPI Executor ` can + still submit to Flux via ``mpi_runner="flux"`` (see below). + +.. tab-set:: + + .. tab-item:: Flux Executor + + .. autoclass:: libensemble.executors.flux_executor.FluxExecutor + :members: + :show-inheritance: + :exclude-members: serial_setup, sim_default_app, gen_default_app, get_app, default_app, set_resources, get_task, set_workerID, set_worker_info, new_tasks_timing, add_platform_info, set_gen_procs_gpus + + .. automethod:: __init__ + + .. tab-item:: Flux Task + + Like the base :ref:`Task `, ``FluxTask`` objects are created and + returned by ``FluxExecutor.submit()``. ``poll()``, ``kill()``, and ``wait()`` + are overridden to query and control the job via Flux's own job-lifecycle + API instead of subprocess/signal-based mechanisms. + + .. autoclass:: libensemble.executors.flux_executor.FluxTask + :members: + :show-inheritance: + :exclude-members: reset + +Two ways to run under Flux +--------------------------- + +libEnsemble offers two independent ways to run applications under Flux: + +1. **MPI Executor with the Flux runner.** The standard :doc:`MPI Executor ` + can subprocess ``flux run`` like any other MPI launcher. This requires no + Python bindings, works with the usual resource-manager and GPU + auto-detection, and is a good default choice:: + + from libensemble.executors import MPIExecutor + + exctr = MPIExecutor(custom_info={"mpi_runner": "flux"}) + + or, equivalently, by selecting the built-in ``"flux"`` :ref:`platform`:: + + libE_specs["platform"] = "flux" + +2. **Flux Executor.** ``FluxExecutor`` instead submits jobs directly to Flux + through its Python API (``flux.job.submit_async``), bypassing any launcher + subprocess entirely. This is particularly useful inside containers or other + environments where a standard MPI launcher isn't available, and gives more + direct access to Flux's own job states. Since Flux handles aren't + thread-safe, ``FluxExecutor`` is best suited to process-based libEnsemble + runs (e.g., multiprocessing or MPI workers) rather than threaded workers. + +**Basic usage** + +.. code-block:: python + + from libensemble import Ensemble + from libensemble.executors.flux_executor import FluxExecutor + + exctr = FluxExecutor() + exctr.register_app(full_path="/home/user/forces.x", app_name="forces") + ensemble = Ensemble(executor=exctr) + +**In user simulation function**:: + + def sim_func(H, persis_info, sim_specs, libE_info): + exctr = libE_info["executor"] + + task = exctr.submit( + app_name="forces", + num_procs=8, + num_nodes=2, + stdout="out.txt", + stderr="err.txt", + ) + + # Wait for task to complete + task.wait() + +GPUs can be requested with ``num_gpus``, and jobs can be submitted without +blocking on start-up via ``wait_on_start`` (an integer gives a timeout in +seconds). See the :doc:`Forces tutorial <../tutorials/executor_forces_tutorial>` +for a complete calling script and simulation function using the ``forces.x`` +application; substituting ``FluxExecutor`` for ``MPIExecutor`` there requires +no other changes. diff --git a/docs/executor/ex_index.rst b/docs/executor/ex_index.rst index a4f33cb39..ac5c1c66b 100644 --- a/docs/executor/ex_index.rst +++ b/docs/executor/ex_index.rst @@ -1,6 +1,6 @@ .. _executor_index: -**Overview** \|\| `Base Executor `__ \|\| `MPI Executor `__ +**Overview** \|\| `Base Executor `__ \|\| `MPI Executor `__ \|\| `Flux Executor `__ Executors ========= @@ -14,6 +14,7 @@ portable interface for running and managing user applications. ex_overview ex_base ex_mpi + ex_flux The **Executor** provides a portable interface for running applications on any system and any number of compute resources. diff --git a/docs/executor/ex_mpi.rst b/docs/executor/ex_mpi.rst index 59a36f9e5..bddd2caa1 100644 --- a/docs/executor/ex_mpi.rst +++ b/docs/executor/ex_mpi.rst @@ -1,7 +1,7 @@ MPI Executor ============ -`Overview `__ \|\| `Base Executor `__ \|\| **MPI Executor** +`Overview `__ \|\| `Base Executor `__ \|\| **MPI Executor** \|\| `Flux Executor `__ .. automodule:: mpi_executor :no-undoc-members: diff --git a/docs/executor/ex_overview.rst b/docs/executor/ex_overview.rst index 2cbbed265..fe4d6c45e 100644 --- a/docs/executor/ex_overview.rst +++ b/docs/executor/ex_overview.rst @@ -1,7 +1,7 @@ Overview ======== -**Overview** \|\| `Base Executor `__ \|\| `MPI Executor `__ +**Overview** \|\| `Base Executor `__ \|\| `MPI Executor `__ \|\| `Flux Executor `__ The **Executor** provides a portable interface for running applications on any system and any number of compute resources. diff --git a/docs/platforms/flux.rst b/docs/platforms/flux.rst new file mode 100644 index 000000000..dd1ce1e2b --- /dev/null +++ b/docs/platforms/flux.rst @@ -0,0 +1,35 @@ +====================== +libEnsemble with Flux +====================== + +Flux_ is a flexible, hierarchical resource manager and job scheduler used +on a growing number of systems, including LLNL's El Capitan. + +libEnsemble can read Flux resource lists and partition these to workers. By +default this is done by :ref:`reading an environment variable` +(``FLUX_URI``), which is then used to query ``flux resource list`` for the +available nodes. + +There are two independent ways to run applications under Flux with +libEnsemble; see the :doc:`Flux Executor <../executor/ex_flux>` page for a +full comparison and usage examples of both: + +1. Tell the :doc:`MPIExecutor<../executor/ex_index>` to use ``flux run`` as its + launcher, either directly:: + + from libensemble.executors import MPIExecutor + exctr = MPIExecutor(custom_info={"mpi_runner": "flux"}) + + or via the built-in ``flux`` platform:: + + libE_specs["platform"] = "flux" + +2. Use the native :doc:`FluxExecutor <../executor/ex_flux>`, which submits + jobs directly to Flux via its Python API instead of subprocessing a + launcher. This requires the ``flux-core`` Python bindings and is + particularly useful in containerized environments:: + + from libensemble.executors.flux_executor import FluxExecutor + exctr = FluxExecutor() + +.. _Flux: https://flux-framework.org/ diff --git a/docs/platforms/platforms_index.rst b/docs/platforms/platforms_index.rst index 5983499ce..4628d6663 100644 --- a/docs/platforms/platforms_index.rst +++ b/docs/platforms/platforms_index.rst @@ -226,6 +226,7 @@ libEnsemble on specific HPC systems. aurora bebop + flux frontier improv perlmutter diff --git a/docs/resource_manager/resource_detection.rst b/docs/resource_manager/resource_detection.rst index e294b82b9..d27bcfcfa 100644 --- a/docs/resource_manager/resource_detection.rst +++ b/docs/resource_manager/resource_detection.rst @@ -16,8 +16,13 @@ SLURM SLURM_NODELIST COBALT COBALT_PARTNAME LSF LSB_HOSTS/LSB_MCPU_HOSTS PBS PBS_NODEFILE +Flux FLUX_URI =========== =========================== +Flux is detected via the ``FLUX_URI`` environment variable, but unlike the +other schedulers above, the nodelist itself is obtained by running +``flux resource list`` rather than parsing an environment variable directly. + These environment variable names can be modified via the :ref:`resource_info` :class:`libE_specs` option. diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt index 92c7a5601..bc286a828 100644 --- a/docs/spelling_wordlist.txt +++ b/docs/spelling_wordlist.txt @@ -19,6 +19,7 @@ bugfix calc cancelled Cancelling +Capitan chwirut COBYLA comms @@ -159,6 +160,7 @@ str subprocess subprocessed subprocesses +subprocessing symlinks tasmanian th diff --git a/libensemble/executors/mpi_executor.py b/libensemble/executors/mpi_executor.py index 5a0190d5c..eea5eed2d 100644 --- a/libensemble/executors/mpi_executor.py +++ b/libensemble/executors/mpi_executor.py @@ -50,7 +50,7 @@ class MPIExecutor(Executor): .. parsed-literal:: **'mpi_runner'** [string]: - Select runner: `'mpich'`, `'openmpi'`, `'aprun'`, `'srun'`, `'jsrun'`, `'custom'` + Select runner: `'mpich'`, `'openmpi'`, `'aprun'`, `'srun'`, `'jsrun'`, `'flux'`, `'custom'` All except `'custom'` relate to runner classes in libEnsemble. Custom allows user to define their own run-lines but without parsing arguments or making use of auto-resources.