Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/data_structures/platform_specs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,13 @@ E.g., in the command line or batch submission script:

export LIBE_PLATFORM="perlmutter_g"

.. note::

The ``flux`` platform configures the :doc:`MPI Executor <../executor/ex_mpi>`
to launch applications by subprocessing ``flux run``. This is independent
of the :doc:`Flux Executor <../executor/ex_flux>`, which instead submits
jobs directly to Flux via its Python API bindings.

.. _known-platforms:

Known Platforms List
Expand Down
2 changes: 1 addition & 1 deletion docs/executor/ex_base.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Base Executor
=============

`Overview <ex_overview.html>`__ \|\| **Base Executor** \|\| `MPI Executor <ex_mpi.html>`__
`Overview <ex_overview.html>`__ \|\| **Base Executor** \|\| `MPI Executor <ex_mpi.html>`__ \|\| `Flux Executor <ex_flux.html>`__

.. automodule:: executor
:no-undoc-members:
Expand Down
99 changes: 99 additions & 0 deletions docs/executor/ex_flux.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
Flux Executor
==============

`Overview <ex_overview.html>`__ \|\| `Base Executor <ex_base.html>`__ \|\| `MPI Executor <ex_mpi.html>`__ \|\| **Flux Executor**

.. automodule:: flux_executor
:no-undoc-members:

.. note::

The ``FluxExecutor`` requires the ``flux-core`` Python bindings to be
installed and importable (e.g., via ``conda install -c conda-forge flux-core``
or Spack), and must either be run inside a Flux instance or be given a
Flux URI to connect to. If these bindings aren't available, ``FluxExecutor``
won't be importable, but the standard :doc:`MPI Executor <ex_mpi>` can
still submit to Flux via ``mpi_runner="flux"`` (see below).

.. tab-set::

.. tab-item:: Flux Executor

.. autoclass:: libensemble.executors.flux_executor.FluxExecutor
:members:
:show-inheritance:
:exclude-members: serial_setup, sim_default_app, gen_default_app, get_app, default_app, set_resources, get_task, set_workerID, set_worker_info, new_tasks_timing, add_platform_info, set_gen_procs_gpus

.. automethod:: __init__

.. tab-item:: Flux Task

Like the base :ref:`Task <task_tag>`, ``FluxTask`` objects are created and
returned by ``FluxExecutor.submit()``. ``poll()``, ``kill()``, and ``wait()``
are overridden to query and control the job via Flux's own job-lifecycle
API instead of subprocess/signal-based mechanisms.

.. autoclass:: libensemble.executors.flux_executor.FluxTask
:members:
:show-inheritance:
:exclude-members: reset

Two ways to run under Flux
---------------------------

libEnsemble offers two independent ways to run applications under Flux:

1. **MPI Executor with the Flux runner.** The standard :doc:`MPI Executor <ex_mpi>`
can subprocess ``flux run`` like any other MPI launcher. This requires no
Python bindings, works with the usual resource-manager and GPU
auto-detection, and is a good default choice::

from libensemble.executors import MPIExecutor

exctr = MPIExecutor(custom_info={"mpi_runner": "flux"})

or, equivalently, by selecting the built-in ``"flux"`` :ref:`platform<datastruct-platform-specs>`::

libE_specs["platform"] = "flux"

2. **Flux Executor.** ``FluxExecutor`` instead submits jobs directly to Flux
through its Python API (``flux.job.submit_async``), bypassing any launcher
subprocess entirely. This is particularly useful inside containers or other
environments where a standard MPI launcher isn't available, and gives more
direct access to Flux's own job states. Since Flux handles aren't
thread-safe, ``FluxExecutor`` is best suited to process-based libEnsemble
runs (e.g., multiprocessing or MPI workers) rather than threaded workers.

**Basic usage**

.. code-block:: python

from libensemble import Ensemble
from libensemble.executors.flux_executor import FluxExecutor

exctr = FluxExecutor()
exctr.register_app(full_path="/home/user/forces.x", app_name="forces")
ensemble = Ensemble(executor=exctr)

**In user simulation function**::

def sim_func(H, persis_info, sim_specs, libE_info):
exctr = libE_info["executor"]

task = exctr.submit(
app_name="forces",
num_procs=8,
num_nodes=2,
stdout="out.txt",
stderr="err.txt",
)

# Wait for task to complete
task.wait()

GPUs can be requested with ``num_gpus``, and jobs can be submitted without
blocking on start-up via ``wait_on_start`` (an integer gives a timeout in
seconds). See the :doc:`Forces tutorial <../tutorials/executor_forces_tutorial>`
for a complete calling script and simulation function using the ``forces.x``
application; substituting ``FluxExecutor`` for ``MPIExecutor`` there requires
no other changes.
3 changes: 2 additions & 1 deletion docs/executor/ex_index.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _executor_index:

**Overview** \|\| `Base Executor <ex_base.html>`__ \|\| `MPI Executor <ex_mpi.html>`__
**Overview** \|\| `Base Executor <ex_base.html>`__ \|\| `MPI Executor <ex_mpi.html>`__ \|\| `Flux Executor <ex_flux.html>`__

Executors
=========
Expand All @@ -14,6 +14,7 @@ portable interface for running and managing user applications.
ex_overview
ex_base
ex_mpi
ex_flux

The **Executor** provides a portable interface for running applications on any system and
any number of compute resources.
Expand Down
2 changes: 1 addition & 1 deletion docs/executor/ex_mpi.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
MPI Executor
============

`Overview <ex_overview.html>`__ \|\| `Base Executor <ex_base.html>`__ \|\| **MPI Executor**
`Overview <ex_overview.html>`__ \|\| `Base Executor <ex_base.html>`__ \|\| **MPI Executor** \|\| `Flux Executor <ex_flux.html>`__

.. automodule:: mpi_executor
:no-undoc-members:
Expand Down
2 changes: 1 addition & 1 deletion docs/executor/ex_overview.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Overview
========

**Overview** \|\| `Base Executor <ex_base.html>`__ \|\| `MPI Executor <ex_mpi.html>`__
**Overview** \|\| `Base Executor <ex_base.html>`__ \|\| `MPI Executor <ex_mpi.html>`__ \|\| `Flux Executor <ex_flux.html>`__

The **Executor** provides a portable interface for running applications on any system and
any number of compute resources.
Expand Down
35 changes: 35 additions & 0 deletions docs/platforms/flux.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
======================
libEnsemble with Flux
======================

Flux_ is a flexible, hierarchical resource manager and job scheduler used
on a growing number of systems, including LLNL's El Capitan.

libEnsemble can read Flux resource lists and partition these to workers. By
default this is done by :ref:`reading an environment variable<resource_detection>`
(``FLUX_URI``), which is then used to query ``flux resource list`` for the
available nodes.

There are two independent ways to run applications under Flux with
libEnsemble; see the :doc:`Flux Executor <../executor/ex_flux>` page for a
full comparison and usage examples of both:

1. Tell the :doc:`MPIExecutor<../executor/ex_index>` to use ``flux run`` as its
launcher, either directly::

from libensemble.executors import MPIExecutor
exctr = MPIExecutor(custom_info={"mpi_runner": "flux"})

or via the built-in ``flux`` platform::

libE_specs["platform"] = "flux"

2. Use the native :doc:`FluxExecutor <../executor/ex_flux>`, which submits
jobs directly to Flux via its Python API instead of subprocessing a
launcher. This requires the ``flux-core`` Python bindings and is
particularly useful in containerized environments::

from libensemble.executors.flux_executor import FluxExecutor
exctr = FluxExecutor()

.. _Flux: https://flux-framework.org/
1 change: 1 addition & 0 deletions docs/platforms/platforms_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,7 @@ libEnsemble on specific HPC systems.

aurora
bebop
flux
frontier
improv
perlmutter
Expand Down
5 changes: 5 additions & 0 deletions docs/resource_manager/resource_detection.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,13 @@ SLURM SLURM_NODELIST
COBALT COBALT_PARTNAME
LSF LSB_HOSTS/LSB_MCPU_HOSTS
PBS PBS_NODEFILE
Flux FLUX_URI
=========== ===========================

Flux is detected via the ``FLUX_URI`` environment variable, but unlike the
other schedulers above, the nodelist itself is obtained by running
``flux resource list`` rather than parsing an environment variable directly.

These environment variable names can be modified via the :ref:`resource_info<resource_info>`
:class:`libE_specs<libensemble.specs.LibeSpecs>` option.

Expand Down
2 changes: 2 additions & 0 deletions docs/spelling_wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ bugfix
calc
cancelled
Cancelling
Capitan
chwirut
COBYLA
comms
Expand Down Expand Up @@ -159,6 +160,7 @@ str
subprocess
subprocessed
subprocesses
subprocessing
symlinks
tasmanian
th
Expand Down
2 changes: 1 addition & 1 deletion libensemble/executors/mpi_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ class MPIExecutor(Executor):
.. parsed-literal::

**'mpi_runner'** [string]:
Select runner: `'mpich'`, `'openmpi'`, `'aprun'`, `'srun'`, `'jsrun'`, `'custom'`
Select runner: `'mpich'`, `'openmpi'`, `'aprun'`, `'srun'`, `'jsrun'`, `'flux'`, `'custom'`
All except `'custom'` relate to runner classes in libEnsemble.
Custom allows user to define their own run-lines but without parsing
arguments or making use of auto-resources.
Expand Down
Loading