From 81913c843be7b2e897cb3e84bf3367a2ca17423e Mon Sep 17 00:00:00 2001 From: Erik Schnetter Date: Thu, 10 Sep 2026 12:37:34 -0400 Subject: [PATCH] Call MPI_Abort when a test process fails A test file that dies from an uncaught exception (a failing `@test`, an `MPIError`, ...) exits nonzero, and `MPI.Init`'s atexit hook then skips `MPI.Finalize()`, correctly, since it is collective. But the failing rank still runs its object finalizers afterwards, and `free(::Win)`, `free(::Comm)` and `close(::FileHandle)` all make collective MPI calls, gated only on `!Finalized()` -- which is false on this path. So the failing rank *enters* a collective while its peers are blocked somewhere else entirely. Every process is still alive, so there is no dead process for the launcher to notice, and the job hangs until the CI job-level `timeout-minutes: 20` expires. The run is then reported as a job timeout rather than as a named failing test, with no information about the other test files. Julia runs atexit hooks before object finalizers, so an atexit hook is the right place to call `MPI_Abort`: it preempts both the blocked peers and the finalizer deadlock. Load it into every test process with `julia -L` rather than editing each test file, since only 26 of the 50 test files include `common.jl` and a new test file would otherwise have to remember to opt in. Note that launchers truncate the abort code to 8 bits when reporting it as the job's exit status, so the errcode has to be sanitized: passing a raw exit code of 256 makes mpiexec exit 0, turning a hard failure into a green run. Verified on both MPICH and Open MPI. Also correct usage.md and external.md, which both described the atexit hook as calling `MPI.Finalize` when Julia exits, omitting the exit-code condition that causes all of the above. --- docs/src/external.md | 2 +- docs/src/usage.md | 8 +++++++- test/abort_on_error.jl | 44 ++++++++++++++++++++++++++++++++++++++++++ test/runtests.jl | 10 +++++++++- 4 files changed, 61 insertions(+), 3 deletions(-) create mode 100644 test/abort_on_error.jl diff --git a/docs/src/external.md b/docs/src/external.md index 37166fd25..6878749a2 100644 --- a/docs/src/external.md +++ b/docs/src/external.md @@ -23,7 +23,7 @@ ccall((:cfunc2, lib), Cint, (Ptr{MPI.MPI_Comm},), comm) ## Object finalizers and `MPI.Finalize` -External libraries may allocate their own MPI handles (e.g., create or duplicate MPI communicators), which need to be cleaned up before MPI is finalized. If these are attached to [object finalizers](https://docs.julialang.org/en/v1/base/base/#Base.finalizer), they may not be guaranteed to be called before `MPI.Finalize`, which can result in an error upon program exit. (By default, MPI.jl will install an [`atexit`](https://docs.julialang.org/en/v1/base/base/#Base.atexit) hook that calls `MPI.Finalize` if it hasn't already been invoked.) +External libraries may allocate their own MPI handles (e.g., create or duplicate MPI communicators), which need to be cleaned up before MPI is finalized. If these are attached to [object finalizers](https://docs.julialang.org/en/v1/base/base/#Base.finalizer), they may not be guaranteed to be called before `MPI.Finalize`, which can result in an error upon program exit. (By default, MPI.jl will install an [`atexit`](https://docs.julialang.org/en/v1/base/base/#Base.atexit) hook that calls `MPI.Finalize` if it hasn't already been invoked. Note that this hook does nothing if Julia is exiting with a nonzero exit code, since `MPI.Finalize` is collective: in that case MPI is still initialized when the finalizers run, so a finalizer that makes a collective call such as `MPI_Comm_free` can block indefinitely against ranks that are elsewhere.) There are two typical solutions to this problem: diff --git a/docs/src/usage.md b/docs/src/usage.md index ef4969f6c..5d17bd552 100644 --- a/docs/src/usage.md +++ b/docs/src/usage.md @@ -22,7 +22,13 @@ println("Hello world, I am $(MPI.Comm_rank(comm)) of $(MPI.Comm_size(comm))") MPI.Barrier(comm) ``` -Calling [`MPI.Finalize()`](@ref) at the end of the program is optional, as it will be called automatically when Julia exits. +Calling [`MPI.Finalize()`](@ref) at the end of the program is optional, as it will be +called automatically when Julia exits *normally*, i.e. with exit code `0`. If Julia +exits with a nonzero exit code, for example because of an uncaught exception, then +`MPI.Finalize()` is deliberately not called: it is collective, and the other ranks are +not finalizing. Those ranks will instead remain blocked in their next collective call. +If you want a rank that fails to bring the whole job down promptly, call +[`MPI.Abort`](@ref) yourself. The program can then be launched via an MPI launch command (typically `mpiexec`, `mpirun` or `srun`), e.g. ``` diff --git a/test/abort_on_error.jl b/test/abort_on_error.jl new file mode 100644 index 000000000..59f73649c --- /dev/null +++ b/test/abort_on_error.jl @@ -0,0 +1,44 @@ +# Loaded into every MPI test process via `julia -L abort_on_error.jl test_foo.jl`; see +# the `cmd` definition in runtests.jl. Deliberately not named `test_*.jl`, so that +# `istest` in runtests.jl does not pick it up as a test file. +# +# When a test file dies from an uncaught exception (a failing `@test`, an `MPIError`, +# ...), Julia prints it and calls `exit(1)`. MPI.jl's own atexit hook then skips +# `MPI.Finalize()`, because it is collective. Two bad things follow: +# +# * the other ranks stay blocked in their next collective, and +# * this rank's object finalizers still run, and `free(::Win)`, `free(::Comm)` and +# `close(::FileHandle)` are collective, so this rank blocks too -- a deadlock in +# which every process is still alive and which therefore no launcher can detect. +# +# atexit hooks run before object finalizers, so aborting here heads off both. +using MPI + +atexit() do exitcode + # Only interfere with abnormal termination. + exitcode == 0 && return + # MPI must be up, and not already finalized: calling MPI_Abort after MPI_Finalize + # is not allowed. (Several tests assert after their own MPI.Finalize() call.) + (MPI.Initialized() && !MPI.Finalized()) || return + + # MPI_Abort terminates this process immediately: no later atexit hook, object + # finalizer or stream flush will run. The `Test Failed / Expression: ...` detail + # block goes to stdout, which may be block-buffered under mpiexec, so get it out + # first rather than risk losing the only description of what went wrong. A broken + # stream must not be able to skip the abort itself, hence the try/catch. + try + flush(stdout) + flush(stderr) + catch + end + + # Launchers report errcode as the job's exit status and typically truncate it to + # 8 bits, so MPI_Abort(comm, 256) is reported as *success*. Never do that for a + # failing test. + errcode = Cint(exitcode) + if mod(errcode, 256) == 0 + errcode = Cint(1) + end + + MPI.Abort(MPI.COMM_WORLD, errcode) +end diff --git a/test/runtests.jl b/test/runtests.jl index 1e170c014..b9654fdf6 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -91,9 +91,14 @@ testdir = @__DIR__ istest(f) = endswith(f, ".jl") && startswith(f, "test_") && !in(f, excludefiles) testfiles = sort(filter(istest, readdir(testdir))) +# Loaded into every test process so that a rank which dies from an uncaught +# exception calls `MPI_Abort`, instead of leaving the job deadlocked. See the +# comments in that file. +abort_on_error = joinpath(testdir, "abort_on_error.jl") + @testset "$f" for f in testfiles cmd(n=nprocs) = - addenv(`$(mpiexec()) -n $n $(Base.julia_cmd()) --startup-file=no $(joinpath(testdir, f))`, + addenv(`$(mpiexec()) -n $n $(Base.julia_cmd()) --startup-file=no -L $(abort_on_error) $(joinpath(testdir, f))`, # `JULIA_MPI_TEST_NUM_PROCESSES` is used in `test_gather.jl` to # test number of processes. "JULIA_MPI_TEST_NUM_PROCESSES"=>string(n)) @@ -112,6 +117,9 @@ testfiles = sort(filter(istest, readdir(testdir))) end end elseif f == "test_error.jl" + # This test fails on purpose. The nonzero exit status comes from the + # `MPI_Abort` in `abort_on_error.jl`, so an "application called + # MPI_Abort(MPI_COMM_WORLD, 1)" line in the log below is expected. r = run(ignorestatus(cmd())) @test !success(r) elseif f == "test_errorhandler.jl" && MPI.MPI_LIBRARY in ("unknown", "FujitsuMPI")