Call MPI_Abort when a test process fails

docs/src/external.md

@@ -23,7 +23,7 @@ ccall((:cfunc2, lib), Cint, (Ptr{MPI.MPI_Comm},), comm)
 
 ## Object finalizers and `MPI.Finalize`
 
-External libraries may allocate their own MPI handles (e.g., create or duplicate MPI communicators), which need to be cleaned up before MPI is finalized. If these are attached to [object finalizers](https://docs.julialang.org/en/v1/base/base/#Base.finalizer), they may not be guaranteed to be called before `MPI.Finalize`, which can result in an error upon program exit. (By default, MPI.jl will install an [`atexit`](https://docs.julialang.org/en/v1/base/base/#Base.atexit) hook that calls `MPI.Finalize` if it hasn't already been invoked.)
+External libraries may allocate their own MPI handles (e.g., create or duplicate MPI communicators), which need to be cleaned up before MPI is finalized. If these are attached to [object finalizers](https://docs.julialang.org/en/v1/base/base/#Base.finalizer), they may not be guaranteed to be called before `MPI.Finalize`, which can result in an error upon program exit. (By default, MPI.jl will install an [`atexit`](https://docs.julialang.org/en/v1/base/base/#Base.atexit) hook that calls `MPI.Finalize` if it hasn't already been invoked. Note that this hook does nothing if Julia is exiting with a nonzero exit code, since `MPI.Finalize` is collective: in that case MPI is still initialized when the finalizers run, so a finalizer that makes a collective call such as `MPI_Comm_free` can block indefinitely against ranks that are elsewhere.)
 
 There are two typical solutions to this problem:
 

docs/src/usage.md

@@ -22,7 +22,13 @@ println("Hello world, I am $(MPI.Comm_rank(comm)) of $(MPI.Comm_size(comm))")
 MPI.Barrier(comm)
 ```
 
-Calling [`MPI.Finalize()`](@ref) at the end of the program is optional, as it will be called automatically when Julia exits.
+Calling [`MPI.Finalize()`](@ref) at the end of the program is optional, as it will be
+called automatically when Julia exits *normally*, i.e. with exit code `0`. If Julia
+exits with a nonzero exit code, for example because of an uncaught exception, then
+`MPI.Finalize()` is deliberately not called: it is collective, and the other ranks are
+not finalizing. Those ranks will instead remain blocked in their next collective call.
+If you want a rank that fails to bring the whole job down promptly, call
+[`MPI.Abort`](@ref) yourself.
 
 The program can then be launched via an MPI launch command (typically `mpiexec`, `mpirun` or `srun`), e.g.
 ```

test/abort_on_error.jl

@@ -0,0 +1,44 @@
+# Loaded into every MPI test process via `julia -L abort_on_error.jl test_foo.jl`; see
+# the `cmd` definition in runtests.jl.  Deliberately not named `test_*.jl`, so that
+# `istest` in runtests.jl does not pick it up as a test file.
+#
+# When a test file dies from an uncaught exception (a failing `@test`, an `MPIError`,
+# ...), Julia prints it and calls `exit(1)`.  MPI.jl's own atexit hook then skips
+# `MPI.Finalize()`, because it is collective.  Two bad things follow:
+#
+#  * the other ranks stay blocked in their next collective, and
+#  * this rank's object finalizers still run, and `free(::Win)`, `free(::Comm)` and
+#    `close(::FileHandle)` are collective, so this rank blocks too -- a deadlock in
+#    which every process is still alive and which therefore no launcher can detect.
+#
+# atexit hooks run before object finalizers, so aborting here heads off both.
+using MPI
+
+atexit() do exitcode
+    # Only interfere with abnormal termination.
+    exitcode == 0 && return
+    # MPI must be up, and not already finalized: calling MPI_Abort after MPI_Finalize
+    # is not allowed.  (Several tests assert after their own MPI.Finalize() call.)
+    (MPI.Initialized() && !MPI.Finalized()) || return
+
+    # MPI_Abort terminates this process immediately: no later atexit hook, object
+    # finalizer or stream flush will run.  The `Test Failed / Expression: ...` detail
+    # block goes to stdout, which may be block-buffered under mpiexec, so get it out
+    # first rather than risk losing the only description of what went wrong.  A broken
+    # stream must not be able to skip the abort itself, hence the try/catch.
+    try
+        flush(stdout)
+        flush(stderr)
+    catch
+    end
+
+    # Launchers report errcode as the job's exit status and typically truncate it to
+    # 8 bits, so MPI_Abort(comm, 256) is reported as *success*.  Never do that for a
+    # failing test.
+    errcode = Cint(exitcode)
+    if mod(errcode, 256) == 0
+        errcode = Cint(1)
+    end
+
+    MPI.Abort(MPI.COMM_WORLD, errcode)
+end

test/runtests.jl

@@ -91,9 +91,14 @@ testdir = @__DIR__
 istest(f) = endswith(f, ".jl") && startswith(f, "test_") && !in(f, excludefiles)
 testfiles = sort(filter(istest, readdir(testdir)))
 
+# Loaded into every test process so that a rank which dies from an uncaught
+# exception calls `MPI_Abort`, instead of leaving the job deadlocked.  See the
+# comments in that file.
+abort_on_error = joinpath(testdir, "abort_on_error.jl")
+
 @testset "$f" for f in testfiles
     cmd(n=nprocs) =
-        addenv(`$(mpiexec()) -n $n $(Base.julia_cmd()) --startup-file=no $(joinpath(testdir, f))`,
+        addenv(`$(mpiexec()) -n $n $(Base.julia_cmd()) --startup-file=no -L $(abort_on_error) $(joinpath(testdir, f))`,
                # `JULIA_MPI_TEST_NUM_PROCESSES` is used in `test_gather.jl` to
                # test number of processes.
                "JULIA_MPI_TEST_NUM_PROCESSES"=>string(n))
@@ -112,6 +117,9 @@ testfiles = sort(filter(istest, readdir(testdir)))
             end
         end
     elseif f == "test_error.jl"
+        # This test fails on purpose.  The nonzero exit status comes from the
+        # `MPI_Abort` in `abort_on_error.jl`, so an "application called
+        # MPI_Abort(MPI_COMM_WORLD, 1)" line in the log below is expected.
         r = run(ignorestatus(cmd()))
         @test !success(r)
     elseif f == "test_errorhandler.jl" && MPI.MPI_LIBRARY in ("unknown", "FujitsuMPI")