segmentation faults when combined with @threads with memory allocation

[twhitehead]

One of our users was having problems with their hybrid MPI/threaded Julia code segment faulting on our clusters.

OS: Linux (CentOS 7)
Julia: 1.3.0
OpenMPI: 3.1.2

I simplified their code down the following demo

using MPI

function main()
    MPI.Init()

    Threads.@threads for i in 1:100
        A = rand(1000,1000)
        A1 = inv(A)
        oops = A1[1.6]
    end

    MPI.Finalize()
end

main()

• exceptions sometimes turn into segmentation faults inside of @threads for loops
• for a reliable segmentation fault you need to perform a reasonable amount of work in the loop

$ export JULIA_NUM_THREADS=2
$ mpirun -n 2 example.jl

[1578695090.632505] [gra797:13151:0]         parser.c:1369 UCX  WARN  unused env variable: UCX_MEM_MMAP_RELOC (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[1578695091.181319] [gra797:13152:0]         parser.c:1369 UCX  WARN  unused env variable: UCX_MEM_MMAP_RELOC (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
[gra797:13151:1:13155] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2b6ccc961008)
[gra797:13152:1:13156] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2b5877538008)
==== backtrace ====
 0 0x0000000000010e90 __funlockfile()  ???:0
 1 0x00000000000a9904 maybe_collect()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_threads.h:283
 2 0x00000000000a9904 jl_gc_managed_malloc()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gc.c:3116
 3 0x000000000007a160 _new_array_()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:109
 4 0x000000000007db3e jl_array_copy()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:1135
 5 0x000000000005d22c _jl_invoke()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gf.c:2135
 6 0x0000000000078e19 jl_apply()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia.h:1631
===================
==== backtrace ====
 0 0x0000000000010e90 __funlockfile()  ???:0
 1 0x00000000000a8d74 maybe_collect()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_threads.h:283
 2 0x00000000000a8d74 jl_gc_pool_alloc()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gc.c:1096
 3 0x000000000005d22c _jl_invoke()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gf.c:2135
 4 0x0000000000078e19 jl_apply()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia.h:1631
===================
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 13151 on node gra797 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

Here is some possibly relevant info from ompi_info as well

...
  Configure command line: '--prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/gcc7.3/openmpi/3.1.2'
                          '--build=x86_64-pc-linux-gnu'
                          '--host=x86_64-pc-linux-gnu' '--enable-shared'
                          '--with-verbs' '--enable-mpirun-prefix-by-default'
                          '--with-hwloc=external' '--without-usnic'
                          '--with-ucx' '--disable-wrapper-runpath'
                          '--disable-wrapper-rpath' '--with-munge'
                          '--with-slurm' '--with-pmi=/opt/software/slurm'
                          '--enable-mpi-cxx' '--with-hcoll'
                          '--disable-show-load-errors-by-default'
                          '--enable-mca-dso=common-libfabric,common-ofi,common-verbs,atomic-mxm,btl-openib,btl-scif,coll-fca,coll-hcoll,ess-tm,fs-lustre,mtl-mxm,mtl-ofi,mtl-psm,mtl-psm2,osc-ucx,oob-ud,plm-tm,pmix-s1,pmix-s2,pml-ucx,pml-yalla,pnet-opa,psec-munge,ras-tm,rml-ofi,scoll-mca,sec-munge,spml-ikrit,'
...
          C++ exceptions: no
          Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,
                          OMPI progress: no, ORTE progress: yes, Event lib:
                          yes)
...

EDIT: Removed exception bit as as noted below it isn't required.

[simonbyrne]

What version of MPI.jl are you using? You can get it via julia -e 'using Pkg; Pkg.status()'

[simonbyrne]

Hmm, I can reproduce on our cluster (with openmpi 3.1.4 and 4.0.1). It might have something to do with openmpi/UCX's funny malloc hooks, which have caused problems before. cc'ing @kpamnany and @vchuravy who might have some ideas?

mpich doesn't seem to have this problem, so I would suggest using that in the meantime?

[twhitehead]

Noticed after I wrote this that the exception part is not critical.

The program simply has to spend enough time (or maybe do enough allocating?) inside the @thread for loop and you get a crash (e.g., I get reliable crashes for 1000x1000 matricies but when I reduce it to 10x10 it rarely crashes).

using MPI

function main()
    MPI.Init()

    Threads.@threads for i in 1:2
        A = rand(1000,1000)
        A1 = inv(A)
    end

    MPI.Finalize()
end

main()

I'll adjust the title accordingly.

[simonbyrne]

It's something to do with how MPI and Julia malloc interact. The simplest example (which doesn't require this package) that I can reproduce is:

# threads.jl
function main()
    ccall((:MPI_Init, :libmpi), Nothing, (Ptr{Cint},Ptr{Cint}), C_NULL, C_NULL)
    Threads.@threads for i in 1:100
        A = rand(1000,1000)
    end
end

main()

then calling

JULIA_NUM_THREADS=2 mpiexec -n 2 julia threads.jl

[simonbyrne]

My guess is that it's due to UCX, which has caused similar problems in the past (#298).

They've fixed quite a few issues on their main branch, but haven't made a new release for quite a while (I think they've made some breaking API changes, so need to coordinate with openmpi). Hopefully this is one of those that will get fixed.

[simonbyrne]

I tried this on master build of UCX + openmpi and see the same issue.

[kpamnany]

Try calling MPI_Init_thread with MPI_THREAD_MULTIPLE instead of MPI_Init? Just to rule that out.

[simonbyrne]

Yes, that seems to give the same result.

function main()
    r = Ref{Cint}()
    ccall((:MPI_Init_thread, :libmpi), Cint,
          (Ptr{Cint},Ptr{Cvoid},Cint,Ptr{Cint}),
          C_NULL, C_NULL, Cint(3), r)
    @show r[]
    Threads.@threads for i in 1:100
        A = rand(1000,1000)
    end
end

(MPI_THREAD_MULTIPLE == 3)

[twhitehead]

It still blows up for me with OpenMPI 3.1.2 when I force just using tcp and self (this is after allocating 2 tasks with 2 cores each and setting JULIA_NUM_THREADS=2)

$ mpirun --mca pml ob1 --mca btl tcp,self julia example.jl

[gra797:13477:0:13484] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2b984f48b008)
[gra797:13478:0:13485] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2af4b14e9008)
==== backtrace ====
 0 0x0000000000010e90 __funlockfile()  ???:0
 1 0x00000000000a8cf4 maybe_collect()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_threads.h:283
 2 0x00000000000a8cf4 jl_gc_big_alloc()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gc.c:837
 3 0x00000000000a9300 jl_gc_alloc_()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_internal.h:238
 4 0x00000000000a9300 jl_gc_alloc_()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_internal.h:240
 5 0x00000000000a9300 jl_gc_alloc()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gc.c:2940
 6 0x000000000007ac7e _new_array_()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:100
 7 0x000000000007ac7e _new_array()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:163
 8 0x000000000007ac7e jl_alloc_array_1d()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:424
 9 0x000000000005d22c _jl_invoke()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gf.c:2135
10 0x0000000000078e19 jl_apply()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia.h:1631
===================
==== backtrace ====
 0 0x0000000000010e90 __funlockfile()  ???:0
 1 0x00000000000a8cf4 maybe_collect()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_threads.h:283
 2 0x00000000000a8cf4 jl_gc_big_alloc()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gc.c:837
 3 0x00000000000a9300 jl_gc_alloc_()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_internal.h:238
 4 0x00000000000a9300 jl_gc_alloc_()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia_internal.h:240
 5 0x00000000000a9300 jl_gc_alloc()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gc.c:2940
 6 0x000000000007ac7e _new_array_()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:100
 7 0x000000000007ac7e _new_array()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:163
 8 0x000000000007ac7e jl_alloc_array_1d()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/array.c:424
 9 0x000000000005d22c _jl_invoke()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/gf.c:2135
10 0x0000000000078e19 jl_apply()  /dev/shm/ebuser/avx2/Julia/1.3.0/gmkl-2018.3/julia-1.3.0/src/julia.h:1631
===================
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 13477 on node gra797 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

[eschnett]

A few generic comments (that might not apply here, but nevertheless):

• You call to MPI_init_thread requests MPI_THREAD_MULTIPLE, but does not check whether this is actually supported by the installed library. You need to check the return value.
• It is also possible that there is some confusion between the MPI headers that are found by Julia, the shared MPI library that is used for linking, the mpiexec that you are using at run time, and the MPI library that is actually used at run time.
• Finally, do you see the segfaults at run time, or during shutdown? You could add a call to MPI_Barrier when your code is done to find out.

[simonbyrne]

* You call to `MPI_init_thread` requests `MPI_THREAD_MULTIPLE`, but does not check whether this is actually supported by the installed library. You need to check the return value.

Yes, I printed the result: it returns the same value.

* It is also possible that there is some confusion between the MPI headers that are found by Julia, the shared MPI library that is used for linking, the `mpiexec` that you are using at run time, and the MPI library that is actually used at run time.

Yes, I tried it with a specified path for libmpi and mpiexec

* Finally, do you see the segfaults at run time, or during shutdown? You could add a call to `MPI_Barrier` when your code is done to find out.

They are at runtime: I with one rank and printing between the loop and finalize, I get

├ cat threads.jl
const libmpi = expanduser("~/usr/lib/libmpi.so")

function main()
    r = Ref{Cint}()
    ccall((:MPI_Init_thread, libmpi), Cint,
          (Ptr{Cint},Ptr{Cvoid},Cint,Ptr{Cint}),
          C_NULL, C_NULL, Cint(3), r)
    @show r[]
    Threads.@threads for i in 1:100
        A = rand(1000,1000)
    end
    println("done")
    ccall((:MPI_Finalize, libmpi), Cint,
          (),)
end

main()

├ JULIA_NUM_THREADS=2 ~/usr/bin/mpirun -n 1 julia threads.jl
r[] = 3
[hpc-91-09:117009:0:117057] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x2aaaaaaf3008)
==== backtrace (tid: 117057) ====
 0  /home/spjbyrne/usr/lib/libucs.so.0(ucs_handle_error+0x19c) [0x2aaaed9253dc]
 1  /home/spjbyrne/usr/lib/libucs.so.0(+0x2570c) [0x2aaaed92570c]
 2  /home/spjbyrne/usr/lib/libucs.so.0(+0x2597b) [0x2aaaed92597b]
 3  /central/software/julia/1.3.0/bin/../lib/libjulia.so.1(jl_gc_managed_malloc+0x74) [0x2aaaaad788c4]
 4  /central/software/julia/1.3.0/bin/../lib/libjulia.so.1(jl_alloc_array_2d+0x24c) [0x2aaaaad4a08c]
 5  [0x2aaac5335404]
 6  [0x2aaac5335609]
 7  [0x2aaac5334aff]
 8  [0x2aaac5334b1d]
 9  /central/software/julia/1.3.0/bin/../lib/libjulia.so.1(jl_apply_generic+0x53c) [0x2aaaaad2c79c]
10  /central/software/julia/1.3.0/bin/../lib/libjulia.so.1(+0x78e79) [0x2aaaaad47e79]
=================================

signal (11): Segmentation fault
in expression starting at /central/home/spjbyrne/src/MPI.jl/threads.jl:17
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 117009 on node hpc-91-09 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

[simonbyrne]

One way forward would be to see if we can recreate the behavior in C: I guess this would consist of writing a multithreaded C program that internally calls malloc. Unfortunately this is beyond my C skills, so if someone else wanted to take a stab at it I would be grateful.