julia 1.3 writing files with Threads.@threads can get stuck on 1 thread

[scottstanie]

Not sure if this is strictly a JLD.jl bug, but when I use jld write in a loop, it seems to ruin the ability of Threads.@threads to distribute work.

At first, the threading works (here I have set JULIA_NUM_THREADS=56 on a big machine):

julia> versioninfo()
Julia Version 1.3.0-alpha.0
Commit 6c11e7c2c4 (2019-07-23 01:46 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, broadwell)
Environment:
  JULIA_NUM_THREADS = 56

Threads.@threads for i = 1:10
    println("i = $i on thread $(Threads.threadid())")
end
i = 4 on thread 4
i = 10 on thread 10
i = 2 on thread 2
i = 1 on thread 1
i = 9 on thread 9
i = 7 on thread 7
i = 5 on thread 5
i = 6 on thread 6
i = 3 on thread 3
i = 8 on thread 8

Then I try to save arrays in parallel

function f1()
    function innerwrite(layer, i)
          jldopen("testwrite$i.jld", "w") do f
              write(f, "layer", layer)
              println("writing testwrite$i on thread $(Threads.threadid())")
          end
    end
    rr = rand(800, 800, 10);
    Threads.@threads for i = 1:size(rr, 3)
        layer = rr[:, :, i]
        innerwrite(layer, i)
    end
end

julia> f1()
writing testwrite1 on thread 1
writing testwrite2 on thread 1
writing testwrite3 on thread 1
writing testwrite4 on thread 1
writing testwrite5 on thread 1
writing testwrite6 on thread 1
writing testwrite7 on thread 1
writing testwrite8 on thread 1
writing testwrite9 on thread 1
writing testwrite10 on thread 1

and it gets stuck on 1 thread.

Now the weirder part: it breaks the original example:

julia> Threads.@threads for i = 1:10
     println("i = $i on thread $(Threads.threadid())")
 end
i = 1 on thread 1
i = 2 on thread 1
i = 3 on thread 1
i = 4 on thread 1
i = 5 on thread 1
i = 6 on thread 1
i = 7 on thread 1
i = 8 on thread 1
i = 9 on thread 1
i = 10 on thread 1

So not only does the JLD not save in parallel, it makes the former simple example from the blog post fail https://julialang.org/blog/2019/07/multithreading

I'll note that if I save the arrays to .txt files with just open("testwrite$i.txt", "w") do f, it doesn't break the threading.

[JeffBezanson]

I only get the stuck-on-thread-1 behavior after an error is thrown (we don't properly unwind the "threaded region" flag on an error, which will be fixed by #32477). Do you get it even with no error?
More importantly, HDF5 is not thread-safe. Apparently it can be built in a thread-safe mode, but from what I can tell from the docs it just puts a lock around the whole library. In fact when I run this example I get

double free or corruption (fasttop)double free or corruption (fasttop)double free or corruption (fasttop)



signal (11): Segmentation fault

signal (11): Segmentation fault
signal (11): Segmentation fault
1

signal (11): Segmentation fault

signal (11): Segmentation fault

[scottstanie]

I think you're right- I can't get it to stick anymore without another error, and it appears that can only happen when my example writes using JLD2, not JLD or HDF5 (both segfault).