JVM fails to load in 1.1 (JavaCall.jl)

[aviks]

In 1.1, trying to initialise the java virtual machine in Julia via JavaCall.jl causes a stack overflow error. It works correctly when the code is loaded with -i or -e but fails when used from the REPL.

ERROR: StackOverflowError:
Stacktrace:
 [1] init(::Array{String,1}) at /Users/rene/.julia/packages/JavaCall/toamy/src/jvm.jl:176
 [2] top-level scope at none:0

Ref: JuliaInterop/JavaCall.jl#96

The call in question is the JNI_CreateJavaVM function in the JVM. More information about the JVM invocation API is at https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/invocation.html

This is a regression from 1.0.x

[aviks]

@JeffBezanson suggested trying with ALWAYS_COPY_STACKS. However defining this in options.h and rebuilding julia did not change the failure.

[aviks]

Maybe related? JuliaInterop/RCall.jl#293

[JeffBezanson]

I get a hang instead of a stack overflow:

julia> using JavaCall

julia> JavaCall.init(["-Xmx128M"])

stack trace:

#0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
#1  0x00007ff55d7204e2 in os::PlatformEvent::park(long) () from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#2  0x00007ff55d70bd93 in ObjectMonitor::EnterI(Thread*) () from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#3  0x00007ff55d70c143 in ObjectMonitor::enter(Thread*) () from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#4  0x00007ff55d4fbf1a in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#5  0x00007ff55d4fc571 in instanceKlass::initialize(Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#6  0x00007ff55d4fc30a in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#7  0x00007ff55d4fc571 in instanceKlass::initialize(Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#8  0x00007ff55d4fc30a in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#9  0x00007ff55d4fc571 in instanceKlass::initialize(Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#10 0x00007ff55d4fc30a in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#11 0x00007ff55d4fc571 in instanceKlass::initialize(Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#12 0x00007ff55d4fc30a in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#13 0x00007ff55d4fc571 in instanceKlass::initialize(Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#14 0x00007ff55d4589de in Exceptions::new_exception(Thread*, Symbol*, Symbol*, JavaCallArguments*, Handle, Handle, Handle)
    () from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#15 0x00007ff55d458fc2 in Exceptions::new_exception(Thread*, Symbol*, char const*, Handle, Handle, Handle, Exceptions::ExceptionMsgToUtf8Mode) () from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#16 0x00007ff55d4594aa in Exceptions::_throw_msg(Thread*, char const*, int, Symbol*, char const*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#17 0x00007ff55d4fabf3 in instanceKlass::set_initialization_state_and_notify(instanceKlass::ClassState, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#18 0x00007ff55d4fc05c in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#19 0x00007ff55d4fc571 in instanceKlass::initialize(Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#20 0x00007ff55d4fc30a in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#21 0x00007ff55d4fc571 in instanceKlass::initialize(Thread*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
---Type <return> to continue, or q <return> to quit---
#22 0x00007ff55d8532e6 in Threads::create_vm(JavaVMInitArgs*, bool*) ()
   from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so
#23 0x00007ff55d55c624 in JNI_CreateJavaVM () from /usr/lib/jvm/default-java/jre/lib/amd64/server/libjvm.so

It looks like the JVM is trying to throw an exception. I'll see if I can find the error message.

[Keno]

Jeff and I spent a lot of time looking into this today. The story is basically as follows:

1. The JVM uses the stack extents to determine whether a held lock is owned by the current thread (if it's on the current thread stack, it is presumed to be owned).
2. If this determination is wrong, things break in all sorts of ways (it tried to throw an exception before the system is initialized causing stack overflows or hangs, etc.)
3. In order to find out the stack bounds, the JVM queries pthread_attr_getstack for the current stack extents.
4. There is no officially sanctioned API to change this value.

The reason that ALWAYS_COPY_STACKS does not work is because that doesn't actually run on the main stack, it just only uses one stack, which is nice, but insufficient.

Jeff is trying to get ALWAYS_COPY_STACKS to go back to using the actual process stack, which may make this work temporarily, but certainly not on the pre-built binaries.

A full fix will require an enhancement to the JNI to allow setting the stack extents of the current stack before entering the JVM thread.

[Keno]

(Alternatively, we could have a way of always executing a ccall on the current thread's main stack).

[Keno]

This is the relevant function in the current jdk, though it looks a bit different in JDK8: https://github.com/unofficial-openjdk/openjdk/blob/3e601613f8c323f1129fa3f8aac389b90456c8c7/src/hotspot/os_cpu/linux_zero/os_linux_zero.cpp#L308

[aviks]

Alternatively, we could have a way of always executing a ccall on the current thread's main stack

That would be much better. Honestly, not many people will use this if it needs recompiling Julia.

[JeffBezanson]

We could possibly enable this with a command line option too.

[aviks]

That'd be great, thanks!

[JeffBezanson]

We've decided that the right way to address this is to add a new calling convention to ccall that performs calls on the original thread stack. That can be used whenever you need to call native code that depends on the stack pointer value like the JVM does. That puts the information in the right place, since it's specific to which C library/function you're calling, and doesn't require any options or configuration. The main limitation is that the native code won't be able to safely call back into julia.

Some other options considered were (1) provide builds of openjdk (via BinaryBuilder) patched to work around the problem (we could still potentially do this if it's practical), and (2) (long term) try to get the JNI to add a call to inform it that the stack has switched.

For 1.2, the most we can do is fix the ALWAYS_COPY_STACKS build option.

[ViralBShah]

I think patched builds of openjdk are not practical for the users of this feature. Most likely the end users of java software are enterprises who are unlikely to want to pick an openjdk we provide.

Providing a patch to openjdk itself may be ok in hopes of future releases fixing this issue.