JRuby crash after update 1.5.0 -> 1.5.3

[perlei]

Hi, I just tried updating my logstash from 1.5.0 to 1.5.3, but are getting a crash when I try to start logstash. This is what i see

LoadError: load error: jopenssl/load -- java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    org/jruby/ext/openssl/X509Cert.sign(Lorg/jruby/runtime/ThreadContext;Lorg/jruby/runtime/builtin/IRubyObject;Lorg/jruby/runtime/builtin/IRubyObject;)Lorg/jruby/runtime/builtin/IRubyObject; @144: invo                                   kevirtual
  Reason:
    Type 'org/bouncycastle/asn1/ASN1ObjectIdentifier' (current frame, stack[1]) is not assignable to 'org/bouncycastle/asn1/DERObjectIdentifier'
  Current Frame:
    bci: @144
    flags: { }
    locals: { 'org/jruby/ext/openssl/X509Cert', 'org/jruby/runtime/ThreadContext', 'org/jruby/runtime/builtin/IRubyObject', 'org/jruby/runtime/builtin/IRubyObject', 'org/jruby/Ruby', 'java/lang/String',                                    'java/lang/String', 'java/lang/String', 'org/bouncycastle/x509/X509V3CertificateGenerator', 'java/util/Iterator', 'org/jruby/ext/openssl/X509Extension', '[B' }
    stack: { 'org/bouncycastle/x509/X509V3CertificateGenerator', 'org/bouncycastle/asn1/ASN1ObjectIdentifier', integer, '[B' }
  Bytecode:
    0x0000000: 2bb4 001c 3a04 2cc0 00c0 b600 d13a 052d
    0x0000010: c000 d2b6 00d3 3a06 2dc0 00d2 b600 d4b6
    0x0000020: 00a0 3a07 1290 1905 b600 ca99 000d 12d5
    0x0000030: 1906 b600 ca9a 0017 1284 1905 b600 ca99
    0x0000040: 0015 12d6 1907 b600 8599 000b 1904 12d7
    0x0000050: b800 32bf 2ab7 00d8 3a08 2ab7 00d9 b900
    0x0000060: da01 003a 0919 09b9 004b 0100 9900 3719
    0x0000070: 09b9 004c 0100 c000 db3a 0a19 0ab6 00dc
    0x0000080: 3a0b 1908 190a b600 dd19 0ab6 00de 190b
    0x0000090: b600 dfa7 000d 3a0b 1904 190b b600 61bf
    0x00000a0: a7ff c519 08bb 0063 59b7 00c1 1906 b600
    0x00000b0: 6612 e0b6 0066 1905 b600 66b6 00c5 b600
    0x00000c0: e12a 1908 2cc0 00c0 b600 e2b6 00e3 b500
    0x00000d0: 15a7 000d 3a09 1904 1909 b800 30bf 2ab4
    0x00000e0: 0015 c700 0d19 0401 c000 31b8 0032 bf2a
    0x00000f0: b400 15b6 0044 b800 e53a 0919 09c7 000c
    0x0000100: 2ab4 0015 b600 443a 092a 1904 1909 b600
    0x0000110: 35b5 0048 2a03 b500 132a b0
  Exception Handler Table:
    bci [123, 147] => handler: 150
    bci [193, 209] => handler: 212
  Stackmap Table:
    full_frame(@56,{Object[#379],Object[#380],Object[#452],Object[#452],Object[#382],Object[#384],Object[#384],Object[#384]},{})
    same_frame(@76)
    same_frame(@84)
    append_frame(@101,Object[#485],Object[#386])
    full_frame(@150,{Object[#379],Object[#380],Object[#452],Object[#452],Object[#382],Object[#384],Object[#384],Object[#384],Object[#485],Object[#386],Object[#486]},{Object[#396]})
    chop_frame(@160,1)
    chop_frame(@163,1)
    same_locals_1_stack_item_frame(@212,Object[#487])
    same_frame(@222)
    same_frame(@239)
    append_frame(@265,Object[#384])

  require at org/jruby/RubyKernel.java:1072
   (root) at /opt/contrib/logstash-1.5.3/vendor/jruby/lib/ruby/shared/openssl.rb:1
  require at org/jruby/RubyKernel.java:1072
   (root) at /opt/contrib/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/patches/stronger_openssl_defaults.rb:1
  require at org/jruby/RubyKernel.java:1072
   (root) at /opt/contrib/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/patches/stronger_openssl_defaults.rb:2
  require at org/jruby/RubyKernel.java:1072
   (root) at /opt/contrib/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/patches.rb:1
  require at org/jruby/RubyKernel.java:1072
   (root) at /opt/contrib/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/patches.rb:4
  require at org/jruby/RubyKernel.java:1072
   (root) at /opt/contrib/logstash-1.5.3/lib/bootstrap/environment.rb:47

I think the problem is related to bouncycastle versions, and that JRuby does not support newer versions of this library. In my Ubuntu 14.04 enviroment i have other applications installed that also uses bouncycastle but of more recent version. I can find bouncycastle jars in my java installation folder at:

/usr/lib/jvm/jre1.8.0_45/lib/ext/bcpkix-jdk15on-151.jar
/usr/lib/jvm/jre1.8.0_45/lib/ext/bcprov-jdk15on-151.jar

and i can also see the bundled bouncycastle jars inside the logstash directory

./vendor/jruby/lib/ruby/shared/org/bouncycastle/bcpkix-jdk15on/1.50/bcpkix-jdk15on-1.50.jar
./vendor/jruby/lib/ruby/shared/org/bouncycastle/bcprov-jdk15on/1.50/bcprov-jdk15on-1.50.jar

So what i think is happening is that logstash/jruby is preferring the bouncycastle libraries in the classpath, but these are incompatible with the jruby version.

Could this be the case? Anyway to get around this problem?

[kotulag]

Did you find a work-around? I am experiencing the same jruby crash with logstash 1.5.3 on CentOS 6.6, openjdk 1.7.0_75.

[perlei]

I have not found a workaround. I'm still stuck at logstash 1.5.0 =(

[jsvd]

A fix for this seems to have entered jruby master 9 days ago

[cheister]

Have you tried updating your jruby-openssl gem to a version greater than 0.9.9 or using a more recent version of JRuby which will include the updated jruby-openssl gem?

[suyograo]

This has not been an issue anymore. Closing it.

[israr75]

Hi I have just installed v1.5.4 on redhat and I am seeing this exact issue. Can anyone please help?

[israr75]

FIX: remove all instances of /usr/java/jre1.7.0.79/lib/ext/bcprov-jdk150n-151.jar from the machine or move it say /tmp and then try running logstash