Intermittent hangs with 10.16.1

[ofrobots]

Over at Google Cloud we are noticing intermittent hangs on some virtual machines with the latest 10.16.1. This is likely due to the OpenSSL upgrade due to 1.1.1c (see openssl/openssl#9078).

Hangs seem to coincide with when the system has low-entropy and has the following stack:

(gdb) bt
#0  0x00007fb4b2d7403f in __GI___select (nfds=18, readfds=0x7ffdd2bf5c30, writefds=0x0, exceptfds=0x0, timeout=0x0) at ../sysdeps/unix/sysv/linux/select.c:41
#1  0x00000000014d3c4c in rand_pool_acquire_entropy ()
#2  0x00000000014d2058 in rand_drbg_get_entropy ()
#3  0x00000000014cf5c4 in RAND_DRBG_instantiate ()
#4  0x00000000014cf8fe in drbg_setup ()
#5  0x00000000014cfbc7 in do_rand_drbg_init_ossl_ ()
#6  0x00007fb4b305d827 in __pthread_once_slow (once_control=0x2667838 <rand_drbg_init>, init_routine=0x14cfb80 <do_rand_drbg_init_ossl_>) at pthread_once.c:116
#7  0x00000000014febe9 in CRYPTO_THREAD_run_once ()
#8  0x00000000014ceb66 in drbg_status ()
#9  0x0000000000a104a0 in node::crypto::EntropySource(unsigned char*, unsigned long) ()
#10 0x00000000017aa753 in v8::base::RandomNumberGenerator::RandomNumberGenerator() ()
#11 0x00000000017a9901 in v8::base::CallOnceImpl(long*, std::function<void ()>) ()
#12 0x00000000017ac19f in v8::base::OS::GetRandomMmapAddr() ()
#13 0x0000000000af7947 in v8::internal::GetRandomMmapAddr() ()
#14 0x0000000000f037af in v8::internal::Heap::SetUp() ()
#15 0x0000000000fb6713 in v8::internal::Isolate::Init(v8::internal::StartupDeserializer*) ()
#16 0x00000000011ffd3a in v8::internal::Snapshot::Initialize(v8::internal::Isolate*) ()
#17 0x0000000000b14988 in v8::Isolate::Initialize(v8::Isolate*, v8::Isolate::CreateParams const&) ()
#18 0x0000000000b149da in v8::Isolate::New(v8::Isolate::CreateParams const&) ()
#19 0x0000000000902954 in node::Start(int, char**) ()
#20 0x00007fb4b2c7eb97 in __libc_start_main (main=0x8bbd50 <main>, argc=2, argv=0x7ffdd2bf6788, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffdd2bf6778) at ../csu/libc-start.c:310
#21 0x00000000008bbe85 in _start ()

[ofrobots]

Here's my experiment on some GCE virtual machines:

for i in $(seq 1 50) ; do echo run $i; cat /proc/sys/kernel/random/entropy_avail ; time npm >/dev/null; sleep 1; done
...
172

real	0m0.257s
user	0m0.222s
sys	0m0.032s
run 6
173

real	0m0.264s
user	0m0.216s
sys	0m0.044s
run 7
5
^C
real	33m50.490s
...

Note the entropy value of 5 at the time of the hang.

[addaleax]

I think this is openssl/openssl#9078 / openssl/openssl#9084, and cherry-picking the latter would resolve this?

[ofrobots]

I believe so. I will try this out in the failing environment tomorrow.

[targos]

I'm pretty sure I already saw an issue here about this a few days ago.

Edit: forget it, it was before 10.16.1

[MylesBorins]

Should we escalate a 10.16.2 to fix this? @ofrobots is there an easy way to test if I can get an rc together?

[MylesBorins]

/cc @nodejs/lts

[ofrobots]

#28983