-
-
Notifications
You must be signed in to change notification settings - Fork 32.2k
Closed
Labels
v8 engineIssues and PRs related to the V8 dependency.Issues and PRs related to the V8 dependency.
Description
- Version:
root@*snip*:~# node -v
v10.15.2
root@*snip*:~# dpkg -l nodejs
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-==============-============-==================================================
ii nodejs 10.15.2~dfsg-2 amd64 evented I/O for V8 javascript - runtime executable
- Platform:
Debian Buster on amd64.
root@*snip*:~# uname -a
Linux *snip* 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux
- Subsystem:
I'm seeing a more or less regular crash of a non-trivial application within libnode.so.64. It's running on a low-traffic staging system. Find below an excerpt from the syslog of the affected machine. Times are in UTC.
Jan 23 16:45:12 *snip* kernel: [502165.519232] node[21284]: segfault at 8 ip 00007feca7142b6b sp 00007feca1bcb110 error 4 in libnode.so.64[7feca6d02000+abe000]
Jan 23 16:45:12 *snip* kernel: [502165.531802] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 17:13:28 *snip* kernel: [503861.549372] node[21318]: segfault at 8 ip 00007f1bd3ed8b6b sp 00007f1bcb7fd110 error 4 in libnode.so.64[7f1bd3a98000+abe000]
Jan 23 17:13:28 *snip* kernel: [503861.565373] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 18:09:58 *snip* kernel: [507250.954363] node[21363]: segfault at 8 ip 00007f9b01d95b6b sp 00007f9afd820110 error 4 in libnode.so.64[7f9b01955000+abe000]
Jan 23 18:09:58 *snip* kernel: [507250.962099] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 18:38:34 *snip* kernel: [508967.449433] node[21430]: segfault at 8 ip 00007fc0ee66cb6b sp 00007fc0e90f5110 error 4 in libnode.so.64[7fc0ee22c000+abe000]
Jan 23 18:38:34 *snip* kernel: [508967.461002] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 19:07:10 *snip* kernel: [510683.485209] node[21480]: segfault at 8 ip 00007f7527a1db6b sp 00007f7522ca7110 error 4 in libnode.so.64[7f75275dd000+abe000]
Jan 23 19:07:10 *snip* kernel: [510683.493478] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 20:04:52 *snip* kernel: [514145.480506] node[26613]: segfault at 8 ip 00007fe9488dcb6b sp 00007fe93fffe110 error 4 in libnode.so.64[7fe94849c000+abe000]
Jan 23 20:04:52 *snip* kernel: [514145.501037] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
After becoming aware of the issue I made sure that the process could dump core and I also installed the relevant debug symbols. My understanding is that node crashes within v8's JIT (?) compiler:
Core was generated by `/usr/bin/node dist/server.js'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x221fc90, b=0x0) at ../deps/v8/src/compiler/node.h:267
267 Type type() const { return type_; }
[Current thread is 1 (Thread 0x7fe93ffff700 (LWP 26613))]
(gdb) list
262
263 // Only NodeProperties should manipulate the op.
264 void set_op(const Operator* op) { op_ = op; }
265
266 // Only NodeProperties should manipulate the type.
267 Type type() const { return type_; }
268 void set_type(Type type) { type_ = type; }
269
270 // Only NodeMarkers should manipulate the marks on nodes.
271 Mark mark() const { return mark_; }
(gdb) bt
#0 v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x221fc90, b=0x0) at ../deps/v8/src/compiler/node.h:267
#1 0x00007fe9488dcc3c in v8::internal::compiler::(anonymous namespace)::MayAlias (b=0x1f2e570, a=0x221fc90) at ../deps/v8/src/utils.h:355
#2 v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x221fc90, b=0x1f2e570) at ../deps/v8/src/compiler/load-elimination.cc:37
#3 0x00007fe9488dde0d in v8::internal::compiler::LoadElimination::AliasStateInfo::MayAlias (this=0x7fe93fffe230, other=0x1f2e570) at ../deps/v8/src/compiler/load-elimination.cc:679
#4 0x00007fe9488dea2d in v8::internal::compiler::LoadElimination::AbstractField::Kill (this=this@entry=0x7fe9340792e0, alias_info=..., name=name@entry=..., zone=zone@entry=0x1ea7490)
at ../deps/v8/src/compiler/load-elimination.cc:372
#5 0x00007fe9488deb7c in v8::internal::compiler::LoadElimination::AbstractState::KillFields (this=this@entry=0x7fe93407a250, object=object@entry=0x221fc90, name=..., zone=0x1ea7490)
at ../deps/v8/src/compiler/load-elimination.cc:653
#6 0x00007fe9488e04f6 in v8::internal::compiler::LoadElimination::ReduceStoreField (this=0x7fe93fffea20, node=0x219eea0) at ../deps/v8/src/zone/zone-allocator.h:37
#7 0x00007fe94891e72b in v8::internal::compiler::(anonymous namespace)::SourcePositionWrapper::Reduce (this=0x229ba18, node=0x219eea0) at ../deps/v8/src/compiler/pipeline.cc:651
#8 0x00007fe94885adc8 in v8::internal::compiler::GraphReducer::Reduce (this=this@entry=0x7fe93fffe920, node=node@entry=0x219eea0) at ../deps/v8/src/compiler/graph-reducer.cc:85
#9 0x00007fe94885b195 in v8::internal::compiler::GraphReducer::ReduceTop (this=this@entry=0x7fe93fffe920) at ../deps/v8/src/compiler/graph-reducer.cc:152
#10 0x00007fe94885b388 in v8::internal::compiler::GraphReducer::ReduceNode (this=this@entry=0x7fe93fffe920, node=<optimized out>) at ../deps/v8/src/compiler/graph-reducer.cc:56
#11 0x00007fe94885b48d in v8::internal::compiler::GraphReducer::ReduceGraph (this=this@entry=0x7fe93fffe920) at ../deps/v8/src/compiler/graph.h:79
#12 0x00007fe94892b19e in v8::internal::compiler::LoadEliminationPhase::Run (this=this@entry=0x7fe93fffeca0, data=0x1fd3f98, temp_zone=temp_zone@entry=0x1ea7490) at ../deps/v8/src/compiler/pipeline.cc:1457
#13 0x00007fe94892c6b3 in v8::internal::compiler::PipelineImpl::Run<v8::internal::compiler::LoadEliminationPhase> (this=0x1fd4118) at ../deps/v8/src/compiler/zone-stats.h:29
#14 v8::internal::compiler::PipelineImpl::OptimizeGraph (this=this@entry=0x1fd4118, linkage=0x220c510) at ../deps/v8/src/compiler/pipeline.cc:1912
#15 0x00007fe94892c710 in v8::internal::compiler::PipelineCompilationJob::ExecuteJobImpl (this=0x1fd3de0) at ../deps/v8/src/compiler/pipeline.cc:847
#16 0x00007fe9487fcf91 in v8::internal::OptimizedCompilationJob::ExecuteJob (this=this@entry=0x1fd3de0) at ../deps/v8/src/compiler.cc:223
#17 0x00007fe9487f7fab in v8::internal::OptimizingCompileDispatcher::CompileNext (this=this@entry=0x1e8ae70, job=0x1fd3de0) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:118
#18 0x00007fe9487f826d in v8::internal::OptimizingCompileDispatcher::CompileTask::RunInternal (this=0x1eb8a90) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:69
#19 0x00007fe9485b1162 in node::BackgroundRunner (data=0x1e1f480) at ../src/node_platform.cc:42
#20 0x00007fe945563fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#21 0x00007fe9454944cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
I have the core dump on record and can provide additional information from the core file on request.
The application is launched using systemd. Find below the unit file for your reference:
Unit File (click to expand)
root@*snip*:~# systemctl cat *snip*
# /etc/systemd/system/*snip*.service
#
# Ansible managed
#
[Unit]
Description=*snip*
Requires=mysql.service
After=mysql.service
Wants=*snip*
[Service]
Restart=always
User=*snip*
WorkingDirectory=/srv/*snip*/
ExecStart=/usr/bin/node dist/server.js
Environment=*snip*
Environment=*snip*
Environment=*snip*
Environment=*snip*
Environment=*snip*
EnvironmentFile=/etc/*snip*/environment
StateDirectory=*snip*
StateDirectoryMode=0700
CacheDirectory=*snip*
CacheDirectoryMode=0700
ReadWritePaths=*snip*
ReadWritePaths=/tmp/
# PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io
NoNewPrivileges=true
LimitCORE=infinity
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/*snip*.service.d/override.conf
[Service]
Environment=DEBUG=*snip*:*
ReadWritePaths=/srv/*snip*/storage/
Find below a list of open files from the process after it was restarted automatically by systemd:
lsof -p $pid (click to expand)
root@*snip*:~# lsof -p 9946
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 9946 *snip* cwd DIR 8,1 4096 149468 /usr/share/*snip*
node 9946 *snip* rtd DIR 8,1 4096 2 /
node 9946 *snip* txt REG 8,1 14264 22337 /usr/bin/node
node 9946 *snip* mem REG 8,1 93000 955 /usr/lib/x86_64-linux-gnu/libresolv-2.28.so
node 9946 *snip* mem REG 8,1 26960 946 /usr/lib/x86_64-linux-gnu/libnss_dns-2.28.so
node 9946 *snip* mem REG 8,1 55792 947 /usr/lib/x86_64-linux-gnu/libnss_files-2.28.so
node 9946 *snip* mem REG 8,1 89016 944 /usr/lib/x86_64-linux-gnu/libnsl-2.28.so
node 9946 *snip* mem REG 8,1 1824496 937 /usr/lib/x86_64-linux-gnu/libc-2.28.so
node 9946 *snip* mem REG 8,1 146968 954 /usr/lib/x86_64-linux-gnu/libpthread-2.28.so
node 9946 *snip* mem REG 8,1 100712 912 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
node 9946 *snip* mem REG 8,1 1579448 941 /usr/lib/x86_64-linux-gnu/libm-2.28.so
node 9946 *snip* mem REG 8,1 1570256 3681 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
node 9946 *snip* mem REG 8,1 35808 956 /usr/lib/x86_64-linux-gnu/librt-2.28.so
node 9946 *snip* mem REG 8,1 14592 940 /usr/lib/x86_64-linux-gnu/libdl-2.28.so
node 9946 *snip* mem REG 8,1 27190832 21975 /usr/lib/x86_64-linux-gnu/libicudata.so.63.1
node 9946 *snip* mem REG 8,1 1886216 21980 /usr/lib/x86_64-linux-gnu/libicuuc.so.63.1
node 9946 *snip* mem REG 8,1 2984248 21976 /usr/lib/x86_64-linux-gnu/libicui18n.so.63.1
node 9946 *snip* mem REG 8,1 593696 4776 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
node 9946 *snip* mem REG 8,1 3031904 4767 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
node 9946 *snip* mem REG 8,1 158128 14137 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.17.1
node 9946 *snip* mem REG 8,1 71544 22287 /usr/lib/x86_64-linux-gnu/libcares.so.2.2.0
node 9946 *snip* mem REG 8,1 178096 22309 /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
node 9946 *snip* mem REG 8,1 121280 2762 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
node 9946 *snip* mem REG 8,1 22266144 22322 /usr/lib/x86_64-linux-gnu/libnode.so.64
node 9946 *snip* mem REG 8,1 165632 932 /usr/lib/x86_64-linux-gnu/ld-2.28.so
node 9946 *snip* 0r CHR 1,3 0t0 7292 /dev/null
node 9946 *snip* 1u unix 0x00000000687bd8d8 0t0 1745496 type=STREAM
node 9946 *snip* 2u unix 0x00000000687bd8d8 0t0 1745496 type=STREAM
node 9946 *snip* 3u a_inode 0,13 0 7223 [eventpoll]
node 9946 *snip* 4r FIFO 0,12 0t0 1745504 pipe
node 9946 *snip* 5w FIFO 0,12 0t0 1745504 pipe
node 9946 *snip* 6r FIFO 0,12 0t0 1745505 pipe
node 9946 *snip* 7w FIFO 0,12 0t0 1745505 pipe
node 9946 *snip* 8u a_inode 0,13 0 7223 [eventfd]
node 9946 *snip* 9u a_inode 0,13 0 7223 [eventpoll]
node 9946 *snip* 10r FIFO 0,12 0t0 1745506 pipe
node 9946 *snip* 11w FIFO 0,12 0t0 1745506 pipe
node 9946 *snip* 12u a_inode 0,13 0 7223 [eventfd]
node 9946 *snip* 13u a_inode 0,13 0 7223 [eventpoll]
node 9946 *snip* 14r FIFO 0,12 0t0 1745507 pipe
node 9946 *snip* 15w FIFO 0,12 0t0 1745507 pipe
node 9946 *snip* 16u a_inode 0,13 0 7223 [eventfd]
node 9946 *snip* 17r CHR 1,3 0t0 7292 /dev/null
node 9946 *snip* 18u IPv4 1745532 0t0 TCP localhost:55926->localhost:mysql (ESTABLISHED)
node 9946 *snip* 19u IPv6 1745544 0t0 TCP *:*snip* (LISTEN)
node 9946 *snip* 20u IPv4 1745536 0t0 TCP localhost:47794->localhost:*snip* (ESTABLISHED)
node 9946 *snip* 21u IPv6 1745545 0t0 UDP *:*snip*
node 9946 *snip* 22u IPv4 1745554 0t0 TCP localhost:55934->localhost:mysql (ESTABLISHED)
node 9946 *snip* 23u IPv4 1745556 0t0 TCP localhost:55936->localhost:mysql (ESTABLISHED)
node 9946 *snip* 24u IPv4 1745558 0t0 TCP localhost:55938->localhost:mysql (ESTABLISHED)
node 9946 *snip* 25u IPv4 1745560 0t0 TCP localhost:55940->localhost:mysql (ESTABLISHED)
node 9946 *snip* 26u IPv4 1745562 0t0 TCP localhost:55942->localhost:mysql (ESTABLISHED)
node 9946 *snip* 27u IPv4 1745564 0t0 TCP localhost:55944->localhost:mysql (ESTABLISHED)
node 9946 *snip* 28u IPv4 1746253 0t0 TCP localhost:55952->localhost:mysql (ESTABLISHED)
node 9946 *snip* 29u IPv4 1746255 0t0 TCP localhost:55954->localhost:mysql (ESTABLISHED)
node 9946 *snip* 30u IPv4 1746776 0t0 TCP localhost:56042->localhost:mysql (ESTABLISHED)
Metadata
Metadata
Assignees
Labels
v8 engineIssues and PRs related to the V8 dependency.Issues and PRs related to the V8 dependency.