Skip to content

Segmentation fault in v8::internal::compiler::(anonymous namespace)::MayAlias (10.15.2 from Debian Buster) #31484

@TimWolla

Description

@TimWolla
  • Version:
root@*snip*:~# node -v
v10.15.2
root@*snip*:~# dpkg -l nodejs
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Architecture Description
+++-==============-==============-============-==================================================
ii  nodejs         10.15.2~dfsg-2 amd64        evented I/O for V8 javascript - runtime executable
  • Platform:

Debian Buster on amd64.

root@*snip*:~# uname -a
Linux *snip* 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux
  • Subsystem:

I'm seeing a more or less regular crash of a non-trivial application within libnode.so.64. It's running on a low-traffic staging system. Find below an excerpt from the syslog of the affected machine. Times are in UTC.

Jan 23 16:45:12 *snip* kernel: [502165.519232] node[21284]: segfault at 8 ip 00007feca7142b6b sp 00007feca1bcb110 error 4 in libnode.so.64[7feca6d02000+abe000]
Jan 23 16:45:12 *snip* kernel: [502165.531802] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 17:13:28 *snip* kernel: [503861.549372] node[21318]: segfault at 8 ip 00007f1bd3ed8b6b sp 00007f1bcb7fd110 error 4 in libnode.so.64[7f1bd3a98000+abe000]
Jan 23 17:13:28 *snip* kernel: [503861.565373] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 18:09:58 *snip* kernel: [507250.954363] node[21363]: segfault at 8 ip 00007f9b01d95b6b sp 00007f9afd820110 error 4 in libnode.so.64[7f9b01955000+abe000]
Jan 23 18:09:58 *snip* kernel: [507250.962099] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 18:38:34 *snip* kernel: [508967.449433] node[21430]: segfault at 8 ip 00007fc0ee66cb6b sp 00007fc0e90f5110 error 4 in libnode.so.64[7fc0ee22c000+abe000]
Jan 23 18:38:34 *snip* kernel: [508967.461002] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 19:07:10 *snip* kernel: [510683.485209] node[21480]: segfault at 8 ip 00007f7527a1db6b sp 00007f7522ca7110 error 4 in libnode.so.64[7f75275dd000+abe000]
Jan 23 19:07:10 *snip* kernel: [510683.493478] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0
Jan 23 20:04:52 *snip* kernel: [514145.480506] node[26613]: segfault at 8 ip 00007fe9488dcb6b sp 00007fe93fffe110 error 4 in libnode.so.64[7fe94849c000+abe000]
Jan 23 20:04:52 *snip* kernel: [514145.501037] Code: 48 89 e5 41 54 53 48 83 ec 10 64 48 8b 04 25 28 00 00 00 48 89 45 e8 31 c0 b8 01 00 00 00 48 39 f7 74 5c 48 8b 47 08 48 89 f3 <48> 8b 76 08 49 89 fc 48 8d 7d e0 48 89 45 e0 e8 31 a9 bc ff 84 c0

After becoming aware of the issue I made sure that the process could dump core and I also installed the relevant debug symbols. My understanding is that node crashes within v8's JIT (?) compiler:

Core was generated by `/usr/bin/node dist/server.js'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x221fc90, b=0x0) at ../deps/v8/src/compiler/node.h:267
267	  Type type() const { return type_; }
[Current thread is 1 (Thread 0x7fe93ffff700 (LWP 26613))]
(gdb) list
262	
263	  // Only NodeProperties should manipulate the op.
264	  void set_op(const Operator* op) { op_ = op; }
265	
266	  // Only NodeProperties should manipulate the type.
267	  Type type() const { return type_; }
268	  void set_type(Type type) { type_ = type; }
269	
270	  // Only NodeMarkers should manipulate the marks on nodes.
271	  Mark mark() const { return mark_; }
(gdb) bt
#0  v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x221fc90, b=0x0) at ../deps/v8/src/compiler/node.h:267
#1  0x00007fe9488dcc3c in v8::internal::compiler::(anonymous namespace)::MayAlias (b=0x1f2e570, a=0x221fc90) at ../deps/v8/src/utils.h:355
#2  v8::internal::compiler::(anonymous namespace)::MayAlias (a=0x221fc90, b=0x1f2e570) at ../deps/v8/src/compiler/load-elimination.cc:37
#3  0x00007fe9488dde0d in v8::internal::compiler::LoadElimination::AliasStateInfo::MayAlias (this=0x7fe93fffe230, other=0x1f2e570) at ../deps/v8/src/compiler/load-elimination.cc:679
#4  0x00007fe9488dea2d in v8::internal::compiler::LoadElimination::AbstractField::Kill (this=this@entry=0x7fe9340792e0, alias_info=..., name=name@entry=..., zone=zone@entry=0x1ea7490)
    at ../deps/v8/src/compiler/load-elimination.cc:372
#5  0x00007fe9488deb7c in v8::internal::compiler::LoadElimination::AbstractState::KillFields (this=this@entry=0x7fe93407a250, object=object@entry=0x221fc90, name=..., zone=0x1ea7490)
    at ../deps/v8/src/compiler/load-elimination.cc:653
#6  0x00007fe9488e04f6 in v8::internal::compiler::LoadElimination::ReduceStoreField (this=0x7fe93fffea20, node=0x219eea0) at ../deps/v8/src/zone/zone-allocator.h:37
#7  0x00007fe94891e72b in v8::internal::compiler::(anonymous namespace)::SourcePositionWrapper::Reduce (this=0x229ba18, node=0x219eea0) at ../deps/v8/src/compiler/pipeline.cc:651
#8  0x00007fe94885adc8 in v8::internal::compiler::GraphReducer::Reduce (this=this@entry=0x7fe93fffe920, node=node@entry=0x219eea0) at ../deps/v8/src/compiler/graph-reducer.cc:85
#9  0x00007fe94885b195 in v8::internal::compiler::GraphReducer::ReduceTop (this=this@entry=0x7fe93fffe920) at ../deps/v8/src/compiler/graph-reducer.cc:152
#10 0x00007fe94885b388 in v8::internal::compiler::GraphReducer::ReduceNode (this=this@entry=0x7fe93fffe920, node=<optimized out>) at ../deps/v8/src/compiler/graph-reducer.cc:56
#11 0x00007fe94885b48d in v8::internal::compiler::GraphReducer::ReduceGraph (this=this@entry=0x7fe93fffe920) at ../deps/v8/src/compiler/graph.h:79
#12 0x00007fe94892b19e in v8::internal::compiler::LoadEliminationPhase::Run (this=this@entry=0x7fe93fffeca0, data=0x1fd3f98, temp_zone=temp_zone@entry=0x1ea7490) at ../deps/v8/src/compiler/pipeline.cc:1457
#13 0x00007fe94892c6b3 in v8::internal::compiler::PipelineImpl::Run<v8::internal::compiler::LoadEliminationPhase> (this=0x1fd4118) at ../deps/v8/src/compiler/zone-stats.h:29
#14 v8::internal::compiler::PipelineImpl::OptimizeGraph (this=this@entry=0x1fd4118, linkage=0x220c510) at ../deps/v8/src/compiler/pipeline.cc:1912
#15 0x00007fe94892c710 in v8::internal::compiler::PipelineCompilationJob::ExecuteJobImpl (this=0x1fd3de0) at ../deps/v8/src/compiler/pipeline.cc:847
#16 0x00007fe9487fcf91 in v8::internal::OptimizedCompilationJob::ExecuteJob (this=this@entry=0x1fd3de0) at ../deps/v8/src/compiler.cc:223
#17 0x00007fe9487f7fab in v8::internal::OptimizingCompileDispatcher::CompileNext (this=this@entry=0x1e8ae70, job=0x1fd3de0) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:118
#18 0x00007fe9487f826d in v8::internal::OptimizingCompileDispatcher::CompileTask::RunInternal (this=0x1eb8a90) at ../deps/v8/src/compiler-dispatcher/optimizing-compile-dispatcher.cc:69
#19 0x00007fe9485b1162 in node::BackgroundRunner (data=0x1e1f480) at ../src/node_platform.cc:42
#20 0x00007fe945563fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#21 0x00007fe9454944cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I have the core dump on record and can provide additional information from the core file on request.

The application is launched using systemd. Find below the unit file for your reference:

Unit File (click to expand)
root@*snip*:~# systemctl cat *snip*
# /etc/systemd/system/*snip*.service
#
# Ansible managed
#

[Unit]
Description=*snip*
Requires=mysql.service
After=mysql.service
Wants=*snip*

[Service]
Restart=always
User=*snip*
WorkingDirectory=/srv/*snip*/
ExecStart=/usr/bin/node dist/server.js

Environment=*snip*
Environment=*snip*
Environment=*snip*
Environment=*snip*
Environment=*snip*
EnvironmentFile=/etc/*snip*/environment

StateDirectory=*snip*
StateDirectoryMode=0700
CacheDirectory=*snip*
CacheDirectoryMode=0700
ReadWritePaths=*snip*
ReadWritePaths=/tmp/
# PrivateTmp=true

ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io
NoNewPrivileges=true
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/*snip*.service.d/override.conf
[Service]
Environment=DEBUG=*snip*:*
ReadWritePaths=/srv/*snip*/storage/

Find below a list of open files from the process after it was restarted automatically by systemd:

lsof -p $pid (click to expand)
root@*snip*:~# lsof -p 9946
COMMAND  PID    USER   FD      TYPE             DEVICE SIZE/OFF    NODE NAME
node    9946 *snip*  cwd       DIR                8,1     4096  149468 /usr/share/*snip*
node    9946 *snip*  rtd       DIR                8,1     4096       2 /
node    9946 *snip*  txt       REG                8,1    14264   22337 /usr/bin/node
node    9946 *snip*  mem       REG                8,1    93000     955 /usr/lib/x86_64-linux-gnu/libresolv-2.28.so
node    9946 *snip*  mem       REG                8,1    26960     946 /usr/lib/x86_64-linux-gnu/libnss_dns-2.28.so
node    9946 *snip*  mem       REG                8,1    55792     947 /usr/lib/x86_64-linux-gnu/libnss_files-2.28.so
node    9946 *snip*  mem       REG                8,1    89016     944 /usr/lib/x86_64-linux-gnu/libnsl-2.28.so
node    9946 *snip*  mem       REG                8,1  1824496     937 /usr/lib/x86_64-linux-gnu/libc-2.28.so
node    9946 *snip*  mem       REG                8,1   146968     954 /usr/lib/x86_64-linux-gnu/libpthread-2.28.so
node    9946 *snip*  mem       REG                8,1   100712     912 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
node    9946 *snip*  mem       REG                8,1  1579448     941 /usr/lib/x86_64-linux-gnu/libm-2.28.so
node    9946 *snip*  mem       REG                8,1  1570256    3681 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.25
node    9946 *snip*  mem       REG                8,1    35808     956 /usr/lib/x86_64-linux-gnu/librt-2.28.so
node    9946 *snip*  mem       REG                8,1    14592     940 /usr/lib/x86_64-linux-gnu/libdl-2.28.so
node    9946 *snip*  mem       REG                8,1 27190832   21975 /usr/lib/x86_64-linux-gnu/libicudata.so.63.1
node    9946 *snip*  mem       REG                8,1  1886216   21980 /usr/lib/x86_64-linux-gnu/libicuuc.so.63.1
node    9946 *snip*  mem       REG                8,1  2984248   21976 /usr/lib/x86_64-linux-gnu/libicui18n.so.63.1
node    9946 *snip*  mem       REG                8,1   593696    4776 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
node    9946 *snip*  mem       REG                8,1  3031904    4767 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
node    9946 *snip*  mem       REG                8,1   158128   14137 /usr/lib/x86_64-linux-gnu/libnghttp2.so.14.17.1
node    9946 *snip*  mem       REG                8,1    71544   22287 /usr/lib/x86_64-linux-gnu/libcares.so.2.2.0
node    9946 *snip*  mem       REG                8,1   178096   22309 /usr/lib/x86_64-linux-gnu/libuv.so.1.0.0
node    9946 *snip*  mem       REG                8,1   121280    2762 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
node    9946 *snip*  mem       REG                8,1 22266144   22322 /usr/lib/x86_64-linux-gnu/libnode.so.64
node    9946 *snip*  mem       REG                8,1   165632     932 /usr/lib/x86_64-linux-gnu/ld-2.28.so
node    9946 *snip*    0r      CHR                1,3      0t0    7292 /dev/null
node    9946 *snip*    1u     unix 0x00000000687bd8d8      0t0 1745496 type=STREAM
node    9946 *snip*    2u     unix 0x00000000687bd8d8      0t0 1745496 type=STREAM
node    9946 *snip*    3u  a_inode               0,13        0    7223 [eventpoll]
node    9946 *snip*    4r     FIFO               0,12      0t0 1745504 pipe
node    9946 *snip*    5w     FIFO               0,12      0t0 1745504 pipe
node    9946 *snip*    6r     FIFO               0,12      0t0 1745505 pipe
node    9946 *snip*    7w     FIFO               0,12      0t0 1745505 pipe
node    9946 *snip*    8u  a_inode               0,13        0    7223 [eventfd]
node    9946 *snip*    9u  a_inode               0,13        0    7223 [eventpoll]
node    9946 *snip*   10r     FIFO               0,12      0t0 1745506 pipe
node    9946 *snip*   11w     FIFO               0,12      0t0 1745506 pipe
node    9946 *snip*   12u  a_inode               0,13        0    7223 [eventfd]
node    9946 *snip*   13u  a_inode               0,13        0    7223 [eventpoll]
node    9946 *snip*   14r     FIFO               0,12      0t0 1745507 pipe
node    9946 *snip*   15w     FIFO               0,12      0t0 1745507 pipe
node    9946 *snip*   16u  a_inode               0,13        0    7223 [eventfd]
node    9946 *snip*   17r      CHR                1,3      0t0    7292 /dev/null
node    9946 *snip*   18u     IPv4            1745532      0t0     TCP localhost:55926->localhost:mysql (ESTABLISHED)
node    9946 *snip*   19u     IPv6            1745544      0t0     TCP *:*snip* (LISTEN)
node    9946 *snip*   20u     IPv4            1745536      0t0     TCP localhost:47794->localhost:*snip* (ESTABLISHED)
node    9946 *snip*   21u     IPv6            1745545      0t0     UDP *:*snip* 
node    9946 *snip*   22u     IPv4            1745554      0t0     TCP localhost:55934->localhost:mysql (ESTABLISHED)
node    9946 *snip*   23u     IPv4            1745556      0t0     TCP localhost:55936->localhost:mysql (ESTABLISHED)
node    9946 *snip*   24u     IPv4            1745558      0t0     TCP localhost:55938->localhost:mysql (ESTABLISHED)
node    9946 *snip*   25u     IPv4            1745560      0t0     TCP localhost:55940->localhost:mysql (ESTABLISHED)
node    9946 *snip*   26u     IPv4            1745562      0t0     TCP localhost:55942->localhost:mysql (ESTABLISHED)
node    9946 *snip*   27u     IPv4            1745564      0t0     TCP localhost:55944->localhost:mysql (ESTABLISHED)
node    9946 *snip*   28u     IPv4            1746253      0t0     TCP localhost:55952->localhost:mysql (ESTABLISHED)
node    9946 *snip*   29u     IPv4            1746255      0t0     TCP localhost:55954->localhost:mysql (ESTABLISHED)
node    9946 *snip*   30u     IPv4            1746776      0t0     TCP localhost:56042->localhost:mysql (ESTABLISHED)

Metadata

Metadata

Assignees

No one assigned

    Labels

    v8 engineIssues and PRs related to the V8 dependency.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions