Memory issue on fedora in latest V8 (8.8) requirement

[gengjiawen]

According to /usr/bin/time -v on my machine, compilation of array-sort-tq-csa.o takes about 810 MB of memory.

Do you know if that's increased from before? It could very well be that this version of V8 has tipped the memory requirements for compilation such that the 2GiB is no longer enough and we need to add either more dedicated or swap to bring the Fedora hosts on par with the others (4GiB seems to be what other similar hosts are on, nodejs/node#36139 (comment)). Maybe open an issue over in nodejs/build?

Originally posted by @richardlau in nodejs/node#36139 (comment)

[richardlau]

From nodejs/node#36139 (comment):

Still failing: https://ci.nodejs.org/job/node-test-commit-linux/39436/nodes=fedora-latest-x64/console

It looks like the host is running out of memory.

09:56:58 make[2]: *** [tools/v8_gypfiles/v8_initializers.target.mk:385: /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/obj.target/v8_initializers/gen/torque-generated/test/torque/test-torque-tq-csa.o] Terminated
09:58:43 FATAL: command execution failed
09:58:43 java.nio.channels.ClosedChannelException

and from the system log on test-rackspace-fedora32-x64-1:

Jan 18 09:58:43 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: A process of this unit has been killed by the OOM killer.
Jan 18 09:58:43 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Main process exited, code=exited, status=143/n/a
Jan 18 09:58:43 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Failed with result 'oom-kill'.
Jan 18 09:58:43 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Consumed 16h 38min 49.487s CPU time.
Jan 18 09:59:13 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Scheduled restart job, restart counter is at 5.
Jan 18 09:59:13 test-rackspace-fedora32-x64-1 systemd[1]: Stopped Jenkins Slave.
Jan 18 09:59:13 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Consumed 16h 38min 49.487s CPU time.
Jan 18 09:59:13 test-rackspace-fedora32-x64-1 systemd[1]: Started Jenkins Slave.

The same for the earlier https://ci.nodejs.org/job/node-test-commit-linux/nodes=fedora-latest-x64/39376/console

08:13:30 make[2]: *** [tools/v8_gypfiles/v8_initializers.target.mk:385: /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/obj.target/v8_initializers/gen/torque-generated/third_party/v8/builtins/array-sort-tq-csa.o] Terminated
08:14:54 make[2]: *** [tools/v8_gypfiles/v8_initializers.target.mk:385: /home/iojs/build/workspace/node-test-commit-linux/nodes/fedora-latest-x64/out/Release/obj.target/v8_initializers/gen/torque-generated/test/torque/test-torque-tq-csa.o] Terminated
08:14:54 FATAL: command execution failed
08:14:54 java.nio.channels.ClosedChannelException

Jan 14 08:14:54 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: A process of this unit has been killed by the OOM killer.
Jan 14 08:14:54 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Main process exited, code=exited, status=143/n/a
Jan 14 08:14:54 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Failed with result 'oom-kill'.
Jan 14 08:14:54 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Consumed 1h 12min 3.598s CPU time.
Jan 14 08:15:24 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Scheduled restart job, restart counter is at 3.
Jan 14 08:15:24 test-rackspace-fedora32-x64-1 systemd[1]: Stopped Jenkins Slave.
Jan 14 08:15:24 test-rackspace-fedora32-x64-1 systemd[1]: jenkins.service: Consumed 1h 12min 3.598s CPU time.
Jan 14 08:15:24 test-rackspace-fedora32-x64-1 systemd[1]: Started Jenkins Slave.

From nodejs/node#36139 (comment):

How much memory does it have compared to other similar hosts?

Appears to be 2GiB

[root@test-rackspace-fedora32-x64-1 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:          1.9Gi       290Mi       402Mi        10Mi       1.2Gi       1.5Gi
Swap:            0B          0B          0B
[root@test-rackspace-fedora32-x64-1 ~]#

For comparison, the other fedora-latest-x64 host:

$ ssh test-digitalocean-fedora32-x64-1 "free -h"
              total        used        free      shared  buff/cache   available
Mem:          1.9Gi       317Mi       202Mi       0.0Ki       1.4Gi       1.4Gi
Swap:            0B          0B          0B
$

The two fedora-last-latest-x64 hosts:

$ ssh test-digitalocean-fedora30-x64-1 "free -h"
              total        used        free      shared  buff/cache   available
Mem:          1.9Gi       292Mi       1.3Gi       0.0Ki       368Mi       1.5Gi
Swap:            0B          0B          0B
$ ssh test-digitalocean-fedora30-x64-2 "free -h"
              total        used        free      shared  buff/cache   available
Mem:          3.8Gi       284Mi       2.0Gi       0.0Ki       1.6Gi       3.3Gi
Swap:            0B          0B          0B
$

centos7-64-gcc8:

$ ssh test-rackspace-centos7-x64-1 "free -h"
              total        used        free      shared  buff/cache   available
Mem:           1.8G        256M        975M        3.5M        600M        1.4G
Swap:          2.0G        295M        1.7G
$ ssh test-softlayer-centos7-x64-1 "free -h"
              total        used        free      shared  buff/cache   available
Mem:           1.8G        142M        1.3G        6.0M        376M        1.5G
Swap:          2.0G        260M        1.7G
$

[richardlau]

I've added the build agenda label to this in case nobody gets around to looking at this. I know that the WG members from Red Hat are busy this week.

@rvagg had some suggestions in nodejs/node#36139 (comment) to see if clearing things up on the existing hosts helps. Otherwise we might look at either adding 2GiB of swap to the Fedora hosts (if we have the disk space) or bumping the allocated memory.

[gengjiawen]

I'm only just seeing this so don't have too much intelligent to add (such as why it's failing) other than:

• Failures on standard configurations are intended to be a signal that something is not right, switching to clang might "fix" this problem, but then you're just shipping software that's likely to fail on the particular configuration that's failing in CI. I see suggestion of memory, is there a known OOM here? I'm not seeing that in the log for that last CI run.
• Fedora 33 is out so fedora-latest needs to be upgraded to that when someone (probably me) has time to do that. But Fedora 32, which is failing here, will still be in the mix as fedora-last-latest. It'd be quite interesting to see whether this is still failing on 33.
• Someone with @nodejs/build test permissions could log in to the two machines (test-rackspace-fedora32-x64-1 and test-digitalocean-fedora32-x64-1), run dnf upgrade, update slave.jar, clear out ~iojs/build/workspace and reboot. I reckon it's been ages since anyone was in these machines and there might be something local that would be fixed up with a clean (maybe there's a memory hogging program in the background?).

Originally posted by @rvagg in nodejs/node#36139 (comment)

[mhdawson]

@richardlau is the failure only on Fedora because the machines were configured with less memory or is it something specific to Fedora?

[richardlau]

@mhdawson I haven't found anything yet to suggest a Fedora specific issue vs a simple memory issue.

[mhdawson]

@richardlau thanks, in terms of:

Someone with @nodejs/build test permissions could log in to the two machines (test-rackspace-fedora32-x64-1 and test-digitalocean-fedora32-x64-1), run dnf upgrade, update slave.jar, clear out ~iojs/build/workspace and reboot. I reckon it's been ages since anyone was in these machines and there might be something local that would be fixed up with a clean (maybe there's a memory hogging program in the background?).

Is that something you will have time to do on one of the machines?

[richardlau]

@mhdawson I'm not sure. I don't have much work time available in the remainder of this week outside of the scheduled Red Hat meetings. I could make time next week.

[rvagg]

I updated those two machines, cleared workspaces and rebooted. Here's a green run for you for that problematic PR: https://ci.nodejs.org/job/node-test-commit-linux/39601/

We've historically targeted ~2Gb ~2 core machines in CI, they should be our most common configuration. If it were a universal memory problem then I'd expect to see it in more places than just one type of machine. My guess is that it's a bug in the toolchain that's been resolved. There were a number of toolchain updates in the big batch of updates installed, including gcc and glibc. The biggest memory hog on the machine is java running Jenkins, sitting at ~200Mb, and they're back near that level after being restarted so it doesn't look like they were bloating and there wasn't anything else taking up very much.

🤷 we'll keep an eye on these machines but for now it seems to be addressed.

[gengjiawen]

Nice work ❤️ @rvagg