cypress run occasionally crashes with node OOM errors

[bsmithEG]

Current behavior:

<--- Last few GCs --->

[902:0x14d4055f4000]  1414367 ms: Mark-sweep 2050.4 (2142.6) -> 2050.4 (2142.6) MB, 2327.2 / 0.0 ms  allocation failure GC in old space requested
[902:0x14d4055f4000]  1416781 ms: Mark-sweep 2050.4 (2142.6) -> 2050.4 (2141.6) MB, 2414.3 / 0.0 ms  last resort 
[902:0x14d4055f4000]  1418947 ms: Mark-sweep 2050.4 (2141.6) -> 2050.4 (2141.6) MB, 2165.5 / 0.0 ms  last resort 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2fbf9512d681 <JSObject>
    2: stringSlice(aka stringSlice) [buffer.js:556] [bytecode=0x193b5ab7ca49 offset=96](this=0x10df42502311 <undefined>,buf=0x334eb421b249 <Uint8Array map = 0x2c7df00ceb21>,encoding=0x2fbf9513e8a9 <String[4]: utf8>,start=0,end=404259)
    4: toString [buffer.js:629] [bytecode=0x193b5ab7c681 offset=151](this=0x334eb421b249 <Uint8Array map = 0x2c7df00ceb21>,encoding=0x2fbf9513e8a9 <String[4]: utf8...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 2: 0x7faa08f32887 [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 3: 0x7faa08aa1a57 [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 4: 0x7faa08aa19d5 [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 6: v8::internal::Factory::NewStringFromUtf8(v8::internal::Vector<char const>, v8::internal::PretenureFlag) [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 7: v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 8: node::StringBytes::Encode(v8::Isolate*, char const*, unsigned long, node::encoding, v8::Local<v8::Value>*) [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
 9: 0x7faa08f4e5d8 [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
10: 0x7faa08a71550 [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
11: 0x7faa088dfd98 [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
12: 0x7faa088df30e [/root/.cache/Cypress/3.0.1/Cypress/libnode.so]
13: 0x67697b843fd

Desired behavior:

No crash

Steps to reproduce:

circleci 2.0 config
chrome67-ff65 docker img
cypress-typescript-preprocessor
100 spec files

Versions

Cypress 3.0.1
Running chrome67-ff65 docker image on CircleCI 2.0 Ubuntu 16.04

[bsmithEG]

Are there any logs I could capture to help troubleshoot this? It seems like it happens about 10% of the time currently (10 out of ~100 runs).

Also, I've tried adjusting the node runtime options with my circleci yaml config:

- run:
  name: Test E2E with Chrome
  command: TOOL_NODE_FLAGS="--max_old_space_size=4096" npm run test:cypress:chrome

[brian-mann]

Different issue but comment is the same for outputting DEBUG logs.

#1953 (comment)

It appears that the process ran out of memory. Are you getting any messages from Circle about the process being killed because it exceeded the memory limit?

You might want to try setting this environment for electron: ELECTRON_ENABLE_STACK_DUMPING=1

I actually am not sure if --max_old_spec_size is actually applying here because I'm not sure if Electron respects those, since that's what's spawning the node process. I did a quick read through their docs and didn't find anything. Their github repo issues may have more to look for.

Since you have 100 specs files, approximately how long does your run take? Does it crash in the same place? different place? At the beginning? middle? end?

[bsmithEG]

@brian-mann

Thanks for the comment on the previous ticket and the electron flag, I'll try adding those to see if I can get any other relevant information to provide.

1. The run on average (from the start of the task that calls cypress run to the end of that task) takes about ~50 minutes.

2. It seems to crash around the last 20 minutes of the run, around the last ~30 tests, but in a different place (on a different spec) each time, so fairly intermittent but obviously after running for some time.

I'm trying to run parallel builds with circleci for this job now as a short term work around (~50 specs per build), I'll report back if I do or don't get any OOM crashes from that to see if it's something memory leaky/duration related to something the tests might be specifically doing.

[brian-mann]

@bsmithEG we are releasing 3.1.x sometime next week which will natively add parallelization. In Circle 2.0 all you'll do is set parallelization in the circle.yml to any number of instances and we'll automatically load balance and split the specs across all of the instances.

With that said - we've never seen OOM problems (especially in Circle) related to the main node process. It does hold some state, but not much, other than holding onto the test results. I suppose that if you're doing something totally unusual and really complex like dynamically generating tests with like a million lines of text, that it could probably OOM as a result.

Perhaps there is an area of the system I'm not accounting for us creating state in, but perhaps you could go into more detail about the nature of your tests and whether you feel any are unusual.

Perhaps try stopping the video recording. We do have to buffer those frames in memory as we write them to the video, but we designed that system to automatically drop frames if it's not able to keep up (if its CPU bound for instance). We might need to expose a flag that takes a memory dump after each spec so that we can detect any leaks or unaccounted for state.

OH: I forgot we have this ondocumented flag: CYPRESS_MEMORY=1

Add that as an environment variable and once a second it will output the current node process's memory. We should be able to easily see this increasing if there's a leak. Try that and see what you get. I would try that first before adding the DEBUG=cypress:* because it will generate a massive, massive stream of data which 99.9% of it is not useful until we get closer to the source of the problem.

[oiurch]

<--- Last few GCs --->
[12852:0x7f95d0828200]   350241 ms: Mark-sweep 2047.1 (2178.0) -> 2047.1 (2178.0) MB, 1834.4 / 0.6 ms  allocation failure GC in old space requested

[12852:0x7f95d0828200]   352641 ms: Mark-sweep 2047.1 (2178.0) -> 2047.0 (2173.5) MB, 2399.3 / 470.6 ms  last resort 

[12852:0x7f95d0828200]   354829 ms: Mark-sweep 2047.0 (2173.5) -> 2047.0 (2172.5) MB, 2186.8 / 465.6 ms  last resort 

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2d72279cb5d1 <Window map = 0x3224ecf0cd89>

    2: replace(this=0x270e3876ae71 <Very long string[361452]>,0x10f949d9aa09 <JSRegExp <String[23]: url\((['"])([^'"]*)\1\)>>,0x270e3876aef9 <JSFunction (sfi = 0x39e103fc5d19)>)

    4: makePathsAbsoluteToStylesheet [https://website/__cypress/runner/cypress_runner.js:61280] [bytecode=0x39e103fc5e21 offset=84](this=0x89076328f1 <JSGlobal Object>,styles=0x270e3876a..

The same issue is happening for me running locally on my machine. This happens after running just 5 UI tests, on execution of 6, interface becomes laggy and after some time it crashes.

Increasing --max_old_spec_size is not fixing the issue for me, is there any workaround without the circle ci, for local runs? I don't think that it's ok if this issue happens just after 5 tests execution.

[TLadd]

Since you're using typescript too, it might be the same issue as #2316

[shinobi64]

I've run now into the same issue - Cypress 3.1.0, Mac OS X 10.13.6, Electron 59
The issue occurs while running a suite which contains of two tests. In the second test on the same request it fails with

<--- Last few GCs --->

[83531:0x7fe0d680bc00]    57174 ms: Mark-sweep 2046.6 (2157.3) -> 2046.6 (2157.8) MB, 159.9 / 0.5 ms  allocation failure GC in old space requested
[83531:0x7fe0d680bc00]    57398 ms: Mark-sweep 2046.6 (2157.8) -> 2046.6 (2123.8) MB, 223.6 / 34.7 ms  last resort 
[83531:0x7fe0d680bc00]    57604 ms: Mark-sweep 2046.6 (2123.8) -> 2046.6 (2122.3) MB, 205.4 / 33.0 ms  last resort 

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2628cba4b5d1 <Window map = 0x2fd68238cd89>
    2: replace(this=0x170eef7dd8e9 <Very long string[857083]>,0x3e6fee24e431 <JSRegExp <String[23]: url\((['"])([^'"]*)\1\)>>,0x170eef7dd971 <JSFunction (sfi = 0x48f53597449)>)
    3: makePathsAbsoluteToStylesheet [http://localhost:4444/__cypress/runner/cypress_runner.js:~61447] [pc=0x2e42e426b914](this=0x2d94465b28c9 <JSGlobal Object>,styles=0x170eef7dd8e9 <Very long string[8...

Memory statements from CYPRESS_MEMORY=1 did not show a significant increase and as well no direct shortage of memory

memory info { rss: 238333952,
  heapTotal: 129863680,
  heapUsed: 116772008,
  external: 676034 }
memory info { rss: 232624128,
  heapTotal: 126193664,
  heapUsed: 114479928,
  external: 438446 }
memory info { rss: 232689664,
  heapTotal: 126193664,
  heapUsed: 114558776,
  external: 439619 }

The third info was taken after the last request was send - and directly after that the GC statement showed up.

After setting video:false in the configuration there was no change to the behavior. Same for the configuration option numTestsKeptInMemory a reduction from the default value to e.g. did not provided a different result.

We're not running typescript - plain javascript tests.

[jennifer-shehane]

We made several performance improvements in our latest updates, particularly version 3.3.2, can we get an update on whether this issue is still occurring in our latest version? Thanks!

[sunnyyadav100]

I can see memory issue in 3.4.1 -

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Library/Caches/Cypress/3.4.1/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 2: node::FatalError(char const*, char const*) [/Library/Caches/Cypress/3.4.1/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 3: v8::internal::FatalProcessOutOfMemory(char const*) [/Library/Caches/Cypress/3.4.1/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
 4: v8::internal::FatalProcessOutOfMemory(char const*) [/Library/Caches/Cypress/3.4.1/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]


<--- Last few GCs --->
21: v8::internal::compiler::BranchElimination::ControlPathConditions::operator==(v8::internal::compiler::BranchElimination::Cont
[76583:0x7fd4d902c600]  1206187 ms: Mark-sweep 2049.3 (2125.3) -> 2049.3 (2122.3) MB, 1500.0 / 0.0 ms  (+ 0.0 ms in 0 steps sincrolPathConditions const&) const [/Library/Caches/Cypress/3.4.1/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Libraries/libnode.dylib]
e start of marking, biggest step 0.0 ms, walltime since start of marking 1500 ms) last resort GC in old space requested

[mschaaf]

We run into the same issue with version 3.4.1. For now, we solve it by running our test suite in several chunks via the --specs command-line option, so that every chunk runs in a fresh cypress process.

[bondar-artem]

I have the same issue. Cypress 3.4.1, running in Docker container, TypeScript.
Host has 8 GB memory and after 2 hours test run there is an error with run out of the memory... :(

The problem is not in the amount of physical memory on host, cypress crash when Cypress Helper process reach memory size 2.1 GB..

Making this config https://github.com/cypress-io/cypress/issues/2316#issuecomment-516824284 fixed the issue. IMPORTANT: use ts-config of version 4.4.2. The latest version do not work for some reason...