too many worker processes spawned in multi-project repo

[spudly]

🐛 Bug Report

I've been dealing with out of memory and/or EAGAIN errors for a few months now, ever since I upgraded to Jest 20 & started using the multi-project runner. I've been running with --runInBand as a workaround.

I believe this is happening because Jest creates too many worker processes when building the haste-map cache if you have multiple projects & maxWorkers > 1.

• For each project being tested by jest, jest creates a haste map. See https://github.com/facebook/jest/blob/master/packages/jest-core/src/cli/index.ts#L120-L132
• For each haste map, jest creates a worker pool with maxWorkers workers. See https://github.com/facebook/jest/blob/master/packages/jest-haste-map/src/index.ts#L736-L740

This means that instead of restricting the number of workers to maxWorkers overall, it restricts it to maxWorkers per project (maxWorkers * numProjects). In my case that's 226 projects and 8 cpu cores, so 8 * 226 = 1808 worker processes. When I run jest it chugs for about a minute, then either runs out of memory or hits the OS's process limit.

Note: this is only a problem while it is building the cache. Once the cache is built, jest will start actually testing the files and does so with the right number of workers.

To Reproduce

You should be able to reproduce this in any multi-project setup. Just run the tests in multi-project mode and watch how many worker processes get spawned while it's building the haste-map cache, before the tests start running.

I was able to reproduce it with the jest repo:

• Clone the jest repo
• yarn install
• open the mac os "Activity Monitor" (or an equivalent process monitor on another platform) and filter for node processes
• yarn jest --clearCache
• yarn jest
• watch how many new node processes show up in the activity monitor

When I did this with the jest, I saw approximately 25 worker processes running. I expected no more than 8.

Expected behavior

It should never spawn more than maxWorkers processes.

Link to repl or repo (highly encouraged)

See above. I was able to reproduce this with the jest repo.

envinfo

  System:
    OS: macOS High Sierra 10.13.6
    CPU: (8) x64 Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
  Binaries:
    Node: 12.9.1 - ~/.nvm/versions/node/v12.9.1/bin/node
    Yarn: 1.19.1 - ~/.nvm/versions/node/v12.9.1/bin/yarn
    npm: 6.10.2 - ~/.nvm/versions/node/v12.9.1/bin/npm

[thymikee]

Looking at this quickly, it seems like only scanning for haste map is affected, tests should still be scheduled to run on maxWorkers cores, right?

Short-term, I think dividing workers about evenly per project should help, but that heuristic is obviously not perfect and needs heavy testing under various circumstances. Imagine having a small project A and and big project B, both would get 4 cores – project A will be collected sooner, which would leave only 4 cores for project B, while we have 4 more available, such a waste. And I don't think we currently have a way to dynamically change worker count for haste map or even jest-worker.

[spudly]

Looking at this quickly, it seems like only scanning for haste map is affected, tests should still be scheduled to run on maxWorkers cores, right?

Yes, that's right.

Short-term, I think dividing workers about evenly per project should help

Maybe I don't understand the idea, but does that even make sense when there are more projects than cores?

[thymikee]

Or we could run these promises in a sequence instead of in parallel (Promise.all), I think that was the original intention. It also should get rid of my concerns about optimal solution

[spudly]

Would it make sense to hoist the worker pool such that we can give the same pool to every HastMap instance? I'm thinking that jest-core could create a single worker pool and pass it to jest-haste-map as an option/parameter on the HastMap constructor.

This way there will never be too many workers and they will all be fully utilized (they won't be sitting around doing nothing while they wait for other workers to finish, which is what would happen if we just chained the promises in jest-core).

[thymikee]

Seems like a good idea to create worker farm in jest-core and execute createHasteMap on each worker. However, as a band-aid, I quickly put together #9259. Feel free to work on a more complete solution if you like :).

I didn't want to spend too much time on this, as this solution already works better than the current one, and there's some ongoing work (@scotthovestadt should have more details) to improve stuff around haste maps in future anyway.

[spudly]

@scotthovestadt, will the changes you're making address this issue or should I start working on my own PR to fix this? I'm not very familiar with this codebase but It's a pretty big issue for my team.

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.