Memory leak in version > 3.20.0

[kurigashvili-godaddy]

Describe the bug

We have upgraded to 3.24.0 all our team services (10) and all of them started running out of memory.

All of them are running in K8S. It would take between 30 min to 3 hours of a pod run out of memory and restart.

We downgraded to 3.20.0 and the issue disappeared.

We were not seeing issue in a staging environment so I believe the issue can be noticed under higher loads.

To Reproduce

Steps to reproduce the behavior:
Run http service with high load and observe that memory slowly increases until eventually node process dies with "JavaScript heap out of memory" error.

Expected behavior

Do not run our of memory like version 3.20.0

Environment (please complete the following information)

• OS: [Linux]
• Node.js version: 12.22.1
• APM Server version: 7.15.2
• Agent version: 3.24.0

How are you starting the agent? (please tick one of the boxes)

• Calling agent.start() directly (e.g. require('elastic-apm-node').start(...))
• Requiring elastic-apm-node/start from within the source code
• Starting node with -r elastic-apm-node/start

Additional context

• Agent config options

  Click to expand

  replace this line with your agent config options
  

• package.json dependencies:

  Click to expand

  replace this line with your dependencies section from package.json
  

[astorm]

Thanks for reporting @kurigashvili-godaddy, that's definitely not the intended behavior.

Usually the fastest path to resolution on these sorts of issues is an isolated bit of code that reproduces the problem. Do you happen to have an isolated reproduction of this problem (GitHub repository, Dockerfile)? If not, could you share what these services are doing? i.e. what web framework you're using, what libraries you're using to query databases/datastores, any third party promise libraries you might be using, etc.

Basically any information that would help us reproduce the problem so we can diagnose what's going on.

[kurigashvili-godaddy]

I will try to create an isolated reproduction of this problem.
The simplest service that had the problem:
crud service based on express 4.17.1 talking to dynamo db (aws-sdk 2.983.0).

[trentm]

@kurigashvili-godaddy Thanks. Some other questions that might help us isolate the problematic changes:

• Is using the aws-sdk, and specifically DynamoDB, common to all the services that were running out of memory/leaking?
• If you are able to try separately with versions v3.21.0 and v3.22.0 of the Node.js APM agent, that would help us segment agent changes that could be the culprit.

[kurigashvili-godaddy]

• All of the services are using DynamoDB
• I will see if I can run v3.21.0 and v3.22.0. This issue only happened in prod so it is a little tricky to be testing in prod.

I tried reproduce the issue locally but I was not able to reproduce :(

[trentm]

• This issue only happened in prod so it is a little tricky to be testing in prod.

Definitely understood.

I haven't reproduced with a trivial Express-using app (node v12.22.6, elastic-apm-node@3.24.0) loaded up with hey -c 50 -q 10 -z 180m http://localhost:3000/. Obviously this isn't representative.

const apm = require('elastic-apm-node').start({
    serviceName: 'trentm-leakplay'
})

const express = require('express')
const app = express()
const port = 3000

function doAThing(opts, cb) {
  setTimeout(cb, Math.random() * 10)
}

app.get('/', (req, res) => {
  const s = apm.startSpan('thing')
  doAThing({}, function () {
    s.end()
    res.send('pong')
  })
})

app.listen(port, () => {
  console.log(`listening at http://localhost:${port}`)
})

[vkrivtsov-stuzo]

@trentm we have the same issue with this version on particular services, most of the services are doing well, I will provide the report a bit later

[trentm]

@vkrivtsov-stuzo I would love to see details on your experiences here. Anything to isolate common dependencies or exact APM agent versions that could be causing the leak.

[Tirke]

Hello, we also have some strange memory issues recently under load. Do you guys have any update on that?
We are also using DynamoDB in many of our services.

@kurigashvili-godaddy maybe one of the difference between your local test and your production is that you have more errors logged in production?

@trentm From a heap dump in production I see a lot of memory being retained by AsyncHooksRunContextManager (biggest Map is also related to that)

[trentm]

@Tirke I'd love to get some details from you to help track this down, or for me to be able to repro.

• What version(s) of node have you seen this with?
• What version(s) of the APM agent have you seen this with? If you are able to try with any of the ??? versions below, I would very much appreciate it, but I understand if you cannot. So far we have:
  • v3.24.0 leak (this added AsyncHooksRunContextManager and other significant changes, so is a likely leak cause candidate)
  • v3.23.0 ??? (only had a change to support @elastic/elasticsearch@8 so unlikely to be a leak cause)
  • v3.22.0 ???
  • v3.21.x ???
  • v3.20.0 no leak
• Would you be willing to privately share your heap dump so that I can take a look? Again I understand if you cannot. Here is my work email if so.
• Are you able to share the package.json "dependencies" for your application? It is interesting that you both use DynamoDB. However, our instrumentation for aws-sdk hasn't changed between v3.20 and v3.24.
• Any other details you can think of that might help me reproduce would be much appreciated.