TLSSocket appears to be leaking, possibly on Connection Reset with HTTP2

[clshortfuse]

Version

v16.14.2

Platform

Linux host.ec2.internal 4.14.243-185.433.amzn2.x86_64 #1 SMP Mon Aug 9 05:55:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

http2 streams http tls

What steps will reproduce the bug?

Load up an HTTP2 server with SSL certificate. Perform requests, but don't complete them.

eg:

• disconnect before stream can be is read
• send bad SSL data
• disconnect between response can be written

After a while, ServerHttp2Session, nghttp2_memory, and Http2Session vastly outnumber number of Http2ServerRequest or Http2ServerResponse references.

How often does it reproduce? Is there a required condition?

I believe it's from connection reset.

What is the expected behavior?

Objects should be cleared.

What do you see instead?

I have numerous memory snapshots and about 15 production servers exhibiting this issue. I can love dump a heapshot as needed. One shows the issue clearly with 102 ServerHttp2Session objects, but only 18 Http2ServerRequest objects.

TLSSocket appears to clearly show that the references are internal. 21 out of 122 have a distance to window. The rest appear to leak.

---

After rewriting code to eliminate all stream/session listeners to avoid leaking and only using getPipeline(), I have a recent example of 40 sessions, 24 streams, and 21 req/res. The leaking ServerHttp2Session look like this:

An active req/res (with middleware) looks like this:

Additional information

I have have more snapshots and can pull them. The HttpResponse and HttpRequest objects are custom middleware. The other 16 Http2ServerRequest objects are mostly socket.io objects, but they don't appear to leak. If you need the actual files, I'm on Slack and can share there (since snapshots have private information).

[clshortfuse]

Forgot to mention, I've logged the following errors:

HTTP/2 client error [Error: 140455755077568:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:
] {
  library: 'SSL routines',
  function: 'ssl3_get_record',
  reason: 'wrong version number',
  code: 'ERR_SSL_WRONG_VERSION_NUMBER'
}
HTTP/2 client error [Error: 140455755077568:error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:../deps/openssl/openssl/ssl/statem/statem_srvr.c:1714:
] {
  library: 'SSL routines',
  function: 'tls_early_post_process_client_hello',
  reason: 'unsupported protocol',
  code: 'ERR_SSL_UNSUPPORTED_PROTOCOL'
}
HTTP client error [Error: Parse Error: Invalid method encountered] {
  bytesParsed: 0,
  code: 'HPE_INVALID_METHOD',
  reason: 'Invalid method encountered',
  rawPacket: <Buffer 00 0e 38 a6 27 6b 99 23 84 88 be 00 00 00 00 00>
}
HTTP/2 client error [Error: 140455755077568:error:1417A0C1:SSL routines:tls_post_process_client_hello:no shared cipher:../deps/openssl/openssl/ssl/statem/statem_srvr.c:2313:
] {
  library: 'SSL routines',
  function: 'tls_post_process_client_hello',
  reason: 'no shared cipher',
  code: 'ERR_SSL_NO_SHARED_CIPHER'
}
HTTP/2 client error [Error: 140455755077568:error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:../deps/openssl/openssl/ssl/statem/statem_srvr.c:1714:
] {
  library: 'SSL routines',
  function: 'tls_early_post_process_client_hello',
  reason: 'unsupported protocol',
  code: 'ERR_SSL_UNSUPPORTED_PROTOCOL'
}
HTTP/2 client error [Error: 140455755077568:error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol:../deps/openssl/openssl/ssl/statem/statem_srvr.c:1714:
] {
  library: 'SSL routines',
  function: 'tls_early_post_process_client_hello',
  reason: 'unsupported protocol',
  code: 'ERR_SSL_UNSUPPORTED_PROTOCOL'
}
HTTP/2 client error [Error: 140455755077568:error:1420918C:SSL routines:tls_early_post_process_client_hello:version too low:../deps/openssl/openssl/ssl/statem/statem_srvr.c:1714:
] {
  library: 'SSL routines',
  function: 'tls_early_post_process_client_hello',
  reason: 'version too low',
  code: 'ERR_SSL_VERSION_TOO_LOW'
}
HTTP/2 client error [Error: 140455755077568:error:142090FC:SSL routines:tls_early_post_process_client_hello:unknown protocol:../deps/openssl/openssl/ssl/statem/statem_srvr.c:1689:
] {
  library: 'SSL routines',
  function: 'tls_early_post_process_client_hello',
  reason: 'unknown protocol',
  code: 'ERR_SSL_UNKNOWN_PROTOCOL'
}

I also get general connection reset errors. I ensure the socket is destroyed, so I don't think it's that:

this.http2Server.addListener('clientError', (err, socket) => {
  // No clear way to signal back to client why a socket has failed because
  // it's unsure if it's HTTP1 or HTTP2. Just destroy and log server-side.
  if (!socket.destroyed) socket.destroy(err);
  if (err?.code === 'ECONNRESET') {
    console.warn('HTTP/2 client connection reset.');
    return;
  }
  console.error('HTTP/2 client error', err);
});

[clshortfuse]

More diving. Each lingering TLSSocket is referenced by HttpSession. The readableState is reading.

HttpSession[kState] has goawayCode and goawayLastStreamID equal to null. streams is empty. I can't tell if a stream was never set or it was removed. Either way, it seems the leaking TLSSockets are always related to ServerHttp2Session objects with no streams (either already cleaned up or never initiated).

[clshortfuse]

Pinging at a set interval seems to fix it:

/** @type {Set<import('node:http2').Http2Session>} */
const sessions = new Set();
this.http2Server.addListener('session', (session) => {
  sessions.add(session);
});

let max = 0;
let previous = 0;
setInterval(() => {
  max = Math.max(max, sessions.size);
  let removals = 0;
  const added = sessions.size - previous;
  for (const session of sessions) {
    if (session.destroyed) {
      removals++;
      sessions.delete(session);
    } else {
      session.ping(Buffer.from('pingpong'), (err) => {
        if (err) {
          console.error(err);
        }
      });
    }
  }

  const count = sessions.size;
  console.log(`HTTP2 Sessions: ${count} (+${added}/-${removals}) / ${max}`);
  previous = count;
}, 30_000);

It forces the stream to try to communicate with the client and then realize the connection has been dropped, with a ERR_HTTP2_PING_CANCEL error in the ping callback and/or ETIMEDOUT in 'sessionError'. No leaking ServerHttp2Session or TLSSocket so far. Will leave it running overnight and check again tomorrow, but it seems like a solid workaround. I'm trying on both v16.14.0 and v16.14.2

---

8 hours later and no leaks if you use ping.

[clshortfuse]

So... apparently we have to set a timeout on Http2 server objects, but not Http1. The documentation contradicts itself:

The 'timeout' event is emitted when there is no activity on the Server for a given number of milliseconds set using http2secureServer.setTimeout(). Default: 2 minutes.

https://nodejs.org/api/http2.html#event-timeout_3

And yet:

server.timeout
v13.0.0 | The default timeout changed from 120s to 0 (no timeout).
v8.4.0 | Added in: v8.4.0
Timeout in milliseconds. Default: 0 (no timeout)

https://nodejs.org/api/http2.html#servertimeout_1

Apparently it was changed v13. But this also means that you need to manually track sessions and closed them manually. This is prone to memory leak because it's odd for a user to track sessions manually. They'd likely want to track streams instead. We need a better solution to address the original issue.

#27556

[rahbari]

We have this socket leak since v16 and now in 18.7. this is a frustrating problem because node memory usage constantly increase until it crashes and the process must get restarted.
this is part of lsof output:

there are so many of this unhandled socks with http2. the same code works fluently with http.

memory usage with http after one day:

h2 after one day:

h2 after 3 days:

[ywave620]

Could you provide a code snippet to reproduce the problem?

[rahbari]

@ywave620 it happens with a simple http2 server, but for some connections not all of them. maybe as @clshortfuse mentioned:

• disconnect before stream can be is read
• send bad SSL data
• disconnect between response can be written

I tried to simulate these situations with tools like h2i but they can't do this. maybe @clshortfuse know.

[ywave620]

@rahbari
I tried the latest node, and I can't reproduce this issue, either.

[rahbari]

@ywave620 this is not a common thing, and I don't know exactly how this happens. as you can see in attached picture out of 467665 connection of a cluster worker almost 2670 (0.5%) of them are not garbage collected even though they are destroyed and not writable, but they still holds an output buffer. they are for several days ago although I have set timeout of http2 server to 2 minute. Most of them are from the similar IPs which make me think there is a problem with their client or connection that cause node not to garbage collect their sessions and streams. this leads to gradually increase in NodeJS rss till it halts.

I keep a weakref to each stream and each row is a valid one which is not garbage collected yet:

I tried to find a http2 debugger to mimic different behavior of a connection but i couldn't find one.

[ovflowd]

Hmm, this is definitely intriguing. And if true, definitely worth investigation.

cc @nodejs/http2

[mcollina]

Unfortunately, tracking down without reproduction and/or access to your production systems will be extremely hard.

[rahbari]

@mcollina is there any h2 debugger that can simulate some of these situations:

• disconnect before stream can be is read
• send bad SSL data
• disconnect between response can be written
• unnecessary CONTINUATION frames
• request bodies (DATA frames)
• send invalid frames

so i can test what cause this on our production server. I searched a lot but couldn't find one.