http2: proxy does not open new connections when max concurrent streams is reached

[maxbrunet]

Hello! 👋

I have an Envoy deployment (managed via Envoy Gateway) and a Go HTTP server configured with H2C, and it seems Linkerd proxy does not create new connections when it reaches the server's max concurrent streams limit (250 by default)¹. Looking at the goroutines running:

250 @ 0x4a6d2e 0x4a8519 0x4a84f9 0x4c5ed3 0x91731b 0x92403c 0x4f8e8e 0x207975e 0x207971f 0x2079c68 0x24c1d16 0x9521a9 0x24b4e4f 0x9521a9 0x24b5248 0x9521a9 0x24b8883 0x9521a9 0x24b8c15 0x9521a9 0x24b87ca 0x9521a9 0x2498c63 0x9521a9 0x24b9323 0x9521a9 0x24d794e 0x9521a9 0x24b926a 0x9521a9 0x24ba2bd 0x9521a9 0x24ba55d 0x9521a9 0x24badf4 0x9521a9 0x24ba7d4 0x9521a9 0x24bba44 0x9521a9 0x22eeaa3 0x9521a9 0x24bbedc 0x9521a9 0x1db6226 0x1db933c 0x9521a9 0x22f6b24 0x9521a9 0x22f192a 0x9521a9 0x24be63f 0x9521a9 0x2216dd7 0x9521a9 0x24c1b9e 0x9521a9 0x24c19f7 0x9521a9 0x22effed 0x9521a9 0x24bf37e 0x9521a9 0x2216b77 0x9521a9 0x2216831 0x9521a9 0x24bf7f6 0x9521a9 0x24bfa16 0x9521a9 0x1e70753 0x9521a9 0x24c0af2 0x9521a9 0x1e72470 0x9521a9 0x22f4047 0x9521a9 0x1db6d8e 0x972a0e 0x951508 0x9234d0 0x4af0a1
#	0x4a84f8	sync.runtime_notifyListWait+0x138													runtime/sema.go:606
#	0x4c5ed2	sync.(*Cond).Wait+0x72															sync/cond.go:71
#	0x91731a	net/http.(*http2pipe).Read+0xda														net/http/h2_bundle.go:3893
#	0x92403b	net/http.(*http2requestBody).Read+0x9b													net/http/h2_bundle.go:6525

Without the proxy, we can see that as many streams as needed are opened:

3268 @ 0x4a6d2e 0x4a8519 0x4a84f9 0x4c5ed3 0x91731b 0x92403c 0x4f8e8e 0x1abd7fe 0x1abd7bf 0x2079528 0x24c4716 0x9521a9 0x24b76ef 0x9521a9 0x24b7ae8 0x9521a9 0x24bb123 0x9521a9 0x24bb4b5 0x9521a9 0x24bb06a 0x9521a9 0x249ac23 0x9521a9 0x24bbbc3 0x9521a9 0x24da34e 0x9521a9 0x24bbb0a 0x9521a9 0x24bccbd 0x9521a9 0x24bcf5d 0x9521a9 0x24bd7f4 0x9521a9 0x24bd1d4 0x9521a9 0x24be444 0x9521a9 0x22f0a63 0x9521a9 0x24be8dc 0x9521a9 0x120554f 0x1206ebb 0x9521a9 0x12049a6 0x1207abc 0x9521a9 0x22f8ae4 0x9521a9 0x22f38ea 0x9521a9 0x24c103f 0x9521a9 0x22167b7 0x9521a9 0x24c459e 0x9521a9 0x24c43f7 0x9521a9 0x22f1fad 0x9521a9 0x24c1d7e 0x9521a9 0x2216557 0x9521a9 0x2216211 0x9521a9 0x24c21f6 0x9521a9 0x24c2416 0x9521a9 0x197d9b3 0x9521a9 0x24c34f2 0x9521a9 0x197f770 0x9521a9 0x22f6007 0x9521a9 0x120550e 0x972a0e 0x951508 0x9234d0 0x4af0a1
#	0x4a84f8	sync.runtime_notifyListWait+0x138													runtime/sema.go:606
#	0x4c5ed2	sync.(*Cond).Wait+0x72															sync/cond.go:71
#	0x91731a	net/http.(*http2pipe).Read+0xda														net/http/h2_bundle.go:3893
#	0x92403b	net/http.(*http2requestBody).Read+0x9b													net/http/h2_bundle.go:6525

I would expect Linkerd proxy to behave the same as Envoy²:

New connections are established anytime there is a pending request without a connection that can be dispatched to

It seems that in some cases, it can lead to the service being completely unavailable, where the client has reset all the streams and is gone, but with all the pending streams and the ones opened "busy", these resets are never passed to the server, until the connection is closed (e.g. when restarting Envoy). (maybe some kind of HOL blocking?)

(I have observed the same from a Go client using H2C to the Go server)

Have I missed a configuration? Is this behavior expected?

Linkerd version: edge-25.9.3

Footnotes

1. https://github.com/golang/go/blob/go1.25.4/src/net/http/h2_bundle.go#L4007 ↩

2. https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/connection_pooling#http-2 ↩

[maxbrunet]

I wonder if this could what this fix in h2 crate resolves: hyperium/h2#860

Created an issue to upgrade h2: #14908