Client should expose a mechanism to close underlying TCP connections

[jfoy]

Re kubernetes/kubernetes#52176, kubernetes/kubernetes#56720, #342

(Short form: a stalled TCP connection to apiserver from kubelet or kube-proxy can cause ~15 minutes of disruption across a substantial number of nodes until the local kernel closes the socket.)

I believe we need a mechanism for requestCanceler.CancelRequest to invoke transport.CloseIdleConnections based on config. It looks like we could do this with a small http.RoundTripper built to purpose. I hope to submit a PR with this change shortly.

I'm not sure if that behavior should be activated on a case-by-case basis using config.WrapTransport (less invasive, narrower change) or if it should be part of the core config used in transport.HTTPWrappersForConfig (needed in multiple use cases per issues listed above). What's the convention here?

[lavalamp]

Is this failure mode unique to HTTP/2 connections?

[lavalamp]

Is the problem that the timeout is wrong, or that the client doesn't notice when connections have been half-closed?

[jfoy]

I can't tell if it's limited to HTTP/2, but it looks to me like clients maintain persistent TCP connections no matter the application protocol, so they would be hit by the same failure.

It's the latter problem - as far as we can tell, the load balancer is hanging up on decommissioned IP addresses without sending a FIN packet (or we never receive it), so the client thinks the connection is still open. The HTTP timeout is set correctly, but opening a new session appears to reuse the existing half-closed connection.

[stephbu]

Same issue happens also with NLB in AWS.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[warmchang]

/remove-lifecycle stale

[xh3b4sd]

We have situations in which pods using the client have to be restarted in order to work around this problem. What can we do to get some movement here?

[liggitt]

What can we do to get some movement here?

There is sufficient plumbing in the rest.Config to wrap the transport or set up a connection-tracking dialer yourself today, while this is considered as a built-in feature

[r7vme]

There is sufficient plumbing in the rest.Config to wrap the transport or set up a connection-tracking dialer yourself today, while this is considered as a built-in feature

Right, i experimented while ago with own Transport. Unfortunately problem is deeper on TCP level (at least for us). In Transport i can control via dialer some TCP keepalive settings (this was my initial idea for a fix), but this does not help. At least because by default transport TCP keepalive settings are already quite good in Go.

It turned out that our problem was/is with incorrect load-balancer settings. Following happens for us:

1. endpoint (k8s api) hardly killed (e.g. kill -9 on qemu process)
2. load balancer detects that backend died after some timeout
3. load balancer does not reset existing connection to the client and keeps answering heartbeats
4. on client we see connection stays ~15 min in ESTABLISHED state

What helps is to set quite low timeouts on load balancers.