Available ENIs left dangling after node termination

[krzysztof-bronk]

Hello,

I have encountered an issue with aws-cni 1.5.1(+?), where, even in a single node test cluster, if you terminate the node so that the ASG kicks in a replacement, the terminated instance ENI switches back to Available, holding IPs, and is seemingly never deleted.

Eventually one will exhaust the IP pool and pods will fail to be created.

This is a bit surprising as node recycling is the basis of autoscaling groups.
Is there some cleanup mechanism I am not aware of? Or is it a bug?

regards,
Krzysztof Bronk

[mogren]

Thanks @krzysztof-bronk for reporting, we will have to take a look why this is happening. ENIs attached to the terminated node should be freed by EC2 automatically.

[krzysztof-bronk]

Thank you for acknowledging this. If your cluster has a high node churn, or the IP pool is small, this can quickly become an issue. The current workaround, and also an independent report for the issue can be found here: #59 (comment)

[vipulsabhaya]

@krzysztof-bronk Do you happen to have the ipamd logs from one such instance?

[caiconkhicon]

ENIs attached to the terminated node should be freed by EC2 automatically.

No, it's not true. Only the main (eth0) ENI is cleaned up. The additional ones are not cleaned up, just detached and become available.

[krzysztof-bronk]

Here's the situation:

Fresh test cluster, nothing fancy or custom (except external SNAT but I tested both and it's not relevant), a single m5.xlarge worker node.

Private IPs: 10.250.9.56 (eth0), 10.250.11.217 (eth1)
Secondary Private IPs: (a whole bunch of them)

ENI: 3 total
2 In-use (as confirmed by the Private IPs above).
Interestingly, only 1 of them has a Description (aws-K8S-i-0da187e6c45d9d4d5), the other one has an empty Description.
1 Available

aws-node logs are empty (even though they're in DEBUG mode and I've deployed a test nginx container):

===== Starting installing AWS-CNI =========
===== Starting amazon-k8s-agent ===========

Node terminated. ASG spun up a new one.

Private IPs: 10.250.19.53 (eth0), 10.250.16.192 (eth1)
Secondary Private IPs: (a whole bunch of them)

ENI: FOUR total
2 In-use (as confirmed by the Private IPs above).
2 Available - the 1 Available from the earlier instance is still there

aws-node logs (with the successfully running nginx container):

===== Starting installing AWS-CNI =========
===== Starting amazon-k8s-agent ===========
ERROR: logging before flag.Parse: E0912 06:35:31.629160      13 memcache.go:138] couldn't get current server API group list; will keep using cached value. (Get https://172.20.0.1:443/api?timeout=32s: dial tcp 172.20.0.1:443: i/o timeout)

So it looks like aws-cni is leaking the warmup ENIs.

[mogren]

The latest pre-release, v1.6.0-rc2, has some changes to mitigate this problem.

[Pluies]

Good to know this is getting worked on, it's an issue for us as well. 👍

We have nightly infra testing jobs that bring up a cluster (w/ Terraform), run tests, and delete the cluster, and we've noticed that the terraform destroy step fails fairly often due to ENIs being left in "available" state, which prevents the security group from being deleted.

[robin-engineml]

@Pluies We have a similar process, and the same problem. @mogren When can we look forward to a 1.6.0 release with the mitigating changes applied?