[EKS] [request]: Security Groups per Pod

[mike-stewart]

Tell us about your request

It would be really helpful to be able to assign Security Groups at the pod level, such that other pods on the same node don't share the same SG.

Related issue in the CNI repo: aws/amazon-vpc-cni-k8s#208

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Although Network Policies with Calico and friends can provide network segmentation within the cluster, we're missing a straightforward way to control traffic to AWS resources outside of the cluster. For example, I would like to be able to restrict access to RDS instances by referencing the pod SG in the RDS SG.

Authentication (e.g. using IAM roles) can be used in most cases to provide a layer of security, but most compliance requirements also mandate network segmentation or firewall rules as an additional defence in depth step.

Are you currently working around this issue?
My understanding is that the only workaround is to use taints/nodeAffinity to restrict pods to nodes with the right security groups, but this will result in nodes being under scheduled in a multi-tenant cluster. This is less than ideal, as it negates much of the cost benefit of moving a workload into Kubernetes.

Additional context

Attachments

[ahrkrak]

This is possible today with Tigera Secure Cloud Edition - it delivers exactly this functionality.
On Marketplace here: https://aws.amazon.com/marketplace/pp/B07FT3NTWX
Free trial link: https://tigera.io/free-ce

[phivid]

The question is: "Why paying for something that should be a native/integrated solution ?" 😊

[tabern]

We've scoped this as part of our next generation CNI plugin. Take a look and let us know what you think on #398.

Will be leaving this issue open for now as this will be a component of the CNI but not the entire project.

[dudicoco]

@tabern if this will be implemented via annotations, I suggest using the security group name/tag in the annotation rather than the sg ID to remove some hardcoded dependencies.

[minzhang28]

@mike-stewart what is the update on this ticket, does AWS has a timeline for it?

[tabern]

@minzhang28 this is part of our next generation CNI plugin. We are currently working on this.

[phivid]

@tabern, why this request has still "under consideration" label and not "proposed" or simply merged in #398 ?

[mikestef9]

Looking for some more feedback on requirements here.

One of the main issues we are addressing in the next CNI #398 is limited pod density on smaller instance types. Is there any need for security group per pod & density on the same nodes?

For reference, you can view section titled "Supported Amazon EC2 Instance Types" on this ECS doc page and look at task limit per instance to get an idea of how many pods per instance will be supported if you choose the security group per pod networking mode.

[hendrikhalkow]

I think security groups per pod should be available regardless of the node size as I’d like to avoid redesigning my workload because I changed the infrastructure. Not having this on small nodes would also cause additional headaches when spot instances are used because you need to exclude these small machines when using security groups per pod.

Higher pod density is always nice but for me not as important as the first thing. @mikestef9 your comment sounds like these two things might be mutually exclusive?

Can’t you increase the IP address limit per ENI?