Network rules with this approach?

[lilida]

Thanks for sharing the great work. I wonder how network rules are applied in this mode. Are you still using security group or a separate mechanism has been developed?

[paulnivin]

We're use AWS security groups. As part of the k8s deployment, we keep the control plane separate from the rest of the cluster, by keeping the pods on separate subnets w/ security groups. We don't have an immediate use case for implementing Kubernetes NetworkPolicy controls in our environment, but I'd be interested in adding support to cni-ipvlan-vpc-k8s.

[lilida]

Thanks for the response. Can you clarify a bit on the subnet and security group management here? Are you grouping pods with the same security groups into one ENI? If so, how do you manage the schedule decision to avoid the case assigning a pod to a node which has max number of ENIs allocated but already set to different security groups?

[paulnivin]

For a given Kubernetes cluster, all pods use the same security groups -- we don't make scheduling decisions with respect to which Pod an ENI will land on. We currently enforce network controls between Lyft services at layer 7, although we're likely to implement NetworkPolicy support in the near future.

[tasdikrahman]

Thanks a lot for open sourcing this, just had a small question from your last comment @paulnivin on the feature addition of supporting NetworkPolicies. Was curious about the status of it :)

[paulnivin]

@tasdikrahman NetworkPolicy support should land soon -- it's a requirement for some other work at Lyft that's shipping soon. Best current estimate is early May.

[xdrus]

@paulnivin That's a great plugin, thanks for open sourcing.
From the above I assume that there is no way to schedule pod in specific subnet atm? In our case we would like to have a few different networks "zones" (map them to subnets -> hence ENIs) and run specific workload in a specific "zone"/subnet.
And +1 for network policies. We want use calico as a network policy engine, have you had any progress on this?

[lbernail]

@xdrus for this use case you could create separate node pools with different CNI configuration (subnet tags in particular) and use kubernetes taints and labels to control which applications go to which pool.

An alternative would be to select the subnet based on pod annotations but the plugin does not support that today

[xdrus]

Thanks @lbernail that is exactly what we do now with amazon vpc cni plugin, but the alternative approach would allow us to increase density/utilization of nodes.