EKS Feature Request: Allow customer to choose Calico CNI

[trimbleAdam]

Tell us about your request
As an existing Kops user, I am able to choose which CNI/overlay works best for my workloads. We require the ability to choose which CNI implementation is applied to cluster at creation time.

Particularly interested in end-to-end Calico CNI, or kubenet + Calico.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Our workloads require lots of small pods to be binpacked on EC2 minions. Currently a m5.large fits our workloads perfectly within a Kops kubernetes cluster, so the limitation of 29 pods per instance becomes a big waste of money.

Are you currently working around this issue?
80% of our workloads are remaining on Kops cluster until EKS supports our workloads.
For the other 20% we are trying to get cluster-autoscaler to take MAX_PODS into consideration during scale out/in.

Additional context
Had a great call! We look forward to making a switch to EKS.

Attachments
See case # 5756612121.

[tabern]

@trimbleAdam check out #398 and let us know what you think. We think this should solve your binpacking need.

[adaughterson]

We are looking at EKS again, and just wanted to actually put some thought in to a reply.

I read through the thread, and it looks like there is a big desire to make this very thing possible, but I also see a few "any word on this?" posts.

Still want this FR, or one which provides the equivalent net outcome (eg: pod density issues caused by using VPC secondary IP addresses mitigated by allowing the specification of CIDR).

Thanks!

Adam Daughterson

[andelhie]

I just ran in to this problem right now. The ability to have the Calico CNI native to the EKS setup is key. Setting up a secondary CIDR to then have a limits IP range per the NIC to ENI IP limit is very limiting. I know as a fact that i can fit more pod than 11 on a m4.large instance.

I am going to have a talk with my TAM and the EKS team in the coming weeks to look at this.

[kshafiee]

Hi team, any updated on this?

[jwenz723]

I would love to be able to specify to use a CNI other than the AWS VPC CNI. Would love to see some movement on this issue.

[Vitorspk]

good feature developer option to another CNI on creating EKS.

[bjethwan]

VPC IP CIDR range is a scarce resource for large orgs,
with multiple networks, both on-premise & on-cloud connected together.
(secondary cidr ranges are already in use)

aws-vpc-cni helped me greatly in one of the uses-cases, wherein I wanted to run couchbase on k8s (as described here).

But having aws-vpc-cni as the only option for EKS is limiting.

[mszumilak]

So far nobody mentioned it so it is worth adding that using calico on EKS as the only CNI makes using webhooks difficult. Control plane does not understand Calico so the only way to use webhooks is to expose them on the host network or via ingress. Both of those workarounds are awful and might be insecure.

[mikestef9]

We have no plans to add Calico as a native feature of EKS, and will close this issue. However, we will soon launch #923 which will make it much easier to install an alternate CNI like Calico in a new EKS cluster.