Support Antrea as primary CNI in EKS

[jianjuns]

Describe the problem/challenge you have
As of now, Antrea supports only networkPolicyOnly mode in EKS, where EKS native CNI takes care of IPAM and L3 connectivity with VPC native networking, and Antrea only enforces K8s NetworkPolicies. This way has advantages of leveraging VPC networking for connectivity, but has limits on number of Pod IPs per Node. So, it would be good to provide an option to run Antrea as the primary CNI in EKS to address the per Node IP limit issue.

Describe the solution you'd like
Antrea relies on K8s NodeIPAM for IPAM, but it is not enabled in EKS. So, we either need EKS to support enabling NodeIPAM for a cluster, or Antrea might try enabling NodeIPAM after deployment. For the latter, one idea is to restart K8s ControllerManager and enable NodeIPAM by the Antrea initContainer.

[shapirov103]

@jianjuns Any updates on this? Maybe specific benefits that could open up in this mode, such as for example multi-cast support within the network (e.g with overlay).

[jianjuns]

@shapirov103 : unfortunately not much progress yet. You know the key is an IPAM solution, as EKS cannot enable K8s NodeIPAM. We looked at the easiest solution of leverage K8s NodeIPAMController code to allocate a single subnet per Node, but we like to see if we cover other IPAM use cases (like allocating small subnets on demand, subnets per Namespace) with a single solution, and so did not proceed on that route.
If enabing overlay mode is a critical use case, we can definitely prioritize, and maybe reconsider the short term solution of leveraging K8s NodeIPAMController code.
@ksamoray @yasensim

[jianjuns]

@shapirov103 : since you mentioned overlay, we like to understand do you expect the traffic are encapsulated across Nodes? I am asking because there is another possibility that Antrea does not encapsulate the traffic, but let VPC router route the traffic across Nodes. We call this mode noEncap mode in Antrea, and it requires to add the Nodes' Pod CIDRs to the VPC routing table (by K8s AWS Cloud Provider).
Both encap and noEncap modes use Pod CIDRs not in VPC subnets, but one major difference is that - with noEncap mode, the Pod CIDR cannot overlap with VPC subnets, or CIDRs of other clusters in the same VPC.

@antoninbas made a good point for encap mode, that in the EKS case, K8s masters are not worker Nodes, and so it will not join the Pod overlay network, and thus cannot reach Pods in the overlay network directly, but master to Pod connectivity is required by K8s. If we want to encapsulation, then we probably need EKS to provide extra support to enable the connectivity from master to Pod network. In contrast, if we do noEncap mode, we just need to enable the RouteController function of AWS CloudProvider.

Hope my question is clear to you. I definitely like to discuss with you more on this topic in our meeting after New Year.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment, or this will be closed in 180 days