-
Notifications
You must be signed in to change notification settings - Fork 84
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
✨ new operational mode of Direct Ingress Pod access This adds a new operational mode of adding the Ingress pods as Targets in the Target Groups directly, instead of the nodes where the Ingress is accessed through a node port. The core idea is based on the fact that standard AWS EKS cluster running AWS VPC CNI have their pods as first class members in the VPCs. Their IPs are directly reachable from the ALB/NLB target groups like the nodes are too, which means there is no necessity for the NodePort extra hop to take. There are several drivers and advantages accessing the pod directly vs a Nodeport: ## Delay and eventual consistency of AutoscalingGroup to ALB/NLB target group updates This has been the biggest trouble in operations, that the list of nodes in target groups is slower than the nodes are replaced which ends up in a black hole of no Ingresses available at a time. We are facing regularily downtimes especially when spot interruptions or ASG node rollings happen that the ALB/NLB takes up to 2 minutes to reflect the group change. For smaller clusters this leads to no Skipper instance being registered hence no target available to forward traffic. With this new mode the registration happens independently and instantly, the registration of targets up to be serving takes less than 10seconds! ## independent scaling of nodes With nodeports there is an eventual dependency on available nodes to scale the Ingress. Plus the Ingress pod cannot be replaced in place but requires a termination first and then rescheduling. For a certain time which can be more than a minute, this node is offline as an Ingress. With this mode the host networking and node port is obsolete, which allows node indepent scaling of Skipper pods! Skipper becomes a regular deployment and its replicaSet can be indepent on the cluster size. ## Save de/registering and instantaneous response Core idea is the event based registration to Kubernetes using pod `Informer` that receives immediate notifications about pod changes, which allow almost zero delayed updates on the target groups. The registration happens as soon as the pod received an IP from AWS VPC. Hence the readiness probe of the ALB/NLB starts to monitor already during scheduling of the pod, serving the earliest possible. Tests in lab show pods serving ALB traffic well under 10s from scheduling! Deregistration happens bound to the kubernetes event. That means the LB is now in sync with the cluster and will stop sending traffic before the pod is actually terminated. This implement save deregistering without traffic loss. ## TG without unhealthy targets Since the IP based TGs are managed now by this controller, they represent pods and thus all of them are shown healthy, otherwise removed by this controller. # Implementation details: * client-go Informer: This high level functions are providing a convenient access to event registrations of kubernetes. Since the event registration is the key of fast response and efficient compared to high rate polling, using this standard factory methods seems standing to reason. ## Todo Settle on the impelemtation agreement and finish it up by adding unit tests. ## Tests * successful transistion of TG from type Instance to type IP vice versa * the controller registers pods that are discovered * the controller deregisters pods that are "terminating" status * the controller recovers desired state if manual intervention on the TG happened by "resyncing" * it removes pods that are killed or dead ## misc * extended the service account with required RBAC permissions to watch/list pods * added example of Skipper without a HostNetwork and NodePort
- Loading branch information
1 parent
5c66137
commit 664cd10
Showing
19 changed files
with
1,380 additions
and
25 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.