Support for multiple ControlPlaneEndpoints

[randomvariable]

User Story

As a cluster user I want to be able to access my Azure private cluster and/or CAPD cluster on MacOS after the cluster is provisioned without having to hack my /etc/hosts, set up DNS routing or hack my kubeconfig.

Detailed Description

[A clear and concise description of what you want to happen.]

The origins of this discussion originally lay within the machine load balancer proposal with @JoelSpeed . Originally, the idea of having multiple controlPlaneEndpoints was being considered for AWS as having separate internal and external load balancers can lead to cost reductions in AWS, and lower latency between nodes and the control plane. However, since the original investigation, two other use cases have emerged:

• For CAPD on MacOS, the control plane endpoint needs to be modified for the end user's kubeconfig in order to connect into the Docker machine. See also Handle Mac specific kubeconfig requirements for CAPD in clusterctl #5263

• To provide a stable endpoint, CAPZ creates a capz.io DNS zone and links it into the Azure subscription. For private clusters, this means having to make the DNS name resolvable at the client level. This might be done by hacking /etc/hosts or setting up some conditional DNS forwarding to Azure, though this is non-trivial. It would be preferable to provide an external control plane endpoint that points directly at the IP of the cluster load balancer, and have that in the downloadable kubeconfig, and have worker nodes continue to point at the internal stable endpoint - whether that's a DNS entry or an LB IP.

Anything else you would like to add:

@JoelSpeed captured a lot of the original requirements in https://hackmd.io/6nomCHkLTjaRU_d3_pnwpg

[Miscellaneous information that will assist in solving the issue.]

/kind feature

[JoelSpeed]

I don't think I have too much to add that isn't already written in the linked hackmd doc, but I think some of the highlights of the previous conversation are worth pulling out:

• This should be optional - we don't need everyone to use this as it's not strictly necessary, also to keep backwards compatibility, lets make this an optional feature
• We need options to allow switching various traffic paths between the load balancers - we expect that when using internal/external style endpoints, you would want kubelets within the cluster to talk via the internal endpoint, however, CAPI also talks to the API of the cluster for various functions (eg MHC) and as such, we will need some switch to teach CAPI whether it should use the external or internal endpoints
• When multiple endpoints are in play, the control plane controller will need to create multiple kubeconfigs for the various components within the system to consume

[vincepri]

/kind proposal
/milestone Next

[voor]

We actually default to an internal load balancer in CAPA, and can effectively forward traffic for external access; however, this UX is not without its faults. Since the kubeconfig and other methods of accessing the cluster are hardwired in CAPA to the ELB endpoint, this can be inconsistent and possibly error prone if the ELB name is lost for any reason. Providing a way to make CAPI aware of alternative DNS names would provide a great way to avoid this problem.

Also important that this is optional, since owning the DNS zone and even having dynamic DNS as an option are not always possible. Previously we had brought attention to External DNS which is also a Kubernetes SIG project.

[AverageMarcus]

To add another use case: when using CAPA with non-managed clusters we'd like the option to be able to provision both private and internet facing ELBs for the API endpoint. From what I understand this is currently possible when using CAPA with managed clusters (EKS) via the endpointAccess property which instructs EKS which endpoints to create.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[fabriziopandini]

/lifecycle frozen

[sbueringer]

@yastij to bring it up in the next office hours

[AverageMarcus]

@sbueringer did this get discussed in the office hours? I took a look at the meeting notes but couldn't see anything that looked relevant.

[sbueringer]

Not that I'm aware of

[fabriziopandini]

/triage accepted
/help

[k8s-ci-robot]

@fabriziopandini:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/triage accepted
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[nrb]

I've talked with @JoelSpeed, and I will be taking this over going forward. I'll need to get up to speed with the prior work, but hope to get this progressing again.

/assign

[vincepri]

One of the use cases that should be considered is what was also discussed as part of #8500 (comment)

[fabriziopandini]

/priority backlog