Enable configurable API Server Load Balancer

[CecileRobertMichon]

What type of PR is this?
/kind feature
/kind api-change

What this PR does / why we need it: This PR allows configuring the API Server load balancer to enable things like private clusters, eg:

  networkSpec:
    apiServerLB:
      type: Internal

It adds defaulting + validating webhooks for the new Load Balancer spec. It also allows BYO api server public IP via specifying a pre-existing IP.

Check out the docs to see what this exposes to the user: https://deploy-preview-974--kubernetes-sigs-cluster-api-provider-azure.netlify.app/topics/api-server-endpoint.html

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #486, #696, #952, #892

Special notes for your reviewer:

Still work in progress, opening to get early feedback.

Changes are currently breaking.

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squash commits
• documentation
• unit tests
• back compat for public API server
• private cluster needs management cluster in the same vnet
• BYO API Server IP delete
• make sure that private control plane VMs have outbound access

Release note:

Allow configuration of the API Server Load Balancer, including support for private API Server endpoint.
Allow BYO API Server IP

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[fejta-bot]

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/check-cla

[CecileRobertMichon]

@alexeldeib @MirzaSikander this supersedes #652, along with kubernetes-sigs/cluster-api#3715

[CecileRobertMichon]

I still need to fix unit tests and find a way to test private clusters but otherwise, most functionality should be there so this is ready for initial review.

[CecileRobertMichon]

/hold for private cluster test

@devigned @alexeldeib @nader-ziada @mboersma this is ready for initial review

[devigned]

/lgtm

/assign @alexeldeib

[nader-ziada]

🎉 great work Cecile
lgtm

[alexeldeib]

is this the hairpin issue? looks like a timeout hitting the api server with 3 master setup? didn't look too deeply I admit

/test pull-cluster-api-provider-azure-e2e

 Unexpected error:
      <*url.Error | 0xc0004d8090>: {
          Op: "Delete",
          URL: "https://capz-e2e-b3o4h8-a7437409.westus2.cloudapp.azure.com:6443/api/v1/namespaces/default/services/httpd-ilb",
          Err: {
              Op: "read",
              Net: "tcp",
              Source: {IP: [10, 60, 5, 156], Port: 44610, Zone: ""},
              Addr: {IP: [20, 69, 81, 80], Port: 6443, Zone: ""},
              Err: {Syscall: "read", Err: 0x6e},
          },
      }
      Delete https://capz-e2e-b3o4h8-a7437409.westus2.cloudapp.azure.com:6443/api/v1/namespaces/default/services/httpd-ilb: read tcp 10.60.5.156:44610->20.69.81.80:6443: read: connection timed out
  occurred

[CecileRobertMichon]

is this the hairpin issue? looks like a timeout hitting the api server with 3 master setup? didn't look too deeply I admit

no it was the HA public cluster spec that failed capz-e2e: Workload cluster creation With 3 control-plane nodes and 2 worker nodes, looks like a flake

[CecileRobertMichon]

@alexeldeib let me know when you're okay with me rebasing

[CecileRobertMichon]

Rebased + squashed

@alexeldeib @devigned @nader-ziada ready for a final look

[k8s-ci-robot]

@CecileRobertMichon: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-cluster-api-provider-azure-apidiff	fd5c6ad	link	/test pull-cluster-api-provider-azure-apidiff

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[devigned]

/lgtm

[devigned]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: devigned

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [devigned]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment