Custom data model fix

[BrennenMM7]

What type of PR is this?

What this PR does / why we need it:
When Bootstrap token refreshes the VMSS needs to have the latest model in order for autoscaling machinepools to take place succesfully. Stores a hash of the current custom data contents into AzureMachinePool.Status.CustomDataHash. This allows to update the VMSS data if the custom data changes.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2803

Special notes for your reviewer:
This is utilizing the same logic as created in this pull request https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/2803, the only changes are as follows:

• Added to the mock tests to past all unit testing,
• Removed logic from node surge to prevent constant node rolling updates.
• Added the watcher to the Kubeadmconfig resource as mentioned in the issue/PR linked.
• API resource conversions for v1alpha3 and v1alpha4.
  Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

- AzureMachinePool Controller will no watch KubeadmConfig to ensure AzureMachinePool Bootstrap data is updated on the VMSS
- The patch loop for the AzureMachinePool will now compare CustomData field to ensure the VMSS instance is updated with the latest model 

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: BrennenMM7 / name: Brennen Murray (8c67bbb)

[k8s-ci-robot]

Welcome @BrennenMM7!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-azure 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-azure has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @BrennenMM7. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[willie-yao]

/ok-to-test

[CecileRobertMichon]

Should we consider only doing this when the MachinePool has the externally managed replicas annotation? I'd be worried about increasing the number of API calls for users that don't need this as this will now make a PUT API call per AzureMachinePool every 10 minutes or so (or whenever token is refreshed, I forget the exact number) which might not scale well for very large clusters with hundreds of MachinePools (granted this will be a problem for users with externally managed replicas too but if we can at least reduce the blast radius that would be an improvement).

[BrennenMM7]

Should we consider only doing this when the MachinePool has the externally managed replicas annotation? I'd be worried about increasing the number of API calls for users that don't need this as this will now make a PUT API call per AzureMachinePool every 10 minutes or so (or whenever token is refreshed, I forget the exact number) which might not scale well for very large clusters with hundreds of MachinePools (granted this will be a problem for users with externally managed replicas too but if we can at least reduce the blast radius that would be an improvement).

I've added an if block around the return escape if nothing has changed, allowing to have two different behaviors for patching the VMSS object based on externally managed autoscaler annotations.

This should limit the number of API calls for users not utilizing an autoscaler

[CecileRobertMichon]

/retest

[CecileRobertMichon]

@BrennenMM7 build is failing, PTAL

[BrennenMM7]

/retest

[BrennenMM7]

@CecileRobertMichon Fixed the build failures and squashed commits again, should be good to go. 👍

[CecileRobertMichon]

lgtm aside from comment on annotation const and failing unit test job, thanks again and sorry for all the back and forth on this one

[CecileRobertMichon]

/hold cancel

please squash!

[BrennenMM7]

/test pull-cluster-api-provider-azure-e2e

[BrennenMM7]

Can I get a tag on this PR please 😄

[nawazkh]

I don't fully understand the problem, will go back try reproducing the error.
However, I had some doubts and questions jotted down below after reviewing the code.

[CecileRobertMichon]

/lgtm
/hold

@nawazkh I think I've addressed all your questions/comments, let me know if you're good to remove the hold

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 6e7cb98d401329488f5e99e47bdfcf68e0e0bbd3

[nawazkh]

Thanks for the updates @CecileRobertMichon ! Good to remove the hold!
Kudos @BrennenMM7 🎉
/lgtm

[CecileRobertMichon]

/hold cancel

[k8s-infra-cherrypick-robot]

@mboersma: #3134 failed to apply on top of branch "release-1.6":

Applying: Resolved refresh custom data when Bootstrap token rotates.
Using index info to reconstruct a base tree...
M	azure/const.go
M	azure/scope/machinepool.go
M	azure/services/scalesets/mock_scalesets/scalesets_mock.go
M	azure/services/scalesets/scalesets.go
M	azure/services/scalesets/scalesets_test.go
M	exp/controllers/helpers.go
M	main.go
Falling back to patching base and 3-way merge...
Auto-merging main.go
Auto-merging exp/controllers/helpers.go
Auto-merging azure/services/scalesets/scalesets_test.go
CONFLICT (content): Merge conflict in azure/services/scalesets/scalesets_test.go
Auto-merging azure/services/scalesets/scalesets.go
CONFLICT (content): Merge conflict in azure/services/scalesets/scalesets.go
Auto-merging azure/services/scalesets/mock_scalesets/scalesets_mock.go
Auto-merging azure/scope/machinepool.go
CONFLICT (content): Merge conflict in azure/scope/machinepool.go
Auto-merging azure/const.go
CONFLICT (content): Merge conflict in azure/const.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Resolved refresh custom data when Bootstrap token rotates.
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CecileRobertMichon]

/cherry-pick release-1.7

[k8s-infra-cherrypick-robot]

@CecileRobertMichon: new pull request created: #3205

In response to this:

/cherry-pick release-1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jackfrancis]

@BrennenMM7 did this PR fix #2683 as well?

Thank you!

cc @primeroz

[BrennenMM7]

@jackfrancis Yes this should resolve #2683 as the watcher will ensure that the VMSS always has the latest bootstrap token inside the model of the VMSS. 👍

[jackfrancis]

@BrennenMM7 thank you! I'll close #2683 and advise that the original reporter of that issue validate the fix with v1.8.0 after it is released (next week), because this PR will be included in that release.

Thanks again for the fix!