AzureStackHCIVirtualMachine Deletion Bug fix, tests, mocks and fixes for issues including linting, codeql, tsa, etc

[PavanNeerudu]

AzureStackHCIVirtualMachine Deletion Bug fix, tests, mocks and fixes for issues including linting, codeql, tsa, etc

The initial goal of the PR was to fix an issue in the reconcile deletion loop for azurestackhcivirtualmachine(controllers/azurestackhcimachine_controller.go). The deletion of azurestackhcivirtualmachine was being handled in a bit wrong way where the deltion of this VM was assumed as a synchronous operation which is not true. This is because, when we issue a delete to a custom resource, it has to go through the logic in the reconciliation loop for this specific resource and post that the resource is deleted. This PR updates the implementation by doing the following:

• If the Get call for the VM results in an err saying that the vm is not present, then we are good(the machine is already deleted).
• If the Get call for the VM results in other errors, the request is reconciled immediately(a non nil error is send)
• If we are able to get the machine and the deletion timestamp on the machine is not zero(the delete was issue, but the deletion did not complete), then the resource is requeued after a short amount of time. However, if the deletion timestamp is not nil, we go ahead and issue delete for this resource and requeue post successful operation.

Along, with fixing this bug, this PR does the following:

• Adds initial testing framework with gingko based tests and mocks.
• Adds ginkgo based tests covering different scenarios and edge cases for the PR change specific funtion(reconcileVirtualMachineDelete) by using a mock client generated using mockgen for controller-runtime's client.
  
• Fixes linting issues(some bugs in the code were found and addressed) by calling lint target in the test and build recipes. Removes the unused cloudtest package. Skipped linting using markers when not sure of the dependencies downstream.
• Fixes kustomization.yaml by replacing the deprecated feature(patchesStrategicMerge). kustomize edit fix was recursively run against the kustomization.yaml files.
• Removes external docker image references for the image used in the Dockerfile(replaced with cbl mariner's distroless minimal image), kube-rbac-proxy, etc.
  
• Fixes CodeQL and TSA issues failing on the master.
  
• Removes the unused versions v1alpha3 and v1alpha4 from the controller-gen calls and updates these CRDs.
• Enhancements to make recipes of test and manager builder by calling linting, fmt, vet, module cleaning, etc.
• Pipeline build stage enhancements by calling the test recipe as well as a stage to make sure the go generation doesn't generate any excess files/changes than the ones checked in.

[pradipd]

This is great, but, can we break this into 2 pr's?

1. the fix.
2. the rest of the test framework

If we need to cherry pick the fix for this PR, it would be very challenging with all the test fraweowkr stuff.

[PavanNeerudu]

This is great, but, can we break this into 2 pr's?

1. the fix.
2. the rest of the test framework

If we need to cherry pick the fix for this PR, it would be very challenging with all the test fraweowkr stuff.

Hi @pradipd . The changes made are kind of reliant on each other(other than things like kustomization, crds, tsa, etc). Do you think once the changes in this PR are approved, can we raise a PR to the release branch with the changes that are actually needed for the fix(just the azurestackhcimachine controller changes).

[zawachte]

just a few small comments