Add Autoscaler to CSE

[adambarreiro]

Overview

This PR automates the steps to deploy an Autoscaler into a Kubernetes cluster stated in this document:
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/docs/vmw-whitepaper-cluster-auto-scaler.pdf

This procedure affects cluster creation and cluster update operations.

All the logic is implemented in vmware/go-vcloud-director#678

Description

The Go SDK implementation now adds extra fields to work with the Autoscaler in the worker pools.
Hence, the worker_pool block has two new fields, autoscaler_max_replicas and autoscaler_min_replicas, that when set, they deploy the Autoscaler to the cluster to start the automatic adjustment of its nodes.

When autoscaler_max_replicas and autoscaler_min_replicas are set, machine_count must be either unset or 0, and viceversa: when machine_count is set, autoscaler_max_replicas and autoscaler_min_replicas must be 0.

As it can be seen in the Go SDK implementation, once the autoscaler is deployed to the cluster, it is not removed. It's just scaled to 0 or 1 (depending on the mentioned arguments).

Tests

Updated TestAccVcdCseKubernetesCluster (the modifications check Autoscaler updates) and createdTestAccVcdCseKubernetesClusterCreationWithAutoscaler (this one checks cluster creation with Autoscaler). Both check that the autoscaler is created and the information is retrieved (set in the structure) correctly.

TEST_VCD_CSE=1 go test -tags cse -run '^Test_Cse' -v -timeout 0

Anyway, tests only check that the YAML definition is correct. One should check that the autoscaler works, manually:

kubectl --kubeconfig=kubeconfig-n kube-system describe deploy cluster-autoscaler
kubectl --kubeconfig=kubeconfig-n kube-system get po | grep autoscaler
kubectl --kubeconfig=kubeconfig -n kube-system logs cluster-autoscaler-xxxxxxxx-xxxx -f

[Didainius]

Running test TestAccVcdCseKubernetesClusterCreationWithAutoscaler

[Didainius]

Test passed. The only question I have is:

Removal process will remove the CSE cluster that might come out with default envs.

Start leftovers removal
[vcd_nsxt_alb_controller] cse_alb_controller (keep)
[vcd_provider_vdc] pvdc-2024-06-05-13-36-12.485 (keep)
[vcd_provider_vdc] nsxTPvdc1 (keep)
[vcd_network_pool] NSX-T Overlay 1 (keep)
[vcd_network_pool] unicastNetworkPoolName (keep)
[vcd_org] System (keep)
[vcd_org] tenant_org (keep)
        [vcd_org_vdc] tenant_vdc (keep)
                [vcd_vapp] test1 (DELETE)
                 REMOVING vApp tenant_vdc/test1
[vcd_org] solutions_org (keep)
        [vcd_org_vdc] solutions_vdc (keep)
                [vcd_rde] test1 (DELETE)
                 REMOVING RDE test1 WITH TYPE urn:vcloud:type:vmware:capvcdCluster:1.3.0
ok      github.com/vmware/terraform-provider-vcd/v3/vcd 3241.561s