AKS cluster autoscaling tests

[mkyc]

Prepare manual load tests to ensure that cluster scales up and down as expected. Load tests will be included later into epiphany code but not at that moment.

Is your feature request related to a problem? Please describe.
AKS clusters we will provide in epiphany need to have horizontal autoscaling enabled and tested.

Describe the solution you'd like
There should be some Kubernetes Deployment defined with images applying some high load on cluster. With this deployment we should be able to scale number of replicas up and down, and check if vertical autoscaler is responding accordingly and scales nodes up and down.

Deployment and description of commands should be committed to sub-directory with terraform scripts of AKS cluster. There is no need to integrate it with any test pipeline at this point.

Describe alternatives you've considered
Unknown.

Additional context
No.

[mkyc]

This task in fact is extremely similar to #1443 and should be synchronised with that one.

[rpudlowski93]

Autoscaling tests done. In order to reproduce manual load test execute the following points:

1. Create cluster AKS with enabled option: enable_auto_scaling = true with min and max count of nodes and some custom parameters.
2. Apply deployment with simple php-apache image:
   kubectl apply -f https://k8s.io/examples/application/php-apache.yaml
3. Create horizontal pod autoscale based on CPU/Memory or some other from eg. prometheus metrics. Server metrics is installed by default in AKS version higher then 1.10 :
   kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=50
4. Create deployment with a containers in order to increase load on nodes using infinite loop of queries to the php-apache service created in previous point.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: load-generator
spec:
  selector:
    matchLabels:
      run: load-generator
  replicas: 1
  template:
    metadata:
      labels:
        run: load-generator
    spec:
      containers:
      - name: load-generator
        image: busybox
        args:
        - /bin/sh
        - "-c"
        - "while true; do wget -q -O- http://php-apache; done" 

5. Increase load as needed:
   kubectl scale deployment load-generator --replicas=5
6. Check the effect using monitoring tools:
   kubectl get hpa
   or Azure Insights:
   

Summing up: after some tests and observations, autoscaling works fine and after applying high load the cluster scale up and down as expected. By default scale up works very fast and new nodes are created immediately if scale condition is true but by default scale down will be executed after 10 min if scale condition is true (cpu load = 50% in this case).

[przemyslavic]

Autoscaling has been tested. The number of pods and nodes varied depending on the load.