K8s improvements

[atsikham]

See changelog for details.

What I would test here:

• All standard K8s cases including control plane scaling up
• Certs renewal
• kubeadm reset + apply
• During apply and upgrade apiserver certificate should have the same expiration days amount as other certificates
• Apiserver certificate SANs: openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout | grep DNS:

[przemyslavic]

/azp run

[przemyslavic]

/azp run

[przemyslavic]

❌ It fails when promoting the cluster to HA.

2022-01-31T11:03:38.7051497Z[38;21m11:03:38 INFO cli.engine.ansible.AnsibleCommand - TASK [kubernetes_promote : Get kube-proxy config map] **************************
2022-01-31T11:03:39.2226226Z[31;21m11:03:39 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-devprazurubucanal-kubernetes-master-vm-0]: FAILED! => {"changed": true, "cmd": "kubectl get configmap kube-proxy  --namespace kube-system  --output yaml | sed 's|server: .*$|server: https://localhost:3446|' | kubectl apply  --namespace kube-system  -f-\n", "delta": "0:00:00.115034", "end": "2022-01-31 11:03:39.185764", "msg": "non-zero return code", "rc": 1, "start": "2022-01-31 11:03:39.070730", "stderr": "Unable to connect to the server: x509: certificate is valid for ci-devprazurubucanal-kubernetes-master-vm-0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not localhost\nUnable to connect to the server: x509: certificate is valid for ci-devprazurubucanal-kubernetes-master-vm-0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not localhost", "stderr_lines": ["Unable to connect to the server: x509: certificate is valid for ci-devprazurubucanal-kubernetes-master-vm-0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not localhost", "Unable to connect to the server: x509: certificate is valid for ci-devprazurubucanal-kubernetes-master-vm-0, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not localhost"], "stdout": "", "stdout_lines": []}

To reproduce:

1. Deploy non-HA cluster (1 master + x nodes)
2. Add the following configuration:

---
kind: configuration/shared-config
title: "Shared configuration that will be visible to all roles"
name: default
specification:
  use_ha_control_plane: true
  promote_to_ha: true

3. Run apply command again

[przemyslavic]

/azp run

[przemyslavic]

Found another issue that occurred when re-applying the HA configuration with the certificate renewal option enabled:

---
kind: configuration/kubernetes-master
name: default
provider: azure
specification:
  advanced:
    certificates:
      expiration_days: 10
      renew: true

---
kind: configuration/shared-config
title: Shared configuration that will be visible to all roles
name: default
provider: azure
specification:
  use_ha_control_plane: true

2022-01-31T13:44:38.3213879Z[38;21m13:44:38 INFO cli.engine.ansible.AnsibleCommand - TASK [kubernetes_master : Generate new CSR] ************************************
2022-01-31T13:44:39.9738331Z[31;21m13:44:39 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-devazurrhelcanal-kubernetes-master-vm-1]: FAILED! => {"changed": true, "cmd": ["kubeadm", "certs", "generate-csr", "--cert-dir=csr", "--kubeconfig-dir=csr", "--config=/etc/kubeadm/kubeadm-config.yml"], "delta": "0:00:00.032666", "end": "2022-01-31 13:44:39.746734", "msg": "non-zero return code", "rc": 1, "start": "2022-01-31 13:44:39.714068", "stderr": "unable to read config from \"/etc/kubeadm/kubeadm-config.yml\" : open /etc/kubeadm/kubeadm-config.yml: no such file or directory\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["unable to read config from \"/etc/kubeadm/kubeadm-config.yml\" : open /etc/kubeadm/kubeadm-config.yml: no such file or directory", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}
2022-01-31T13:44:40.1148153Z[38;21m13:44:40 INFO cli.engine.ansible.AnsibleCommand -[0m
2022-01-31T13:44:40.1156238Z[38;21m13:44:40 INFO cli.engine.ansible.AnsibleCommand - TASK [kubernetes_master : Restore certificates] ********************************
2022-01-31T13:44:41.8368267Z[38;21m13:44:41 INFO cli.engine.ansible.AnsibleCommand - changed: [ci-devazurrhelcanal-kubernetes-master-vm-1 -> ci-devazurrhelcanal-kubernetes-master-vm-1]
2022-01-31T13:44:41.8666240Z[38;21m13:44:41 INFO cli.engine.ansible.AnsibleCommand -[0m
2022-01-31T13:44:41.8674906Z[38;21m13:44:41 INFO cli.engine.ansible.AnsibleCommand - TASK [kubernetes_master : Fail certificates generation] ************************
2022-01-31T13:44:41.9649903Z[31;21m13:44:41 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ci-devazurrhelcanal-kubernetes-master-vm-1]: FAILED! => {"changed": false, "msg": "Certificates generation failed, restored an initial state"}

[przemyslavic]

/azp run

[przemyslavic]

/azp run