Move servicelb into cloudprovider

[brandond]

We should make better use of the upstream cloud-controller-manager:

• Implement the newer, more efficient InstancesV2 interface
• Move servicelb into cloudprovider and implement the LoadBalancer interface

[ShylajaDevadiga]

Validated using k3s version v1.25.3-rc3+k3s1

Steps:

1. Install k3s with differenct config with servicelb
2. Deploy an app and expose the app using a Load Balancer service

• disable servicelb

$ cat /etc/rancher/k3s/config.yaml 
write-kubeconfig-mode: 644
cluster-init: true
disable:
  - servicelb

$ kubectl describe node | grep -i providerid
ProviderID:                   k3s://ip-172-31-7-58

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-75fc8f8fff-hd8l6                  1/1     Running     0          6m25s
kube-system   helm-install-traefik-26bt5                0/1     Completed   1          6m25s
kube-system   helm-install-traefik-crd-84lnm            0/1     Completed   0          6m25s
kube-system   local-path-provisioner-5b5579c644-blzp7   1/1     Running     0          6m25s
kube-system   metrics-server-5c8978b444-phtbb           1/1     Running     0          6m25s
kube-system   traefik-9c6dc6686-r7mtp                   1/1     Running     0          6m12s

$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
default       test-lb            LoadBalancer   10.43.97.193   <pending>     81:31952/TCP                 3s

curl http://<IP>:81
curl: (7) Failed to connect to <IP> port 81: Connection refused

Validation

1. providerid is available
2. svclb pod for traefik is not available

• disable-cloud-controller

cat /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: 644
cluster-init: true
disable-cloud-controller: true

$ kubectl describe node | grep -i providerid
$

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-75fc8f8fff-ln8cc                  1/1     Running     0          2m59s
kube-system   helm-install-traefik-crd-8wlkh            0/1     Completed   0          2m59s
kube-system   helm-install-traefik-sshm8                0/1     Completed   1          2m59s
kube-system   local-path-provisioner-5b5579c644-7t429   1/1     Running     0          2m59s
kube-system   metrics-server-5c8978b444-kvwh7           1/1     Running     0          2m59s
kube-system   svclb-traefik-c91929f5-f9m54              2/2     Running     0          2m45s
kube-system   traefik-9c6dc6686-snlgs                   1/1     Running     0          2m45s

$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default       test-lb            LoadBalancer   10.43.141.235   <REDACTED>   81:30864/TCP                 15s

$ curl http://18.219.247.109:81
test-lb-745fcc98-scx5l

Validation

1. providerid is not available
2. svclb pod for traefik is available

• disable servicelb
• disable-cloud-controller

$ cat /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: 644
cluster-init: true
disable:
  - servicelb
disable-cloud-controller: true

$ kubectl describe node | grep -i providerid
$

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-75fc8f8fff-h8xm5                  1/1     Running     0          113s
kube-system   helm-install-traefik-crd-9dbpx            0/1     Completed   0          113s
kube-system   helm-install-traefik-zwq5s                0/1     Completed   1          113s
kube-system   local-path-provisioner-5b5579c644-rd9r9   1/1     Running     0          113s
kube-system   metrics-server-5c8978b444-bwk68           1/1     Running     0          113s
kube-system   traefik-9c6dc6686-8cn5b                   1/1     Running     0          99s

$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
default       test-lb            LoadBalancer   10.43.66.7     <pending>     81:31719/TCP                 33s

curl http://<REDACTED>:81
curl: (7) Failed to connect to <REDACTED> port 81: Connection refused

Validation

1. providerid is not available
2. svclb pod for traefik is not available

• servicelb enabled(default)
• cloud-controller enabled(default)

$ cat /etc/rancher/k3s/config.yaml
write-kubeconfig-mode: 644
cluster-init: true

$ kubectl describe node | grep -i providerid
ProviderID:                   k3s://ip-172-31-15-202

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-75fc8f8fff-6sdfq                  1/1     Running     0          43s
kube-system   helm-install-traefik-crd-npmmg            0/1     Completed   0          44s
kube-system   helm-install-traefik-w7z7j                0/1     Completed   1          44s
kube-system   local-path-provisioner-5b5579c644-6mklx   1/1     Running     0          43s
kube-system   metrics-server-5c8978b444-xx2b7           1/1     Running     0          43s
kube-system   svclb-traefik-655c2315-rhrjm              2/2     Running     0          31s
kube-system   traefik-9c6dc6686-qmr7t                   1/1     Running     0          31s

$ kubectl get svc -A
NAMESPACE     NAME             TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default       test-lb            LoadBalancer   10.43.191.25   <REDACTED>   81:31218/TCP                 12s
$ curl http://3.15.218.144:81/name.html
test-lb-745fcc98-c5h97

Validation

1. providerid is available
2. svclb pod for traefik is available