[Release-1.28] - Customize servicelb daemonset tolerations (traefik)

[brandond]

Backport fix for Customize servicelb daemonset tolerations (traefik)

• Customize servicelb daemonset tolerations (traefik) #1988

[mdrahman-suse]

Validated on RC v1.28.13-rc1+k3s1

Testing

Steps: #10698 (comment)

Validation

$ k3s -v
k3s version v1.28.13-rc1+k3s1 (47737e1c)
go version go1.22.5

• Observed custom tolerations

$ k get daemonset -A
NAMESPACE     NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   svclb-nginx-loadbalancer-svc-45ae3a5e   1         1         1       1            1           <none>          2m7s
kube-system   svclb-traefik-ab8352b6                  1         1         1       1            1           <none>          9m25s

$ k describe -n kube-system pod/svclb-nginx-loadbalancer-svc-45ae3a5e-x6wcg
Name:                 svclb-nginx-loadbalancer-svc-45ae3a5e-x6wcg
Namespace:            kube-system
...
Tolerations:        CriticalAddonsOnly op=Exists
                    disk=ssd:NoExecute
                    node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                    node-role.kubernetes.io/master:NoSchedule op=Exists
                    node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                    node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                    node.kubernetes.io/not-ready:NoExecute op=Exists
                    node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                    node.kubernetes.io/unreachable:NoExecute op=Exists
                    node.kubernetes.io/unschedulable:NoSchedule op=Exists
                    public:NoSchedule op=Exists
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m34s  default-scheduler  Successfully assigned kube-system/svclb-nginx-loadbalancer-svc-45ae3a5e-x6wcg to server1