[release-1.28] Add tolerations support for DaemonSet pods

[brandond]

Proposed Changes

This pull request introduces a new feature that allows ServiceLB to handle Kubernetes tolerations defined in a service's annotations. The changes include:

• Implementation of the getTolerations method, which retrieves and validates tolerations from a service's annotations in JSON or YAML string format.
• Updates to the newDaemonSet method to append the retrieved tolerations to the DaemonSet's pod tolerations.

These changes enable users to customize tolerations directly through service annotations, providing greater flexibility in pod scheduling.

Types of Changes

• New Feature

Verification

To verify the changes:

1. Create a Kubernetes Service with the LoadBalancer type and svccontroller.k3s.cattle.io/tolerations annotation containing valid JSON or YAML formatted tolerations list.
2. Deploy the service and ensure that the corresponding ServiceLB DaemonSet's pods include the specified tolerations.
3. Verify that the system handles invalid or malformed tolerations gracefully, emitting appropriate error messages.

yaml example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    svccontroller.k3s.cattle.io/tolerations: |
      - effect: NoSchedule
        key: public
        operator: Exists
      - effect: NoExecute
        key: disk
        operator: Equal
        value: ssd

  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: ingress-nginx

json example:

apiVersion: v1
kind: Service
metadata:
  annotations:
    svccontroller.k3s.cattle.io/tolerations: |
      [
        {
          "effect": "NoSchedule",
          "key": "public",
          "operator": "Exists"
        },
        {
          "effect": "NoExecute",
          "key": "disk",
          "operator": "Equal",
          "value": "ssd"
        }
      ]

  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app.kubernetes.io/name: ingress-nginx

Testing

Linked Issues

• [Release-1.28] - Customize servicelb daemonset tolerations (traefik) #10700

User-Facing Change

- **New Feature**: Users can now define Kubernetes tolerations for ServiceLB DaemonSet directly in the `svccontroller.k3s.cattle.io/tolerations` annotation on services. 

Further Comments

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 18 lines in your changes missing coverage. Please review.

Project coverage is 36.17%. Comparing base (815efbf) to head (9cb0424).

Files	Patch %	Lines
pkg/cloudprovider/servicelb.go	25.00%	17 Missing and 1 partial ⚠️

Additional details and impacted files

@@               Coverage Diff                @@
##           release-1.28   #10705      +/-   ##
================================================
- Coverage         40.65%   36.17%   -4.49%     
================================================
  Files               179      162      -17     
  Lines             18488    14232    -4256     
================================================
- Hits               7516     5148    -2368     
+ Misses             9691     8087    -1604     
+ Partials           1281      997     -284     

Flag	Coverage Δ
e2etests	36.17% <25.00%> (-9.39%)	⬇️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.