feat: support envoy & ratelimit pod affinity and tolerations.

[qicz]

Fixes #1396

support envoy & ratelimit pod affinity and tolerations.

apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  namespace: envoy-gateway-system
  name: test
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        type: LoadBalancer
      envoyDeployment:
        replicas: 2
        container:
          env:
          - name: env_a
            value: env_a_value
          - name: env_b
            value: env_b_name
          image: "envoyproxy/envoy-dev:latest"
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
          securityContext:
            runAsUser: 2000
            allowPrivilegeEscalation: false
        pod:
          annotations:
            key1: val1
            key2: val2
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: cloud.google.com/gke-nodepool
                    operator: In
                    values:
                    - router-node
          tolerations:
          - effect: NoSchedule
            key: node-type
            operator: Exists
            value: "router"
          securityContext:
            runAsUser: 1000
            runAsGroup: 3000
            fsGroup: 2000
            fsGroupChangePolicy: "OnRootMismatch"

[qicz]

when #1294 merged, will append more unit tests.

[codecov]

Codecov Report

Merging #1398 (da4d1c8) into main (b90946c) will increase coverage by 0.04%.
The diff coverage is 26.66%.

@@            Coverage Diff             @@
##             main    #1398      +/-   ##
==========================================
+ Coverage   62.63%   62.67%   +0.04%     
==========================================
  Files          79       79              
  Lines       11116    11131      +15     
==========================================
+ Hits         6962     6976      +14     
- Misses       3699     3701       +2     
+ Partials      455      454       -1     

Impacted Files	Coverage Δ
api/config/v1alpha1/zz_generated.deepcopy.go	0.00% <0.00%> (ø)
...frastructure/kubernetes/proxy/resource_provider.go	87.36% <100.00%> (+0.14%)	⬆️
...tructure/kubernetes/ratelimit/resource_provider.go	98.42% <100.00%> (+0.02%)	⬆️

... and 2 files with indirect coverage changes

[qicz]

another choice about #1379 is that we can provide the API corev1.PodSpec for Deploymentwith some default settings, such as certs, and volumes. the user can override the default settings.

cc @arkodg

[zhaohuabing]

To ensure we're prepared to support additional features in K8s resource spec - such as HPA, VPA, volumes, volumeMounts and more - I suggest we establish a clear design beforehand.

In my research, I came across how Istio handles Ingress Gateway customization. Its configuration is quite complex but could be an excellent reference for our EG implementation. You can find more information on KubernetesResourcesSpec at the following link:

https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#KubernetesResourcesSpec

[arkodg]

To ensure we're prepared to support additional features in K8s resource spec - such as HPA, VPA, volumes, volumeMounts and more - I suggest we establish a clear design beforehand.

In my research, I came across how Istio handles Ingress Gateway customization. Its configuration is quite complex but could be an excellent reference for our EG implementation. You can find more information on KubernetesResourcesSpec at the following link:

https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#KubernetesResourcesSpec

hey @zhaohuabing if you are referring to overlays in Istio, im a -1 on it because as a end user, I found it extremely hard to generate the right config
regarding additional features/fields we can raise them as individual issues and debate about the pros/cons of adding it. Since we dont have a clear owner for the entire feature, we haven't enforced a design doc, just accepting/enabling k8s fields based on use cases

[zhaohuabing]

arding additional features/fields we can raise them as individual issues and de

@arkodg I mean the overall structure of the K8s resource： https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#KubernetesResourcesSpec

But I agree that we can add new fields when users ask for them. The current structure of envoyDeployment looks good to me.

[zhaohuabing]

LGTM

[qicz]

ptal @arkodg

[arkodg]

LGTM, thanks for adding this feature !