Support Node Affinity and Tolerations for EnvoyProxy Pods

[Gentoli]

Description:

I am trying to run a set of proxy on a restricted node pool. The pod needs affinity and tolerations set to be correctly scheduled.

I would assume it would look something like this:

apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: router-public
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 2
        pod:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: cloud.google.com/gke-nodepool
                        operator: In
                        values:
                          - router-node
          tolerations:
            - effect: NoSchedule
              key: node-type
              operator: Exists
              value: "router"

Where affinity and tolerations are propagated to the pod spec in the deployment.

In general, would it be better to allow any fields under spec.provider.pod but exclude anything eg controls? i.e. A user can add anything there and the will be merged to the deployment, but certain fields (such as containers, initContainer) will not be allowed.

Relevant Links:

• https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
• https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

[qicz]

I will support this ASSP

[arkodg]

In general, would it be better to allow any fields under spec.provider.pod but exclude anything eg controls? i.e. A user can add anything there and the will be merged to the deployment, but certain fields (such as containers, initContainer) will not be allowed.

@Gentoli this is tricky from an API language/documentation perspective because the contract between the user and Envoy Gateway is not strict and compatibility cannot be guarantied in the future, if we pursue the merging approach