Controller ends up on fargate nodes, fails to start

[parviste-fortum]

/kind bug

What happened?

If we first run a pod on a Fargate node and let it finish, it takes a while for the node to go away. If an EBS CSI controller pod is scheduled to start during this time it ends up on the Fargate node, and fails to start with the error: Pod not supported: SchedulerName is not fargate-scheduler.

What you expected to happen?

Fargate nodes set taints to prevent non-fargate pods from being scheduled on them. By default, the conroller has tolerations which means the taint is ignored.

I would expect the controller to respect the taint set on the fargate nodes, so that the controller pods are scheduled on nodes where they can actually run.

How to reproduce it (as minimally and precisely as possible)?

1. Set up EKS with a Fargate profile matching e.g. the namespace fargate
2. Run a pod in the fargate namespace
3. Run kubectl get nodes to observe that a fargate node has been created
4. Stop the pod
5. Run kubectl get nodes to confirm that a node is still running
6. Deploy the EBS CSI controller
7. Observe that the pod fails to start due to the error mentioned above

Anything else we need to know?:

Seems related to #591, which is about the DaemonSet and tolerations in general. I think this is slightly different, because even if tolerating everything by default might(?) make sense, tolerating fargate compute nodes definitely does not, because it can't possibly work. Thus, at the very least tolerations should be set for everything except Fargate nodes.

#526 allows tolerations to be configured in the Helm chart. While this is one way to fix the issue, it does not really make sense that the default configuration (and only configuration if we deploy using kustomize?) is broken when fargate nodes are running.

Environment

• Kubernetes version (use kubectl version): v1.17.12-eks-7684af
• Driver version: 7278cef

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ayberk]

/remove-lifecycle stale

I know we're not running the daemonset on fargate, I guess we missed the controller.