Guard against possible race condition in new CNI

[slonka]

Description

TLDR:

If an application pod starts before CNI installer installs CNI, Kubelet has no knowledge of the CNI plugin. The result is that the application pod comes up without iptables rules.

The race condition is best described in the Istio design doc

Istio first solved the issue by labeling the broken pods with a specific label and then killing them (PR). However this approach causes additional churning during a node start. It also produces a number of errors in customer logs and lacks portability. So they switched to the taint controller approach implemented in this PR.

Possible solutions:

1. Reimplement the taint controller
2. At DP startup check if transparent proxy is set and fail the start if it's not.
3. Try to revive "KEP: Node Readiness Gates #1003"
4. Base this on pod healthcheck

WDYT?

[jakubdyszkiewicz]

Triage: Let's go with option 1. implemented in CP. The extra K8S RBAC (update node) should be only set when CNI is enabled.

[slonka]

I started implementing this and it seems that if we had a default probe (like testserver in the test framework has it and demo-client does not) that will not go healthy if the traffic is not redirected to Envoy that would prevent workloads from being accessible. If we couple that with having the CNI on startup getting a list of pods and injecting the rules inside those pods then we should be fine against this race condition. WDYT? decided to stick with the controller