Application controller should be more resilient to network latency or K8s API server hiccups

[jannfis]

Summary

On network links with a very high latency, or which are very slow, sometimes a remote Kubernetes API drops the connection before a certain request (e.g. fetching remote resources, list of available APIs, etc) could be completed. This currently leads to comparison errors in the application or in aborting other operations (e.g. retrieving manifests using argocd app manifests). Effectually, this can prevent a sync from happening when such a hiccup occurs within a given operation, because the errors are considered fatal.

This can be observed whenever the network link between the Argo CD control plane and the managed cluster may be a little flaky.

Motivation

Improve stability on high latency networks and don't fail operations on the first hiccup.

Proposal

On certain, non-permanent errors when accessing remote Kubernetes API endpoints, we should retry the request when it fails.