produce error if encounter ImagePullBackoff

[kdelee]

fixes: #521

This correctly has the work unit fail if the container has ImagePullBackOff error

{'DVxy89tF': {'Detail': 'Error creating pod: container failed to start, ImagePullBackOff',
              'ExtraData': {'Command': '',
                            'Image': '',
                            'KubeConfig': '',
                            'KubeNamespace': 'controller-integration-1641911565',
                            'KubePod': '',
                            'Params': '',
                            'PodName': 'kube-6tw5v'},
              'State': 3,
              'StateName': 'Failed',
              'StdoutSize': 0,
              'WorkType': 'kube'}}

[fosterseth]

@shanemcd I'd like to get your thoughts on this one before merging. Basically the idea is that if the pod cannot enter "running" state due to ImagePullError (incorrect image name in podspec, incorrect credentials are supplied, registry is down, etc), then fail the work unit. The system will retry the image pull a number of times, currently set by imagePullBackOffRetries := 3 in the code.

This feature is similar to the pod_pending_timeout option we currently have in receptor. However, there is a good use case where users will want to set pod_pending_timeout to something much longer, say 1 hour, but still duck out early if images cannot be pulled.

For example, k8s might not be starting pods due to resources not being available (if cpu/memory requests are set in the podspec), and thus pods will stay in pending until resources are freed up. In that situation, users will want to bump the pod_pending_timeout from 5 minutes to something much longer, but having to wait 1 hour to know that an image cannot be pulled because you typed the image name incorrectly is a bad experience.

So this feature targets the ImagePullError case specifically, whereas the pod_pending_timeout is a general catch-all for any situation where the pod cannot be started.

[shanemcd]

Under which conditions would we hit this? Should the error really be nil if the pod can't start?

[fosterseth]

PodRunningandReady is being passed into a watcher function,

ev, err := watch2.UntilWithSync(ctxPodReady, lw, &corev1.Pod{}, nil, podRunningAndReady())

This will call PodRunningAndReady in a loop, one for each "event" it pulls from the k8s api. It only terminates with PodRunningAndReady returns True, or returns an error

so returning false, nil means, "no the pod isn't ready, but there wasn't an error, so keep polling events from the k8s api"