Improve jobrunner wait code to retry on recoverable errors

[dsimansk]

Per PR discussion: https://github.com/knative-sandbox/kn-plugin-event/pull/42/files#r674008140

Let investigate the RetryWatcher from client-go tools and knative/client waiting loop to introduce retry on error in the waiting loop.

There are cases that may result in premature closing of ResultChan, that's caused by recoverable failure. Detecting such occasions and re-creating watcher instance might be very desired.

https://github.com/knative-sandbox/kn-plugin-event/blob/0e1fe36db37682a10548808563a38c7e5ac3814c/pkg/k8s/jobrunner.go#L68-L96

[dsimansk]

/kind enhancement

[cardil]

I think #68 is directly impacted by this.

For visibility linking discussion which lead to opening this issue #42 (comment)

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[cardil]

/remove-lifecycle stale

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.

[cardil]

/remove-lifecycle stale
/triage accepted

[cardil]

@dsimansk Do you think this issue should be part of the Stable release (GA) milestone?

[dsimansk]

@cardil depends if we see too much intermittent failures in tests or in the "production" use. However, my original proposal was to use client-go lib that's probably better tested and covers more recoverable failures.
As a bottom line I'd like to have wait based in kn on that lib as well. But historically it wasn't possible due to supporting Cloud Run in the past.