Meta: Bugs found during Kubecon demo prep

[pmorie]

While preparing the Kubecon demo, I found that the controller can get into a state where new resources are not worked on. The scenario I ran into is:

I had added several brokers to the catalog, some of which were deployed into the same cluster as the catalog itself. Currently the controller has a resync period for all informers hard-coded to 1 minute. When those brokers went sideways for whatever reason, they would time out when the controller tried to contact them. Due to those timeouts, the resync interval would elapse before the controller had time to work on the new resources.

There are a few different facets of this situation that need improvement; I'm going to create issues for them because focused issues are more easy to track, but wanted to tie them together here.

TLDR:

• There should be a flag for the controller resync interval (Controller resync interval should have an option #626)
• The controller shouldn't continually list a broker's catalog (Controller shouldn't continually relist a broker's catalog #627)
• We should use exponential backoff when a broker operation errors (Use exponential backoff when a broker operation errors #628)

[arschles]

Added to 0.0.2 milestone.

[arschles]

Removing this issue from a milestone and adding Meta to the title. All sub-issues for this (#626, #627 and #628) are all assigned to a milestone.

[pmorie]

This is all closed out now, woot.