gateway-client just raises a warning when unable to connect to NATS subscriber upon init

[ptemmer]

Describe the bug
When configuring NATS as the subscriber for a gateway, upon starting the Gateway pod the gateway-client container tries to connect to the NATS server. If the server isn't reachable at that moment, either because it's still booting, or because of a network hick-up, the gateway-client warns that it wasn't possible to connect, but continues booting nonetheless. However, upon receiving events, the gateway-client is unable to dispatch them.

To Reproduce
Steps to reproduce the behavior:

1. Configure a gateway resource with NATS as it's subscriber
2. Stop the NATS server
3. Deploy the gateway resource
4. See error

WARN[2020-06-09 05:16:49] failed to create a transport                  error="nats: no servers available for connection" subscriber="{nats://nats:4222 webhook-subject webhook-sensors}"
INFO[2020-06-09 10:08:25] dispatching event to subscribers              event-source=github-branch
WARN[2020-06-09 10:08:25] unable to send event. no client found for the subscriber  subscriber="{nats://nats:4222 webhook-subject webhook-sensors}"

Expected behavior
I would expect gateway-client to keep trying to connect, or either exit with an error

Version
Argo v0.13

[VaibhavPage]

We are going to introduce EventBus https://docs.google.com/document/d/1uPt2DyvzObEzZVbREqjW-o1gKCZ_hB8QS5Syw7_MLT8/edit?usp=sharing pretty soon which will address this behavior.

[whynowy]

Not an issue any more with new EventBus implementation.

[ptemmer]

Great proposals, especially migrating the Gateway into the EventSource object.
Whether the Eventbus implementation will mitigate the problem I experienced will depend on the NAT client behavior. From what I understand, the EventBus will abstract the NATS configuration for the user, however under the hood there will still be a NATS client and NATS server. Correct me if I'm wrong, but this aint different from what I currently have. So the question remains; what will happen when the NATS server aint reachable when eventsource deployment spins up and tries to connect to it?

[VaibhavPage]

@ptemmer With the new architecture, the logic for connecting to nats streaming server has also changed. We will provide best-effort QoS for NATS server connection, take a look at #749

[ptemmer]

Great! Can't wait to have it up and running