Loki driver prevents containers from starting if Loki server is unreachable

[avkonst]

Describe the bug
Due to timing issues, my logging infrastructure is launched after application containers OR the Loki server may not be available due to cluster partitioning when containers are starting. I have encountered the following error, caused by Loki driver in these conditions:

Creating promtail         ... done
Creating weavescope       ... done
[3] 
[3] ERROR: for nodeexporter  Cannot start service nodeexporter: failed to initialize logging driver: error creating logger: Post http://%2Frun%2Fdocker%2Fplugins%2Fe4361e214006be3ec4c7140e2e828085544e5138c39cbeaf1cac099b1fe4cc75%2Floki.sock/LogDriver.StartLogging: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[3] 
[3] ERROR: for kafka  Cannot start service kafka: failed to initialize logging driver: error creating logger: Post http://%2Frun%2Fdocker%2Fplugins%2Fe4361e214006be3ec4c7140e2e828085544e5138c39cbeaf1cac099b1fe4cc75%2Floki.sock/LogDriver.StartLogging: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[3] Encountered errors while bringing up the project.

Interestingly, this error does not show up for all containers but for some only. Maybe it is timing issue on start up, not sure.
Regardless of the symptoms, loki driver should never prevent a container from starting. It may produce errors and warnings to whatever logs, but containers should start even if the logging backend is down.

To Reproduce
Steps to reproduce the behavior:

I have got huge set of docker-compose files, which are launched across 3 virtual machines. I have got vagrant spec for launching these 3 VMs and a set of commands (about 4 in total) to use in order to trigger start of the containers. This will help to reproduce the problem. I was not able to reproduce it on the downsized version (ie. with fewer machines or fewer containers).
Would you like me posting the link to the project with vagrant specs for VMs?

Expected behavior
Loki logging driver does not prevent containers from starting in any circumstances.

Environment:

• a set of docker-compose across 3 VMs

Screenshots, Promtail config, or terminal output
See above. Tell me what other information you would like. I use loki driver version 1.5.0, but see this on latest too.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[cyriltovena]

Are you saying that if Loki is down you can't start a container with the docker driver ?

[avkonst]

Correct. But it does not happen always. There is some timing issue I believe...