sidecar: increase MaxIdleConnsPerHost or allow it to be configured

[blockloop]

The thanos sidecar is mounted to a single Prometheus instance where it establishes an HTTP connection for queries against that machine. It's using the http.DefaultTransport when connecting to that client which has MaxIdleConnsPerHost set to 2 by default see DefaultMaxIdleConnsPerHost. We run a large, heavily used cluster and we're receiving about 1.7k queries per second from some machines. Because of this load we are constantly over the connection pool limit so new connections are being established and destroyed. This causes our machines to run low on ports since each connection sits in a TIME_WAIT state for 60s, consuming a port.

Since the connection pool size depends entirely on the setup I think there should be a flag to allow us to run sidecar with custom connection pool size. Perhaps --prometheus.connection-pool-size?

Here are some examples showing the TIME_WAIT connections being created in excess in the second example: https://tleyden.github.io/blog/2016/11/21/tuning-the-go-http-client-library-for-load-testing/

This graph shows HTTP requests between sidecar and Prometheus on our machines.