connect_and_idle scenario drops bulk connections (intermittent)

[rpappalax]

On long test runs (>30 mins), ap-loadtester will drop a large number of connections (not usually all). This usually happens at least once on long-running tests, but can happen multiple times.

I've never been able to get a connection test to exceed 11M connections.
https://github.com/mozilla-services/ap-loadtester/blob/master/aplt/scenarios.py#L44

[rpappalax]

NOTE: this test completed successfully after 3 hours reaching the 100K conns / node threshold on each.
so, how to make the ap-loadtester more robust to handle network hiccups?

[pjenvey]

The dip here turned out to be due to a redeploy of stage we weren't aware of, so ap-loadtester actually handled it pretty well except for lack of feedback (#73).

I'll note one more thing -- we discussed that the connect_and_idle testplan took a while to create so many connections. Per datadog, this run did about 25 conns per second. We should able to greatly improve that with a higher aplt_testplan stagger value (how many scenarios to launch per second).

[rpappalax]

@pjenvey ok cool. that's very useful info.
The README indicates that based on this, the 2nd param is stagger value (in this example, set to 1).
In my tests, I've played around with these values, but started setting them low to speed up the tests. I had the value set to 2, but I'll try setting it to 10 instead and try ratcheting it up progressively if needed.

Run 5 instances of the basic scenario, starting one every second, against the dev server:
$ aplt_testplan wss://autopush.dev.mozaws.net/ "aplt.scenarios:basic,5,1,0"