Distributor rate-limiter should call Wait() not Allow()

[bboreham]

I think this will work better under very sudden bursts of input.

Under certain circumstances I have seen a scraping Prometheus (v1.8) expand to 1,000 shards, and then have all 1,000 shards upload at the same time, creating an input burst of x00,000 samples.

Our rate-limit defaults to 25,000 samples per second, per distributor.

The behaviour of Allow() under a large burst is to count up to 25,000 saying 'yes', then say 'no' to the rest.

The behaviour of Wait() is to say 'yes' to the first 25,000, then start a sleep for the next one, then the next one, and so on. If the Context has a timeout, let's say 10 seconds, then it figures out it can allow 250,000 samples to wait, and it starts saying 'no' immediately to subsequent callers.

Note we don't currently have a timeout on the input side of distributor, but we have one defaulted to 2 seconds downstream to ingesters.

[bboreham]

In practice, holding all the pending requests in memory of distributors is a bad idea.