Validating many downloads at once seems to fail sometimes

[Yolgie]

As discussed in #40730 (comment) it seems like the label:Error-Installer-Availability sometimes is created by manifests with many downloads that are then all triggered at the same time and run into timeouts even though all binaries are available.

Would it be possible to stagger the downloads in the azure pipeline?

[denelon]

We don't have many data points to indicate how much time should pass between downloading the subsequent installers. I'm sure we could make it a configurable value, but we need to have a reasonable best guess to get it started. I'd rather not need to track the wait time on a per package/domain basis.

[zachcarp]

@OfficialEsco
@denelon

I can see where this would be a problem more and more as we have more and more versions. Especially when we have packages with dozens and dozens of versions.

Per my comments on another PR - The first initial naive solution that comes to mind to get us out of trouble would be to add some sort of throttling to our validation process for installers where we rate limit our downloads as suggested by @OfficialEsco.

As we don't have any solid numbers - it would be helpful to gather some data. That being said, if we can get an estimate of a threshold where this become a problem (as example more then 10 installers or more then 20 - whatever it may be) and start those sequentially with a short (globally configurable) delay we can reduce the churn on these kinds of items.

Another solution that would likely help us longer term would be to add some level of short term package caching to speed up re-runs and reduce our demand on external CDNs. This would likely be more of an effort then the above.

As always - I am open to additional suggestions.