Heavy traffic from Renovate on the Gradle Plugin Portal

[ljacomet]

How are you running Renovate?

None

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

If you're self-hosting Renovate, select which platform you are using.

None

What is your question?

Hey Renovate team,

We at Gradle are offering the Gradle Plugin Portal for Gradle builds to download Gradle plugins, mostly community contributed.

Over the past 2 weeks we have seen the Renovate traffic increase a lot against the Gradle Plugin Portal, to the point where it impacts its availability.

We have identified a single IP as causing most of the added traffic and have taken measures for this. But even without it, the traffic almost doubled over a month.

Are you aware of recent changes in Renovate that could have caused increased traffic related to Gradle plugins in Gradle projects?
Or have you registered a similar spike in Renovate usage, in particular on Gradle projects?

Regards,
Louis Jacomet
Gradle Build Tool team

Logs (if relevant)

Logs

Copy/paste the relevant log(s) here, between the starting and ending backticks

[rarkins]

Hi Louis, we'll look into it ASAP. Most likely scenario is a regression in caching of lookups

[rarkins]

#27019 could be the same problem. 37.139.4 was released on January 17: https://github.com/renovatebot/renovate/releases/tag/37.139.4. The graph above shows a bit of a ramp up from there and then a bigger rise starting around 28th January.

If most was from one IP, it's probably the Mend hosted app. @nabeelsaabna on which day did we upgrade from <=37.139.3 to >=37.139.4?

Cc @zharinov

[viceice]

yes, renovate should handle 429 gracefully

[rarkins]

@aaltounyan thanks for the info, we understand completely and sorry again for the bug!

[aaltounyan]

Hi team

We are having to update the way our rate limit functions due to imposed changes by our provider. We expect this change to happen imminently this week.

Until now, only requests above the limit (100/s) were blocked, but going forward once the limit is reached, subsequent requests will be blocked with a 429 for 30 seconds. I'm not sure how this will affect your Mend hosted app, but wanted to give you a heads up.

Your .110 IP is still occasionally hitting the limit, with 2.6K 429s out of 2.2M requests in the last 24 hours, but this hasn't been causing any problems for us.

We are working with our provider on this, and at the moment cannot say for sure if this rule will stay as-is or return to the previous state. Please let us know if you are impacted by this change.

[HonkingGoose]

Thundering herd problem?

Until now, only requests above the limit (100/s) were blocked, but going forward once the limit is reached, subsequent requests will be blocked with a 429 for 30 seconds. I'm not sure how this will affect your Mend hosted app, but wanted to give you a heads up.

Thank you for the heads-up! ❤️ Checking if I understand you correctly:

1. A Renovate client (ours, or self-hosted user) overwhelms your server.
2. Your server returns a 429 reponse with 30 seconds wait time.
3. Client waits 30 seconds, and then fires a new request at your server.

So far, so good, but what if:

4. All clients come back at the exact same time, thus still overwhelming your server, just 30 seconds later? Wikipedia: Thundering herd problem

Possible solutions to thundering herd

• Buckets with range of return times
• Exponential backoff
• Setting hard limits for Renovate clients in our code

Buckets with a range of return times

Would it be a good idea to add a bit of randomness to the 429 wait times your server hands out? Like one bucket of clients waits between 30-35 seconds, then the next bucket 35-40, the next 40-45, and so on. That way you break up the load into smaller chunks?

Exponential backoff

Or maybe you can use exponential backoff, where if your server is still overloaded, you return progressively longer wait times?

Set hard limits for Renovate clients

Maybe we should set hard limits for the number of requests Renovate is allowed to fire at your server? I asked the maintainers if that's a good idea, see:

• Apply default rate limits for hosts/registries with known rate limit needs #27802

Summary

I listed some possible solutions, and ideas. I'll let the Renovate maintainers, and you, decide what's best. 😉

[aaltounyan]

It's per IP, so general users of Renovate won't be impacted. We know from the history of this Issue that the hosted Mend service may be impacted. The servers shouldn't be impacted, our DDoS provider would just block them again. From what we can see, there are only occasional peaks where Mend goes over the limit - the workloads behind those peaks may suffer.

We are limited in our options based on our provider, so just explaining how things are likely to work for the time being. The other options you mention are not possible at this moment.