[Feat]: Per-client Rate Limit overrides

[seniorquico]

Is your feature request related to a problem? Please describe.

We are using the built-in rate limiter to help mitigate the obvious concerns. We recently started using the sync plugin to mirror our repositories in different regions. The rate limits we chose are low, but more than sufficient for a standard end user. However, when we bring a new mirror online, or if there is a significant change in the content on our upstream, the mirror may hit the rate limit.

Describe the solution you'd like

It would be great to configure a rate limit override that increases the quota for our mirror server. The override could be matched to a client by the remote IP address or maybe by the bearer token's sub claim?

Describe alternatives you've considered

I'm not sure what other alternatives might exist, but definitely open to ideas if there's something simpler we could apply.

Additional context

No response

[rchincha]

@seniorquico Let me understand this better.

[rate-limited src (like dockerhub)] -> [mirrors]

As the list of "mirrors" grows, the load on the "src" increases?

In this case, you probably want to organize like so.

[rate-limited src (like dockerhub)] -> [src mirror] -> [mirrors]

If "src mirror" is under your control, then you have more options.

Thoughts?

[seniorquico]

In our case, the rate-limited source is another zot server, i.e. our setup is more like [zot 1] -> [zot 2]. We have public clients pulling from both 1 & 2, and we setup the zot rate limit configuration on both 1 & 2- using values that made sense for the expected level of usage of our public clients. However, we noticed that the sync plugin on the 2nd server, when it has a lot of work that results in many API calls, can hit the limit and temporarily fails.

It appears the zot rate limit configuration is one-size-fits-all. For now, we bumped up the rate limit to reduce the temporary synchronization failures. However, that also increases the limit for all of the public clients. If we could tell the 1st server to give, as an example, the IP of the 2nd server a higher rate limit while preserving the default lower limit for public clients- I think that's what I had in mind? 100% open to alternatives, though. None were immediately apparent with zot's current configuration, though.

[rchincha]

@seniorquico I see.

We use this golang pkg (https://github.com/didip/tollbooth) in zot and it appears that rate-limiting by source address may be a solution that works for your case.

https://github.com/project-zot/zot/blob/main/pkg/api/session.go#L40
https://github.com/didip/tollbooth?tab=readme-ov-file#features

We always encourage users of zot to get involved in the engineering/development process since it in their own best interests. In that spirit, pls do consider submitting a PR.

[rchincha]

Also note that we are working on zot "scale-out" and have it working for the cloud case. On-prem case should land soon.
https://zotregistry.dev/v2.1.0/articles/scaleout/

[rchincha]

 "Ratelimit": {
            "Rate": 10,
            "PerClientRate": 10, <---
            "Methods": [
                {
                    "Method": "GET",
                    "Rate": 5
                }
            ]
        }

Maybe add a config param like so

[dtroyer-salad]

[I dug into this a bit further for @seniorquico]

A quick recap:

• tollbooth does use remote IP as part of the bucket key along with the URL path
• zot only exposes method as an additional part of the key, tollbooth can additionally use HTTP headers and Basic Auth username
• tollbooth selects the lowest of the candidate rate values for any given request leaving no mechanism to grant a specific method or other request type higher limits

The solution we are looking for is a way to set a higher rate limit for specific requests above the global value or any other values that might pertain. This will either require changes in tollbooth, or a partial re-implementation of some tollbooth functions such as tollbooth.LimitByRequest() to perform the checks in the manner we desire.

The 'select-highest' change is nearly sufficient, we would also need to implement a RemoteIPRateLimiter() function and expose in zot's config file the ability to set specific rates on IP addresses.

While we need both bits above they are mostly independent of each other. I have not yet designed a detailed implementation of these changes so I do not have an answer to things like how we would indicate which selection (lowest vs highest) to use in the config file.

[rchincha]

so I do not have an answer to things like how we would indicate which selection (lowest vs highest) to use in the config file.

Also if these fields must be dynamically managed (add/remove config) based on the current situation.
If you need bespoke rate-limiting, another option would be a dedicated service in front on zot may be more suitable - although an additional thing to manage etc.

[seniorquico]

If you need bespoke rate-limiting, another option would be a dedicated service in front on zot may be more suitable

I've already been thinking about this alternative for some other use cases... We leverage Let's Encrypt for public certificates on the Internet. Every time these update, we have to recycle the zot service for the new certs to take effect. iirc, we also had to setup a workaround to get both IPv6 and IPv4 traffic routed to zot (we could only get it to bind to one stack/port?). We already leverage HAProxy as a frontend for other services, and they have solutions for rate limiting, TLS cert management, and multiple stack/port binding support.

We could take a different tack and contribute a "how to" guide for running a frontend like HAProxy for more advanced configuration scenarios like these? Do you think others would find a guide like that helpful?

[rchincha]

Do you think others would find a guide like that helpful?

Absolutely. zot currently has limited hot-reload. That said, upgrades, and config and hot-reload while traffic is ongoing will require some choreography and multiple replicas.