Docker Hub rate limiting

[gatti2602]

I'm not clear on this email that our team received:

Seems Docker hub is already enforcing the rate limits since 2020 so what is changing on September 30 that we have to be aware of?

[terencet-dev]

Hi @gatti2602, thanks for reaching out to us. @JXavierMSFT can help out with this question.

[bkarakashev]

Hi @terencet-dev @JXavierMSFT any update on this? we have the same question.

[squillace]

Hi @gatti2602 if you have not been affected by "anonymous-pull" rate limiting from Azure IPs, there's a chance that this is because Azure IP ranges are exempted from rate limiting until the end of the month of September. BUT there is also the chance that you simply do not pull frequently enough to be limited or that you're doing authenticated pulls with Docker credentials. In both of the latter cases, you won't see any change in behavior. We can't know how much you pull from Docker Hub library, so please do some testing to discover whether you're exceeding the documented limits for authenticated pulls from the library content.

If you are currently pulling anonymously from Docker Hub library more than the documented limits, you may begin to see rate limiting take effect in October.

And I should also mention that all public registries do rate limiting; this discussion is merely that the content in the Docker Hub library is so frequently used that it's easy to pull more than the anonymous limits without thinking about it. The fact that Azure and Docker have supported this up until now has "masked" the situation. If you use ACR Caching service, however, you're going to a) skip worrying about this and b) be able to establish a more reliable and performant delivery of DH library content going forward.

Does this help? Please, feel free to continue asking questions here about your situation.

[gatti2602]

I understand that Azure IPs in case that:

1. Private network does not have a load balancer with outbound rules
2. The VM pulling the image does not have a public IP

Are a shared pool of IP addresses across different users in the cloud (Azure names it Default outbound IPs if I'm not mistaken) so if other users are anonymously pulling images from Docker Hub my limit can be affected, correct?

[wziska]

It's 10/2 today and I run a pipeline with following script:

- script: |
    TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
    curl -I -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
  displayName: 'Test docker hub limits'

and in output I get:

HTTP/1.1 200 OK
content-length: 2782
content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
docker-content-digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
docker-distribution-api-version: registry/2.0
etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
date: Mon, 02 Oct 2023 12:27:32 GMT
strict-transport-security: max-age=31536000
docker-ratelimit-source: 13.79.197.48

There is no ratelimit-limit nor ratelimit-remaining in response what according to https://docs.docker.com/docker-hub/download-rate-limit/ means image or IP is unlimited. That's exactly the same response I was getting week ago.
So, did anything really change after 30 September 2023 or not?

[toby-coleman]

Does anyone know the current status of this? I received an email today warning of rate limiting at the end of June 2024, but am unsure as to exactly what, if anything, is changing on this date? @JXavierMSFT @terencet-dev

[JXavierMSFT]

Hey @toby-coleman,

I can answer this question. We pushed this date back to give customers more time to mitigate the changes. However, we won't be extending the deadline post June 30th. You are correct, the email you received today is the 3rd attempt to warn customers. Nothing has changed from the emails except the date.

[JXavierMSFT]

Hey @wziska,

You are correct, rate limiting didn't begin on September 30th. We pushed the date back to give customers more time to mitigate the changes. The rate limit begins on June 30th.

[bmo-linklog]

We received the email and we have GitHub action hosted runners as well.

Does this change only apply to Azure hosted pipelines or also can affect GitHub action hosted runners as well?

Thanks in advance for your answer. @JXavierMSFT @terencet-dev

[bmo-linklog]

Hey @JXavierMSFT @terencet-dev, Any update on this? :-)

[squillace]

I'm fairly sure this does not affect GH for the time being, but I'll have the team confirm.

[JXavierMSFT]

@squillace is correct. This change only affects Azure resources. GitHub is unaffected by this change. @bmo-linklog if you see any rate limiting on your GitHub actions. Please open a ticket with GitHub.

[Hazmi35]

Hi, I know there's ACR Cache available but is there an equivalent of mirror.gcr.io and public.ecr.aws on Azure? Preferably public too