Official image pulls are being rate-limited (nginx, busybox, traefik)

[rfay]

Our DDEV builds on github are showing rate-limiting failures on official images like nginx, traefik, and busybox. It seems like something has gone wrong with the dockerhub rate-limiting today?

#3 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/nginx/manifests/sha256:a98c2360dcfe44e9987ed09d59421bb654cb6c4abe50a92ec9c912f252461483: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
failed to pull image busybox:stable: exit status 1

ERROR: failed to solve: traefik:3.0: failed to resolve source metadata for docker.io/library/traefik:3.0: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/traefik/manifests/sha256:a208c74fd80a566d4ea376053bff73d31616d7af3f1465a7747b8b89ee34d97e: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

I hope you can fix that. I think "official images" and "DSOS images" should not be having this problem, right? And of course, rate-limiting from Github Actions runners is a serious problem for the world, because all those IP addresses are used by so many of us.

[benja-M-1]

Hi @rfay indeed something was wrong and we are sorry for the inconvenience 🙁
We rolled back a change so the issue should be solved by now. Let us know if you still experience any problem.

[rfay]

Thank you very much!

[zachkirsch]

@benja-M-1 I am stil seeing 429 Too Many Requests - Server message: toomanyrequests when pulling node:22-alpine, not sure if that applies to this issue or that is supposed to be rate-limited.

For context, this is within github actions where (I believe) I don't have the ability to log in and reduce the rate limiting

[benja-M-1]

@zachkirsch it is most likely not related to the issue we had.

Regarding authenticating within a GitHub Action you can use that action https://github.com/docker/login-action?tab=readme-ov-file#docker-hub.

[zachkirsch]

ah cool thanks @benja-M-1 !

[rfay]

But... you shouldn't need to authenticate on an official image either.

[brianmlink]

This is still happening.

[jcarter3]

Which part is still happening? Official images are subject to rate limiting, but the specific issue happening from GHA was resolved. Can you be more specific where you are seeing an issue?

[sudo-bmitch]

I can confirm that I saw the issue several days ago and it was quickly resolved in my own GHA job.

[chenele]

It started to happen to us also.,

#3 [internal] load metadata for docker.io/library/node:18
#3 ERROR: failed to copy: httpReadSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/node/manifests/sha256:ddd173cd94537e155b378342056e0968e8299eb3da9dd5d412d3b7f796ac38c0: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
------
 > [internal] load metadata for docker.io/library/node:18:
------

we use Github Actions for our CI-CD and always used anonymous docker login to pull the official images.

is there a concrete issue now that causing this?
is it just coincidence as Github Runners IP are shared by everyone?

trying to understand if we should switch to authenticated user everywhere.

[jcarter3]

Do you explicitly pull docker.io/library/node:18? Or just node:18? I wonder if this is a mismatch in how the credential resolver loads credentials.

By default, credentials are saved & used by domain name -> if you do a docker login registry.foobar.io, credentials are saved under registry.foobar.io and pulls that start with registry.foobar.io will use those creds. Docker Hub credentials are usually saved under index.docker.io or registry-1.docker.io - pulls that use docker.io/... will fail to leverage them and pull anonymously (and get rate limited).

Github Actions workers are usually exempt from rate limits, but things like 👆 can cause the mechanism to fail.

[lukehsiao]

@jcarter3

Github Actions workers are usually exempt from rate limits, but things like 👆 can cause the mechanism to fail.

Just to check, are you suggesting we should pull node:18, and not docker.io/library/node:18 to ensure we are except from rate limits?

We are suddenly seeing these today (we didn't last month). Granted, pgvector is not an official image.

[cabello]

Just got hit by this with postgres while on self hosted runners.

We are using the official docs https://docs.github.com/en/actions/use-cases-and-examples/using-containerized-services/creating-postgresql-service-containers

jobs:
  services:
        postgres:
          image: postgres:14
          env:
            POSTGRES_PASSWORD: postgres
          options: >-
            --health-cmd "pg_isready -d postgres -U postgres"
            --health-interval 5s
            --health-timeout 5s
            --health-retries 5
          ports:
            - 5432:5432

The mentioned docs do not cover how to point to a non Docker hub image.

This one does: https://docs.github.com/en/actions/use-cases-and-examples/using-containerized-services/about-service-containers#authenticating-with-image-registries

Trying this now.

[jcarter3]

Just to check, are you suggesting we should pull node:18, and not docker.io/library/node:18 to ensure we are except from rate limits?

Yes, the thinking is that this should help it. docker.io is not a registry (it's either index.docker.io or registry-1.docker.io) and so credentials will not be sent.

Also for clarity - Docker Official Images are not exempt from rate limits.

[mezotaken]

Yeah, still happening with just a single image pulled

[cabello]

Follow up, in our case we use Ubicloud and they were on top of the problem: ubicloud/ubicloud#2244 (reply in thread)

Turns out it was this: