Pushing an image to GHCR under a new tag fails with 401 Unauthorized

[jauderho]

This just started happening today for some reason. I have multiple GH actions building docker images that has been working without issue.

What is odd is that my action is able to login to GH with no issue but is somehow unable to push the built image.

Here is the error: https://github.com/jauderho/dockerfiles/runs/3353835705?check_suite_focus=true#step:8:7905

GitHub Action: https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/lego.yml

I was able to find a similar issue here (#446) but it did not appear that there was any solution.

[forana]

Seeing this as well and actively trying to debug - what it looks like is happening is that the staging with --push isn't using authentication no, that doesn't make sense. It's got to be something on GHCR's side.

[crazy-max]

cc @chrispat

[forana]

I can confirm two things:

1. Pushing to GHCR without buildx still works just fine.
2. Nothing about the runner environment, docker, or buildx setup changed between working and failing pushes.

[Zenexer]

I've noticed that it's only happening for me when I try to push multiple platforms: docker/build-push-action#437

[Zenexer]

Minimal repro: https://github.com/Zenexer/docker-bug-test/actions Notice that the only difference between the two workflows is that the platforms are split into separate build steps in the working workflow.

[forana]

Interesting - though, neither of my failing builds use --platform at all.

[Zenexer]

@forana, are you able to link to or attach your workflow?

[forana]

@Zenexer I can't link directly because they're private repos, but anonymized excerpt:

 buil:
    name: Build
    runs-on: ubuntu-latest
    env:
      TEST_IMAGE_NAME: ghcr.io/something/something:something
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Install buildx
        uses: docker/setup-buildx-action@v1
      - name: Docker login
        run: echo ${{ secrets.SOME_TOKEN }} | docker login ghcr.io -u username --password-stdin
      - name: Build test image
        run: |
          docker buildx build \
            --build-arg SOMETHING=something \
            --tag ${TEST_IMAGE_NAME} \
            --push \
            .

It does the build successfully, then hits the 401 while pushing.

[Zenexer]

@forana I've made a few attempts to reproduce it with that, but I haven't had any luck: https://github.com/Zenexer/docker-bug-test/actions?query=branch%3Atest2

So far, I've only been able to reproduce it with platforms.

[jauderho]

Confirming @forana 's comment about images being built with only the push failing.

From what I can tell from my Actions history, things were working at least 19hrs ago.

[Zenexer]

@jauderho, are you attempting to use multiple platforms? So far, I've only been able to reproduce it when attempting to push multiple platforms.

[jauderho]

@Zenexer That is correct. See the action file that I posted up top: https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/lego.yml

[Zenexer]

Ah, sorry, didn't realize you were the OP.

As a temporary workaround for workflows with multiple platforms, I've found that duplicating the build/push step so as to create a separate step for each platform works. That being said, it burns through Actions minutes.

[chrispat]

Not sure what is going on here but I have pinged the team.

[razor-x]

Just starting see this. Builds that were OK yesterday are now failing without any changes on all projects. In case it helps, here is an example I just triggered on one of my public repos: https://github.com/makenew/nodejs-app/runs/3357369781?check_suite_focus=true

[catusax]

same problem，the build succeeded yesterday and failed today when pushing to ghcr.io。 https://github.com/LinuxHub-Group/dockerfiles/runs/3359946398?check_suite_focus=true

[muzi502]

same problem +1

[stefanprodan]

@chrispat any news on this? In FluxCD org all our release pipelines are broken.

[crazy-max]

That's an issue with the GitHub Container Registry. Our e2e tests on multi registries work fine except for GHCR:

cc @nilofer

[forana]

Got a response on my github support ticket - they've pushed out a fix, and I've confirmed that my builds are now working again.

[crazy-max]

Looks solved to me too docker/build-push-action#437 (comment)

[timburgan]

Yes, should be working again now

[thaJeztah]

Thanks @timburgan !

[jauderho]

@timburgan Are you able to share more on what broke?

[Zenexer]

Before I spend a lot of time looking into this, are images now getting tagged with the wrong architectures? I don't think this is actually resolved, but I don't want to waste time debugging it if the issue I'm running into now is unrelated.

[Zenexer]

I've got one repo with an untouched workflow that now has ARM64 artifacts tagged as AMD64. I suspect the cause is related to the issue described here, as it broke at the same time. cc @timburgan

Symptoms include seeing the following error when attempting to use affected images:

standard_init_linux.go:228: exec user process caused: exec format error

The image will be tagged for the appropriate platform, but the contents will be from a different platform's build. For example:

% docker inspect sha256:a6ce4e... | grep Architecture
        "Architecture": "amd64",

% docker create sha256:a6ce4e...
667813...

% docker cp 667813..:/bin/bash bad-bash
% file bad-bash
... ARM aarch64 ...

It's possible this has something to do with my usage of cache-from and cache-to, but the same workflow was functioning just fine a few days ago.

Excerpt from the relevant workflow:

      - name: "Build and push"
        uses: "docker/build-push-action@v2"
        with:
          context: "."
          platforms: "linux/amd64,linux/arm64"
          push: ${{ github.event_name != 'pull_request' }}
          cache-from: "type=local,src=/tmp/.buildx-cache"
          cache-to: "type=local,dest=/tmp/.buildx-cache-new"  # TODO: Remove this once it's no longer necessary; see "Move cache" step
          tags: "${{ steps.meta.outputs.tags }}"
          labels: "${{ steps.meta.outputs.labels }}"

      # TODO: Remove this once it's no longer necessary
      # https://github.com/docker/build-push-action/issues/252#issuecomment-804898823
      # https://github.com/docker/build-push-action/issues/252
      # https://github.com/moby/buildkit/issues/1896
      - name: "Move Docker cache"
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

I'll report shortly whether it's possible to work around the bug by removing cache-from/cache-to and the cache move step.

[timburgan]

I'll report shortly whether it's possible to work around the bug by removing cache-from/cache-to and the cache move step.

@Zenexer thank you, that will help. Please also create a new support ticket. They helps greatly with our process for investigation, and we can back-and-forth with you as needed.

[Zenexer]

Update: Removing the cache-from/cache-to step does not work around the bug. I'm just double-checking that it isn't anything obvious on my end; then I'll open a ticket.

Edit: Ticket # 1278544

[Zenexer]

Now that I'm rested, I'm noticing that I didn't revert my workaround for the original 401 issue whereby I split the platforms into separate steps in the parent image. It appears that likely resulted in the oddities I observed when the issue was eventually resolved. If anyone else implemented the same workaround, double-check that you've now reverted it.