Pushing 2 tags to Quay gives unauthorized error

[sozercan]

What steps did you take and what happened:
When CI is pushing 2 tags to quay.io (for <sha> and dev or latest and <version>), Quay doesn't seem to support this behavior using buildx (this seems to work fine in Docker Hub).

CI pushes the first tag successfully, and then gives ERROR: failed commit on ref "manifest-sha256:cf45078539d967014f41593b81f702079247e8261cf48919d06d3391da55d66f": unexpected status: 401 UNAUTHORIZED error when pushing 2nd tag.
example: https://github.com/open-policy-agent/gatekeeper/runs/736616057?check_suite_focus=true

We should fix this before the next release, otherwise we can't push latest and <version> tags with same hashes.

relevant issue: moby/buildkit#1440

cc @xunholy

[xunholy]

@sozercan that issue seems slightly unrelated I think because their issue is caused when they run --export-cache which we don't do.

Do we have a contingency plan? Such as using docker hub rather than quay if this issue persists?

I can take a further look shortly - when is our next release scheduled?

[xunholy]

Upon reading through that issue again, it could be related based on the last comment 14 hours ago

Speaking with someone else, they are having trouble pushing 2 tags... e.g. -t foo -t bar, getting the same 401 error.

We should test out a few options. Perhaps we should either use --load and then try push each tag individually? OR we might just need to do a 1:1 build for each tag until quay supports multi-tag manifests being pushed?

Either of these approaches would need someone with permission to push to quay to test and confirm what works best.

[mbarbero]

FYI, I done the test in moby/buildkit#1440. Pushing 2 tags at once works, at least by directly calling buildctl.

buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --opt filename=Dockerfile --output "type=image,\"name=quay.io/mbarbero/test-quay:latest,quay.io/mbarbero/test-quay:other\",push=true"

[xunholy]

@mbarbero did you also attempt to do it with the same approach as being done in the Makefile?

[sozercan]

added a workaround in #672. I am suspecting either local or quay cache issue as the first tag seems to be pushed fine, but when second tag is pushed it fails.

[xunholy]

@sozercan I left some feedback to the PR, I think there may have been a slight misunderstanding of the problem.