bug(digests): Multible Tags get pushed with diferent SHA digests

[k3rnelpan1c-dev]

Description

Pushing an image with more than one tag seems to result in the same image pushed with a different digest per pushed tag.

To reproduce

1. add a basic GH workflow using the buildah-build and push-to-registry action
2. build a basic test image with at least 2 tags (e.g. tags: unstable {{ github.sha }})
3. let the workflow push the image with both tags
4. see the result in the registry (2 digests for the same image 😕)

Example

I stumbled over this while working on some PRs for dohq/dockerhub_ratelimit_exporter where I wanted to add an incremental build workflow and ultimately release workflow pushing a Container Image.
I tried pushing to both the new ghcr.io (tried formats: docker and oci) as well as docker.io (tried format: docker), both resulting in the image getting pushed successfully, but with 2 different digests for the two tags.

my Fork's inc-build Workflow
the result on ghcr.io

The "2" tags in the screenshot are the result of the same inc-build => the same image.

Alternative "solution"

Use the docker based alternative actions to build and push, as these don't seem to face this problem ☹️

[tetchel]

Interesting.. I don't see this problem with our test workflow's images. Both tags have sha 35665.... But this is from quay.io, not either of the registries you used.

We will have a look

[tetchel]

This looks suspiciously like containers/podman#6496

[k3rnelpan1c-dev]

Interesting, I will try and give Quay a try to see if I can replicate this there too (which I should not be able too if your test workflow is anything to go by). I will report back as soon as I have the results.
Regarding the Podman Issue you liked, I should have read up on that too till then 😉

EDIT: This comment in the linked Issue seems interesting

[tetchel]

In the main branch, I've put in a commit which logs the digest after each push instead of at the end. So we can detect this one step earlier.

This confirms again that at least in our test workflow, the digests match.

https://github.com/redhat-actions/push-to-registry/runs/1943110175?check_suite_focus=true

[k3rnelpan1c-dev]

Well I just changed the target registry to quay and I seem to get the same result tbh 😕 (aka I can reproduce this there too)
https://quay.io/repository/k3rnel-pan1c/dockerhub_ratelimit_exporter?tab=tags

[k3rnelpan1c-dev]

the only difference I can see between our workflows is that I am using a "Containerfile" (aka. Dockerfile) to build the Image, while you seem to build from source

[k3rnelpan1c-dev]

I'll install skopeo, just to see if I can see the same difference between the 2 tags as described in this comment

[k3rnelpan1c-dev]

okay well this is interesting.
skopeo inspect docker://quay.io/k3rnel-pan1c/dockerhub_ratelimit_exporter:unstable

Output

{
    "Name": "quay.io/k3rnel-pan1c/dockerhub_ratelimit_exporter",
    "Digest": "sha256:46fcdc9e5ce82ed4b4b3496075fbf25d6b29adc8839ddff5013757113ed44dc9",
    "RepoTags": [
        "unstable",
        "20f9b0c02c10738bae1af9ec73ad448eec1ef6df"
    ],
    "Created": "2021-02-20T19:42:35.949902314Z",
    "DockerVersion": "",
    "Labels": {
        "description": "A exporter for prometheus to check the pull limit of the DockerHub",
        "io.buildah.version": "1.19.2",
        "name": "dockerhub_ratelimit_exporter",
        "url": "https://github.com/dohq/dockerhub_ratelimit_exporter",
        "version": "1.1.0"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:56d2cc464a85a998ee278fe62d1cfe82a5c6da1ae9b60ca35362d36defcd458a",
        "sha256:e116040e706ed480eb26cfb6709d460356f61227129835125ac12666e8c70b9a"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

skopeo inspect docker://quay.io/k3rnel-pan1c/dockerhub_ratelimit_exporter:20f9b0c02c10738bae1af9ec73ad448eec1ef6df

Output

{
    "Name": "quay.io/k3rnel-pan1c/dockerhub_ratelimit_exporter",
    "Tag": "20f9b0c02c10738bae1af9ec73ad448eec1ef6df",
    "Digest": "sha256:55a559bee5ebf72faf691fe66ec08e0eb663043bc8322b652deef07e84aaef20",
    "RepoTags": [
        "unstable",
        "20f9b0c02c10738bae1af9ec73ad448eec1ef6df"
    ],
    "Created": "2021-02-20T19:42:35.949902314Z",
    "DockerVersion": "",
    "Labels": {
        "description": "A exporter for prometheus to check the pull limit of the DockerHub",
        "io.buildah.version": "1.19.2",
        "name": "dockerhub_ratelimit_exporter",
        "url": "https://github.com/dohq/dockerhub_ratelimit_exporter",
        "version": "1.1.0"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:56d2cc464a85a998ee278fe62d1cfe82a5c6da1ae9b60ca35362d36defcd458a",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
        "sha256:e116040e706ed480eb26cfb6709d460356f61227129835125ac12666e8c70b9a"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

edit:
Not as interesting as I thought unfortunately. I fired the same commands against the image on ghcr.io (with the corrected git hash ofc) and the only diff there is the tag digest 😕

[tetchel]

I tried a dockerfile build, and also added a LABEL to my dockerfile because I noticed yours had some and mine didn't. workflow

Still, the SHAs match - and I used dockerhub, too.

I will try to dive deeper tomorrow...

[tetchel]

hey @k3rnelpan1c-dev - I cloned your repo and went through the following steps here

1. changed the workflow to use my quay.io repo.
2. successfully reproduced your bug
3. Tried this workaround which worked, having the shas match.
4. Noticed that the podman version in your workflow (2.2.1) was much older than the podman version in push-to-registry's (3.0.0). This is because you targeted ubuntu-latest which is 18.04 while this repo uses 20.04.
5. Removed the workaround and updated the ubuntu version
6. The bug went away!

So, this is indeed containers/image#733 which appears to have been fixed in podman 3.0.0. You should be able to fix this for yourself by updating your workflow to use runs-on: ubuntu-20.04

[k3rnelpan1c-dev]

Hi @tetchel.

first of all THANK YOU for this detailed analysis and solution 👍
I am a bit annoyed about myself not thinking of something like this myself 😅

Regardless, this should maybe be noted in the readme until GitHub / Microsoft have exchanged the image behind ubuntu-latest (at least in combination with trying to push multiple tags). I can open such a PR if you want me to.

Thank you for all the work and very quick resolution 🎉

[offa]

@k3rnelpan1c-dev / @tetchel ubuntu-latest has moved to 20.04 this march; see actions/runner-images#1816 or available environments overview.