Docker image missing for fluxcd/flux:1.24.1

[BitProcessor]

Describe the bug

Docker image is missing for docker.io/fluxcd/flux:1.24.1

Resulting in:
Warning Failed 8s kubelet Failed to pull image "docker.io/fluxcd/flux:1.24.1": rpc error: code = Unknown desc = Error response from daemon: manifest for fluxcd/flux:1.24.1 not found: manifest unknown: manifest unknown

Steps to reproduce

Try to upgrade flux to 1.24.1

Expected behavior

Image exists on Docker Hub

Kubernetes version / Distro / Cloud provider

v1.18.20-eks-8c579e

Flux version

Flux v1.24.1

Git provider

No response

Container Registry provider

No response

Additional context

No response

Maintenance Acknowledgement

• I am aware of Flux v1's maintenance status

Code of Conduct

• I agree to follow this project's Code of Conduct

[BitProcessor]

Probably related to: #3544 (comment)

But the release was published, people got emails (including me) and telling flux to upgrade itself results in a broken state.

[kingdonb]

Thank you for opening the report, I am aware of the issue and working on it right now.

Just for my information, what is emailing you/how are these notifications being received? (Is it the GitHub release?) The release workflow failed uploading binaries to the GitHub release, which only happens after the GitHub release is tagged, so unfortunately I'm not sure there is any way I could have discovered this issue without tagging the release and broadcasting those notifications out.

The other thing that could have notified you would be the Checkpoint API, which leaves messages in the Flux logs. (The new release is already noted in there too.)

In general I know not everyone is using the Helm chart, but I wouldn't try to upgrade before you see the Helm chart release.

I will keep this issue open and update it when the problem is resolved 👍

[BitProcessor]

It was indeed the GitHub release that notified me via email.
Would you have been able to avoid the upgrade message in the logs?

Because it's indeed also saying:
flux ts=2021-09-09T12:12:24.968203683Z caller=checkpoint.go:21 component=checkpoint msg="update available" latest=1.24.1 URL=https://github.com/fluxcd/flux/releases/tag/1.24.1

Thanks in advance for fixing :)

I'm not using the Helm chart btw. I guess I bootstrapped this cluster about a small year ago using the fluxctl tool and it was not using Helm for flux back then (or was it 🤔 ) - anyway...

[kingdonb]

I think both ways (fluxctl and the helm chart) are meant to be supported, I'm not sure which came first but that is fine.

I have narrowed in on the issue and I believe I have found a fix. Since no release image was actually published for this tag, and I have all the permissions needed to delete the tag that failed, I think I can fix it and it should be resolved inside of an hour.

You're going to receive another notification from GitHub about a new release, same version as before, (only this time with more feeling...)

[kingdonb]

The missing image has been published and the situation is resolved. (The helm chart should be along later this afternoon, in #3548.)