Create container repo tags for each FCOS release

[jdoss]

Can we get container tags for each FCOS release so we can pin to a specific version or roll back to a specific version if we encounter issues with one of the stream layer updates?

For example coreos/butane#428 has me blocked with the current quay.io/fedora/fedora-coreos:stable tag being unusable. I can't pull an older image by SHA because quay.io seems to prune untagged SHAs over time which is making it hard for me to roll back until coreos/butane#428 gets addressed. I wish I could just pull quay.io/fedora/fedora-coreos:36.20221001.3.0 for example.

It would be super great to have a history of each FCOS release as a container tag so we can have a safety net for rollbacks.

[dustymabe]

Transferring this to the FCOS tracker for more discussion there.

[cgwalters]

For reference, older builds are currently available in the raw builds browser (e.g. stable):

$ podman load -i https://builds.coreos.fedoraproject.org/prod/streams/stable/builds/37.20221127.3.0/x86_64/fedora-coreos-37.20221127.3.0-ostree.x86_64.ociarchive
...
Loaded image: localhost/latest:latest
$ podman tag localhost/latest:latest quay.io/fedora/fedora-coreos:37.20221127.3.0

Of course, we should very clearly be pushing these things to at least some registry with these tags as you say.

Perhaps we could push tags, but with auto-expiration dates by default to discourage using very old images. (OTOH, we aren't garbage collecting the full cosa builds either; one can still access "Fedora CoreOS 31.20191210.3.0 — 3 years ago")

[jlebon]

Long-term, we need to enact the GC policy agreed upon in #99. And I don't see why container images wouldn't be part of that. In that context, it probably makes more sense to not rely on auto-expiring tags and instead have the GC centralized by whatever process will execute it at the cosa build level. Short-term, some aggressive auto-expiring tags (e.g. 3 months?) sounds like a reasonable way to give some breathing room to users opting into the experimental stuff.

[cgwalters]

Another option here is to move older images to e.g. quay.io/fedora-archive for a while. Something like:

• Retain 2-3 version tags for stable at least
• Move older stable tags to fedora-archive

[dustymabe]

This was discussed in the Fedora CoreOS community meeting today. We discussed the merits of different options but ultimately ran out of time. A few of us agreed to meet and bring back a proposal to the community:

12:36:58  dustymabe | #action jlebon bgilbert walters dustymabe get together to work on proposal
                    | for short term and longer term solution to the problem of being able to pull
                    | older FCOS releases from a registry

[dustymabe]

In the meeting @jlebon @cgwalters @bgilbert and I had today we came up with this proposal:

- create versioned tags - i.e. `37.20221225.2.2` 
- prune after 2m (production) 2w (development); extend in future if desired by users
- we will not prune barrier releases
- no fedora-archive registry repo; it all happens in our `quay.io/fedora/fedora-coreos` repo.
- no use of quay.io auto-expiration tags due to concerns about pruning when copied to other repos, etc.
    - we'd add an extra step to our current release process to do the pruning

[jdoss]

+1 That proposal sounds fantastic guys. Thanks!!

[Ultra980]

Will there also be major version tags updated with each stable release (such as 37, not just 37.[date])?

[dustymabe]

Will there also be major version tags updated with each stable release (such as 37, not just 37.[date])?

No. Fedora CoreOS has a stream update model that's more like a rolling release. The version numbers in each build ID are mostly informational (see explanation of build ID here). They're not intended to be used for pinning on a particular set of content for an extended period of time.

[cgwalters]

OK, looking at #1263 I think having tagged versions in the repository is a dependency. This was actually noted in #1263 (comment)

A corollary to that is that we cannot GC the version tags beyond the oldest build that acts as an upgrade barrier to the oldest release for which we want to support upgrades. This is a bit subtle, but simplifying imagine that we have releases 34, 35, 36, 37), where there's a barrier for each release. Fedora 34 was released almost two years ago. Let's assume that we don't care about people updating from 34 to 35 anymore - everyone should be on at least 35. Hence, we can GC the tags for 34 and 35 at this point, but not 36 or 37.

[bgilbert]

Ah yeah, good catch on the upgrade barriers. Isn't it the case though that we could GC all old releases except for the barrier releases?

[jlebon]

Updated proposal to skip pruning barrier releases as discussed in today's community meeting.

[cgwalters]

So there's two models to implementing tagging:

• Have an external controller manage GC
• Program a GC policy into the registry

The nice thing about the latter is there's one less thing running operationally. I did a bit of looking and quay.io does have a programmatic API to set the expiration for an image: https://access.redhat.com/documentation/en-us/red_hat_quay/3.9/html/red_hat_quay_api_guide/red_hat_quay_application_programming_interface_api#changetag

So we could tweak our upload process to do this.

Or I guess, have the controller just set one if it sees an image without.

[dustymabe]

• Program a GC policy into the registry

We touched on this in #1367 (comment):

• no use of quay.io auto-expiration tags due to concerns about pruning when copied to other repos, etc.
  - we'd add an extra step to our current release process to do the pruning

[cgwalters]

Note that quay.io has two modes of implementing expiration - one via a label, and one via API. The label one is obviously super dangerous for copying the image and honestly is just overall a bad idea IMO. I'm talking about the API driven one.

[dustymabe]

I'm talking about the API driven one.

That's definitely interesting and not something I was aware of before. I don't think I understand how the link enables you to change the expiration date of the image, though. Is there an example somewhere on the internet of how to make an API call to do it?

[dustymabe]

OK now I see there is a field for expiration in the body of that API call. When I first looked at it I thought that next table was for the next API endpoint and didn't look at it. 👍

[travier]

Related work in progress:

• cmd/prune-containers: add a GC script for containers images coreos-assembler#3826
• jobs/release: tag containers images with version fedora-coreos-pipeline#995

[jbtrystram]

Landed in coreos/fedora-coreos-pipeline@a1a6db8

[travier]

Thanks, looks like this is working: https://quay.io/repository/fedora/fedora-coreos?tab=tags