tracker: Rebase onto Fedora 41

[jlebon]

Rebase to a new version of Fedora (N=41)

At previous Fedora major release

Open tickets to track related work for this release

• Fedora Changes Considerations (tracker: Fedora 41 changes considerations #1714)
• Package Additions/Removals (tracker: package additions/removals for F41 #1785)
• Test Week (tracker: F41 Test Week on Oct 7 #1801)
• Communications Tracker (Tracker for Fedora 41 changes to notify users #1715)

At Branching

Branching is when a new stream is "branched" off of rawhide. This eventually becomes the next major Fedora (N).

Release engineering changes

• Verify that a few tags were created when branching occurred:

• f${N+1}-coreos-signing-pending

• f${N+1}-coreos-continuous

• Add and tag a package (any package) which is in the stable repos into the continuous tag. This will create the initial yum repo that's used as input for building the COSA container.

• koji add-pkg --owner ${FAS_USERNAME} f${N+1}-coreos-continuous $PKG

  • example: koji add-pkg --owner dustymabe f36-coreos-continuous fedora-release
  • This example uses the fedora-release RPM, but it could be any other.

• koji tag-build f${N+1}-coreos-continuous $BUILD

  • example: koji tag-build f36-coreos-continuous fedora-release-36-0.16

• Add the N+1 signing key short hash (usually found here) to the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 32/33/34/35 keys. You'll most likely have to get someone from releng to run the second command (edit-tag).

  • koji taginfo coreos-pool
  • koji edit-tag coreos-pool -x tag2distrepo.keys="12c944d0 9570ff31 45719a39 9867c58f"

See update tag2distrepo.keys for coreos-pool with f42 key: https://pagure.io/releng/issue/12264

coreos-installer changes

• Update coreos-installer to know about the signing key used for the future new major version of Fedora (N+1).
  • The current set of trusted signing keys is available at https://fedoraproject.org/security/.
  • If the Fedora (N+1) signing key isn't available yet at that site, you can also get it from https://src.fedoraproject.org/rpms/fedora-repos/tree/rawhide.
• Drop the signing key for the obsolete stable release (N-2).
• Add F42 key, drop F39 key: signing-keys: add Fedora 42 key, drop Fedora 39 key coreos-installer#1512

Example PR: coreos/coreos-installer#1113

Update rawhide stream

• Update manifest.yaml to list N+1 as the releasever (example PR) ([rawhide] manifest: Rawhide is now F42 fedora-coreos-config#3093)

Enable branched stream

• Update manifest.yaml to list N as the releasever (example PR) ([branched] Start tracking Fedora 41 fedora-coreos-config#3092)
• Update config.yaml to un-comment out the branched stream definition (example PR)

Misc

• Tag Ship dnf in FCOS and RHCOS #1687 with the meeting label to rediscuss it.

At Fedora (N) Beta

Update fedora-coreos-config next-devel

• Bump releasever in manifest.yaml ([next-devel] Start tracking fedora 41 fedora-coreos-config#3159)

• Add the fedora-candidate-compose repo in manifest.yaml ([next-devel] Start tracking fedora 41 fedora-coreos-config#3159)

• Update the repos in manifest.yaml if needed

• Run cosa fetch --dry-run --update-lockfile

  • this updates the x86_64 lockfile - the others will get updated when bump-lockfile runs.
  • in the future we may support this in cosa fetch directly

• PR the result ([next-devel] Start tracking fedora 41 fedora-coreos-config#3159)

• Re-enable next-devel if needed (docs)

  • (config: enable next-devel 2024-09 fedora-coreos-pipeline#1035)

• Disable branched stream since it is no longer needed. (config: disable branched stream fedora-coreos-pipeline#1036)

  • Update config.yaml to comment out the branched stream definition.

Ship rebased next

• Ship next
• Set a new update barrier for the final release of N-1 on next. In the barrier entry set a link to the docs. See discussion

Preparing for Fedora (N) GA

Do these steps as soon as we have a Go confirmation for GA, usually the Thursday of the week before GA.

Ship a final next release

If the packages in next-devel don't exactly match the last next release that was done, we need to do a release with the final GA content. This ensures that what we'll promote to testing has the exact content in GA (plus version fast-tracks). This usually happens on the Thursday of the announcement of Go.

• Ensure final next release has GA content

Build rebased testing and final stable release on N-1

• Build stable; promote it from the testing branch, which should still be on N-1. Don't release it yet (i.e. don't run the release job).
• Build testing; promote it from the next branch instead of testing-devel. Don't release it yet (i.e. don't run the release job).

Update fedora-coreos-config testing-devel

• Bump releasever in manifest.yaml
• Update the repos in manifest.yaml if needed
• Sync the lockfiles for all arches from next-devel
• Bump the base Fedora version in ci/buildroot/Dockerfile
• PR the result

At Fedora (N) GA

Do these steps on GA day.

Release rebased testing and final stable release on N-1

• Run the release job for the staged testing and stable builds and start rollout.
• Set a new update barrier for the final release of N-1 on testing. In the barrier entry set a link to the docs. See discussion

Disable next-devel stream if not needed

We prefer to disable next-devel when there is no difference between testing-devel and next-devel. This allows us to prevent wasting a bunch of resources (bandwidth, storage, compute) for no reason. After the switch to N if next-devel and testing-devel are in lockstep, then disable next-devel.

• Follow the instructions here to disable next-devel

Switch upstream packages to shipping release binaries from Fedora (N)

• Update repo-templates config.yaml with the version number and GPG key ID for Fedora (N).
  • config: update fedora version and signing key to 41 repo-templates#274

Disable the fedora-candidate-compose repo

• Remove from the manifest.yaml of next-devel the fedora-candidate-compose repo

After Fedora (N) GA

Ship rebased stable

• Ship stable
• Set a new update barrier for the final release of N-1 on stable. In the barrier entry set a link to the docs. See discussion

Untag old packages

koji untag N-2 packages from the pool (at some point we'll have GC in place to do this for us, but for now we must remember to do this manually or otherwise distRepo will fail once the signed packages are GC'ed). For example the following snippet finds all RPMs signed by the Fedora 32 key and untags them. Use this process:

• Find the key short hash. Usually found here. Then:

f32key=12c944d0
key=$f32key
echo > untaglist # create or empty out file
for build in $(koji list-tagged --quiet coreos-pool | cut -f1 -d' '); do
    if koji buildinfo $build | grep $key 1>/dev/null; then
        echo "Adding $build to untag list"
        echo "${build}" >> untaglist
    fi
done

Now we have a list of builds to untag. But we need a few more sanity checks.

• Make sure none of the builds are used in N based FCOS. Check by running:

f32key=12c944d0
key=$f32key
podman run -it --rm quay.io/fedora/fedora-coreos:testing-devel rpm -qai | grep -B 9 $key
podman rmi quay.io/fedora/fedora-coreos:testing-devel

If there are any RPMs signed by the old key they'll need to be investigated. Maybe they shouldn't be used any longer. Or maybe they're still needed. One example of this is the shim RPM where the same build could be used for many Fedora releases. In this case you'll need to untag the RPM from coreos-pool, run a koji distrepo, which will remove that RPM from the repo metadata, and then re-tag it into the pool. The RPM in the repo will now be signed with a newer signing key.

• After verifying the list looks good, untag:

# use xargs so we don't exhaust bash string limit
cat untaglist | xargs -L50 koji untag-build -v coreos-pool

• Now that untagging is done, give a heads up to rpm-ostree developers that N-2 packages have been untagged and that they may need to update their CI compose tests to freeze on a newer FCOS commit.

• Remove the N-2 signing key from the tag info for the coreos-pool tag. The following commands view the current settings and then update the list to the 33/34/35 keys. You'll most likely have to get someone from releng to run the second command (edit-tag).

  • koji taginfo coreos-pool
  • koji edit-tag coreos-pool -x tag2distrepo.keys="9570ff31 45719a39 9867c58f"

Open ticket for the next Fedora rebase

• Create a new ticket from the rebase template
  • label with FN label where N is the Fedora version.

Miscellaneous container updates

These are various containers in use throughout our ecosystem. We should update or open a ticket to track updating them once a new Fedora release is out. If you open a ticket instead of doing the update add a link to the ticket as comment.

• Update coreos-assembler or open ticket to update:
  • Dockerfile
    • (Bump to Fedora 41 coreos-assembler#3975)
• Update coreos-installer
  • Dockerfile
    • (dockerfile: update to fedora:41 coreos-installer#1566)
• Update Ignition
  • Dockerfile.validate
    • (Dockerfile.validate: update to fedora:41 ignition#1982)
• Update Butane
  • Dockerfile
    • (Dockerfile: update to Fedora 41 butane#565)
• Update fedora-coreos-cincinnati
  • Dockerfile
    • (Dockerfile: update to Fedora 41 fedora-coreos-cincinnati#102)
  • https://pagure.io/fedora-infra/ansible/pull-request/1843
    • ImageStream
    • BuildConfig
  • Git Hash Variables (Optional)
    • https://pagure.io/fedora-infra/ansible/pull-request/2397
• Update config-bot
  • Dockerfile
    • (various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191)
• Update coreos-koji-tagger
  • Dockerfile
    • (various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191)
  • https://pagure.io/fedora-infra/ansible/pull-request/1843
    • ImageStream
    • BuildConfig
• Update coreos-ostree-importer
  • Dockerfile
    • (various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191)
  • https://pagure.io/fedora-infra/ansible/pull-request/1843
    • ImageStream
    • BuildConfig
• Update fedora-ostree-pruner
  • Dockerfile
    • (various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191)
  • https://pagure.io/fedora-infra/ansible/pull-request/1843
    • ImageStream
    • BuildConfig
• Update openshift/os
  • Dockerfile
    • (NO-JIRA: extensions/Dockerfile: Update to Fedora 41 openshift/os#1655)

[marmijo]

• Update rawhide manifest: [rawhide] manifest: Rawhide is now F42 fedora-coreos-config#3093
• Updates to branched stream: [branched] Start tracking Fedora 41 fedora-coreos-config#3092

[marmijo]

Fedora Releng ticket requesting permissions for @marmijo to perform the "add and tag package into f42-coreos-continuous tag" step: https://pagure.io/releng/issue/12263

[marmijo]

Fedora Releng ticket to add the F42 signing key short hash to the coreos-pool tag: https://pagure.io/releng/issue/12264

[marmijo]

Add the Fedora 42 signing key: coreos/coreos-installer#1512

[travier]

Package diff initial investigation in coreos/fedora-coreos-config#3092 (comment):

Copying the diff here for reference:

[2024-08-19T17:04:44.229Z] Downgraded:
[2024-08-19T17:04:44.234Z]   json-glib 1.8.0-3.fc40 -> 1.8.0-1.fc40
[2024-08-19T17:04:44.234Z] Removed:
[2024-08-19T17:04:44.234Z]   atheros-firmware-20240220-1.fc40.noarch
[2024-08-19T17:04:44.234Z]   bind-license-32:9.18.21-4.fc40.noarch
[2024-08-19T17:04:44.234Z]   brcmfmac-firmware-20240220-1.fc40.noarch
[2024-08-19T17:04:44.234Z]   fuse-2.9.9-21.fc40.x86_64
[2024-08-19T17:04:44.234Z]   fuse-libs-2.9.9-21.fc40.x86_64
[2024-08-19T17:04:44.234Z]   gvisor-tap-vsock-gvforwarder-6:0.7.3-2.fc40.x86_64
[2024-08-19T17:04:44.234Z]   mt7xxx-firmware-20240220-1.fc40.noarch
[2024-08-19T17:04:44.234Z]   pigz-2.8-4.fc40.x86_64
[2024-08-19T17:04:44.234Z]   polkit-pkla-compat-0.1-28.fc40.x86_64
[2024-08-19T17:04:44.234Z]   realtek-firmware-20240220-1.fc40.noarch
[2024-08-19T17:04:44.234Z] Added:
[2024-08-19T17:04:44.234Z]   clevis-pin-tpm2-0.5.3-7.fc41.x86_64
[2024-08-19T17:04:44.234Z]   dnf5-5.2.5.0-2.fc41.x86_64
[2024-08-19T17:04:44.234Z]   docker-cli-27.1.1-3.fc41.x86_64
[2024-08-19T17:04:44.234Z]   kdump-utils-1.0.44-2.fc41.x86_64
[2024-08-19T17:04:44.234Z]   libdnf5-5.2.5.0-2.fc41.x86_64
[2024-08-19T17:04:44.234Z]   libdnf5-cli-5.2.5.0-2.fc41.x86_64
[2024-08-19T17:04:44.234Z]   libkcapi-hasher-1.5.0-3.fc41.x86_64
[2024-08-19T17:04:44.234Z]   libtextstyle-0.22.5-6.fc41.x86_64
[2024-08-19T17:04:44.234Z]   lld-libs-18.1.8-2.fc41.x86_64
[2024-08-19T17:04:44.234Z]   llvm-libs-18.1.8-2.fc41.x86_64
[2024-08-19T17:04:44.234Z]   makedumpfile-1.7.5-13.fc41.x86_64
[2024-08-19T17:04:44.234Z]   moby-filesystem-27.1.1-3.fc41.x86_64
[2024-08-19T17:04:44.234Z]   qed-firmware-20240811-2.fc41.noarch
[2024-08-19T17:04:44.234Z]   sdbus-cpp-1.5.0-3.fc41.x86_64
[2024-08-19T17:04:44.234Z]   tini-static-0.19.0-9.fc41.x86_64

Downgrade is https://bugzilla.redhat.com/show_bug.cgi?id=2297094 which we should directly make a PR to fix as it's really minor and maybe we should just ignore it.

Removed:

• fuse is related to packaging/rpm-ostree.spec: Update to sync with rawhide rpm-ostree#5047
• polkit-pkla-compat removal should be investigated
• pigz removal might need investigation
• The firmware removal are part of the wifi firmware cleanup if I'm not mistaken so should be good

Added:

• qed-firmware is unexpected
• sdbus-cpp is unexpected
• the rest looks dnf5, docker or podman related and should be good.

[travier]

Fixing the json-glib pin in https://src.fedoraproject.org/rpms/json-glib/pull-request/4

[travier]

#1785 (comment)

[travier]

We've discussed this topic in today's community meeting.

The priority is resolving the issues that we ave with systemd 256 to be able to unpin it. Then looking at the status of composefs and kdump.

[yasminvalim]

We've discussed this topic in today's community meeting.

[marmijo]

• Update branched manifest to track Fedora 41: [next-devel] Start tracking fedora 41 fedora-coreos-config#3159
• Re-enable next-devel: config: enable next-devel 2024-09 fedora-coreos-pipeline#1035
• Disable branched: config: disable branched stream fedora-coreos-pipeline#1036

[marmijo]

As we have done in the past we will be fast-tracking packages in next-devel/next to ensure no upgrade transition will ever include downgraded packages.

[dustymabe]

koji edit-tag coreos-pool -x tag2distrepo.keys="a15B79cc e99d6ad1 105ef944"

got Kevin Fenzi to run this for me:

koji edit-tag coreos-pool -x tag2distrepo.keys="a15B79cc e99d6ad1 105ef944"

[dustymabe]

all 709 F39 RPMs were removed from coreos-pool.

[dustymabe]

Update coreos-assembler or open ticket to update:

Done in coreos/coreos-assembler#3975

[marmijo]

Container updates:

• coreos-installer
  • dockerfile: update to fedora:41 coreos-installer#1566
• ignition
  • Dockerfile.validate: update to fedora:41 ignition#1982
• config-bot
  • various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191
• fedora-coreos-cincinatti
  • Dockerfile: update to Fedora 41 fedora-coreos-cincinnati#102
  • https://pagure.io/fedora-infra/ansible/pull-request/2408
• coreos-koji-tagger
  • various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191
  • https://pagure.io/fedora-infra/ansible/pull-request/2408
• coreos-ostree-importer
  • various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191
  • https://pagure.io/fedora-infra/ansible/pull-request/2408
• fedora-ostree-pruner
  • various CoreOS tools: Update to Fedora 41 fedora-coreos-releng-automation#191
  • https://pagure.io/fedora-infra/ansible/pull-request/2408

[marmijo]

Update repo-templates config.yaml with the version number and GPG key ID for Fedora (N).

coreos/repo-templates#274

[marmijo]

The rebase to Fedora 41 is complete.