Podman regression in Ubuntu 22.04 image released Feb 25th?

[fwilhe]

Description

I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.

The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated.

Our build fails with this error link to run:

STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac
Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac...
Getting image source signatures
Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1
Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1
Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5
Writing manifest to image destination
Storing signatures
COMMIT localhost/builder
--> 1eba10d0345
Successfully tagged localhost/builder:latest
Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac
1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5
Error: OCI runtime error: chmod `run/shm`: Operation not supported
Error: Process completed with exit code 126.

This is with this image version:

Current runner version: '2.313.0'
Operating System
  Ubuntu
  22.04.4
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20240225.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1

Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear.

Tried to reproduce in this repo, but I'm not able to get this with image version 20240225.1.0.

Is this a known issue and version 20240225.1.0 is not in use anymore?

This blog post seems to suggest that the crun version is too old.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Fails with: 20240225.1.0
Works with: 20240218.1.0

Failing build: https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535

Is it regression?

yes

Expected behavior

Podman build should work

Actual behavior

Podman build fails with message:

Error: OCI runtime error: chmod `run/shm`: Operation not supported

Repro steps

Create a workflow running on ubuntu 22.04 and execute those steps:

    - name: install required packages
      shell: bash
      run: |
        sudo rm -rf /{usr/{local/{lib/{android,heroku},.ghcup,share/{dotnet,powershell,miniconda,swift}},share/{dotnet,miniconda,swift}},opt/{hostedtoolcache,microsoft},imagegeneration}
        sudo docker system prune -a -f
        sudo apt-get update
        sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends qemu-user-static

    - run: |
        git clone https://github.com/gardenlinux/gardenlinux
        cd gardenlinux
        ./build base-amd64 base-arm64

[shamil-mubarakshin]

Hey @fwilhe,
We are looking into this

[fwilhe]

Thanks @shamil-mubarakshin

I think I have even more evidence that it broke in that specific image update.

In this job run it seems like different image versions were mixed randomly?

Looking at a few failed and a few successful jobs, the pattern is clear:

Failed:

Image Version: 20240225.1.0 Job Name: build / images (arm64, metal_secureboot_readonly)
Image Version: 20240225.1.0 Job Name: build / images (arm64, metal_secureboot_readonly_persistence)

Worked:

Image Version: 20240218.1.0 Job Name: build / images (amd64, metal)
Image Version: 20240218.1.0 Job Name: build / images (amd64, kvm_secureboot)

Is there a way to force a specific image version in the runs-on stanza?

Let me know if I can help with this further.

[shamil-mubarakshin]

@fwilhe, thanks for the provided blogpost link, looks like following commit has made it to 6.5.0-1015-azure kernel. crun is indeed installed as a podman dependency from official ubuntu repo and there is no backport yet.
Could you try adding following workaround:

    - name: patch crun
      shell: bash
      run: |
        curl -Lo ./crun https://github.com/containers/crun/releases/download/1.14.3/crun-1.14.3-linux-amd64
        sudo install crun /usr/bin/crun

Unfortunately, image version cannot be selected in runs-on. Self-hosted runners can be used to run workflows with custom software/hardware.

[fwilhe]

I was hoping we could avoid manually messing with the installed components, as also the podman version is quite old..

If I understand this correctly, there is no way to use podman build successfully on ubuntu 22.04 with a 6.5 linux kernel, right? That's a bummer.

Is there any chance that podman and crun will be updated in the ubuntu 22.04 image? I understand that you'd like to avoid to use it from non-official repos, but for the time being that sounds like it just won't work.

[mafalb]

Is there any chance that podman and crun will be updated in the ubuntu 22.04 image? I understand that you'd like to avoid to use it from non-official repos, but for the time being that sounds like it just won't work.

I am affected too but I am asking for another solution: Is there any chance that this incompatible kernel is rolled back? The logic would be like that: incompatible kernel caused this issue, obvious fix for this is reverting. Seriously.

[mikhailkoliada]

We can not roll the kernel back, unfortunately

[fwilhe]

We can not roll the kernel back, unfortunately

Too bad. Any updates on an updated version of crun and/or podman in that image that fixes the issue?

[shamil-mubarakshin]

@fwilhe, podman, buildah, skopeo with their dependencies are installed using apt-get from official ubuntu repo. At the moment there are no plans on changing the installation method. You could contact ubuntu to get updates to 22.04. In the meantime please use workaround.

[mafalb]

One thought: as an alternative workaround you could try using runc instead of crun

$ apt install runc
$ podman run -ti --runtime runc systemd-container

[fauust]

Hi, I am also impacted by this, see:

• https://github.com/fauust/docker-systemd/actions/runs/8139697623
• https://github.com/fauust/ansible-role-mariadb/actions/runs/8151318185

I'll give the workaround a try.

[fauust]

Just opened https://bugs.launchpad.net/cloud-images/+bug/2056442

[j5awry]

Hi, thanks for opening the bug in cloud-images. I'll get podman and crun added to the bug as well. If i'm understanding all this correctly, there is a change in the 6.5 kernel which causes a break in podman, specifically with it's dependency on crun, and a fix is needed.

podman and crun are both universe packages. Universe packages are considered to have "community" support, meaning the majority of work comes from the upstream packaging in Debian. looks like the version sync'd from Debian at Ubuntu 22.04 release (3.4.4) isn't available in Debian anymore, and based on the versioning, i'm going to guess that it already has some patches applied. crun Ubuntu has the same version as Bullseye, so it may be worth seeing if this is reproducible in Debian Bullseye. If so, we could see about getting an upstream patch applied.

Let me get try and get the bug assigned to the correct packages, and see if there is anyone that can help out.

[j5awry]

quick update -- follow on the launchpad bug: https://bugs.launchpad.net/cloud-images/+bug/2056442

we're already working on it. and thanks to @fauust for opening the bug. For anyone following, if you ever hit something in a cloud-based Ubuntu image on one of the big public clouds, feel free to open a bug in https://bugs.launchpad.net/cloud-images . I promise we check, and will try to get things routed to the correct places if it's not an image problem :)

[fwilhe]

Thanks @j5awry , @fauust

Much appreciated

[fauust]

podman run --systemd always seems to be another workaround.
Apparently not, see below.

BTW, the fix is on it's way, see: https://bugs.launchpad.net/cloud-images/+bug/2056442

[dillfrescott]

podman run --systemd always seems to be another workaround.

[dillfrescott]

One thought: as an alternative workaround you could try using runc instead of crun

$ apt install runc
$ podman run -ti --runtime runc systemd-container

This solution works though!

[fwilhe]

As the new 24.04 image is now in beta this is the way forward I guess (maybe after the image is not beta anymore)