Jobs fail with "Unable to find process isolation executable: podman"

[mensler]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

I was running the docker-compose installation today and now all jobs fail with

Worker output:
Unable to find process isolation executable: podman

Podman has been installed with version 5.0.2 compared to last week's 5.0.0.
When I try to invoke podman manually in the tools_awx_1 container I get the following error:

ERRO[0000] path "/var/lib/awx/.config" exists and it is not owned by the current user

Not sure if this is really the root cause tbh, but might be introduced with this change: containers/storage@1d38a27, since the /var/lib/awx/.config dir belongs to root.

After downgrading podman to 5.0.0 again, it works as expected.

AWX version

24.3.1

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

docker development environment

Modifications

no

Ansible version

core 2.15.11

Operating system

Amazon Linux 2023

Web browser

No response

Steps to reproduce

Follow the docker-compose installation instructions. When finished start a job, e.g. a source control update.

Expected results

Job executed normally.

Actual results

Fails with
Worker output:
Unable to find process isolation executable: podman

Additional information

From the install logs last week:

#32 [stage-1 16/36] RUN dnf install -y podman && rpm --restore shadow-utils 2>/dev/null
#32 1.161 Last metadata expiration check: 0:04:57 ago on Tue 07 May 2024 04:07:24 AM UTC.
#32 1.328 Dependencies resolved.
#32 1.332 ================================================================================
#32 1.332  Package                    Arch       Version              Repository     Size
#32 1.332 ================================================================================
#32 1.332 Installing:
#32 1.332  podman                     x86_64     2:5.0.0-1.el9        appstream      16 M
#32 1.332 Installing dependencies:
#32 1.332  conmon                     x86_64     3:2.1.10-2.el9       appstream      52 k
#32 1.332  containers-common          x86_64     2:1-61.el9           appstream     142 k
#32 1.332  fuse-overlayfs             x86_64     1.13-1.el9           appstream      67 k
#32 1.332  fuse3                      x86_64     3.10.2-8.el9         appstream      54 k
#32 1.332  fuse3-libs                 x86_64     3.10.2-8.el9         appstream      92 k
#32 1.332  iptables-libs              x86_64     1.8.10-2.el9         baseos        461 k
#32 1.332  iptables-nft               x86_64     1.8.10-2.el9         baseos        204 k
#32 1.332  jansson                    x86_64     2.14-1.el9           baseos         46 k
#32 1.332  libnetfilter_conntrack     x86_64     1.0.9-1.el9          baseos         59 k
#32 1.332  libnfnetlink               x86_64     1.0.1-21.el9         baseos         30 k
#32 1.332  libnftnl                   x86_64     1.2.6-2.el9          baseos         84 k
#32 1.332  libslirp                   x86_64     4.4.0-7.el9          appstream      69 k
#32 1.332  netavark                   x86_64     2:1.10.3-1.el9       appstream     4.0 M
#32 1.332  nftables                   x86_64     1:1.0.9-1.el9        baseos        424 k
#32 1.332  shadow-utils-subid         x86_64     2:4.9-8.el9          baseos         87 k
#32 1.332  slirp4netns                x86_64     1.2.3-1.el9          appstream      47 k
#32 1.332 Installing weak dependencies:
#32 1.332  aardvark-dns               x86_64     2:1.9.0-1.el9        appstream     1.0 M

Today:

#32 [stage-1 16/36] RUN dnf install -y podman && rpm --restore shadow-utils 2>/dev/null
#32 1.334 Last metadata expiration check: 0:05:22 ago on Tue 14 May 2024 04:12:51 AM UTC.
#32 1.508 Dependencies resolved.
#32 1.512 ================================================================================
#32 1.512  Package                 Arch    Version                       Repository  Size
#32 1.512 ================================================================================
#32 1.512 Installing:
#32 1.512  podman                  x86_64  2:5.0.2-1.el9                 appstream   16 M
#32 1.512 Installing dependencies:
#32 1.512  conmon                  x86_64  3:2.1.10-2.el9                appstream   52 k
#32 1.512  containers-common       x86_64  2:1-61.el9                    appstream  142 k
#32 1.512  fuse-overlayfs          x86_64  1.13-1.el9                    appstream   67 k
#32 1.512  fuse3                   x86_64  3.10.2-8.el9                  appstream   54 k
#32 1.512  fuse3-libs              x86_64  3.10.2-8.el9                  appstream   92 k
#32 1.512  iptables-libs           x86_64  1.8.10-2.el9                  baseos     461 k
#32 1.512  iptables-nft            x86_64  1.8.10-2.el9                  baseos     204 k
#32 1.512  jansson                 x86_64  2.14-1.el9                    baseos      46 k
#32 1.512  libnetfilter_conntrack  x86_64  1.0.9-1.el9                   baseos      59 k
#32 1.512  libnfnetlink            x86_64  1.0.1-21.el9                  baseos      30 k
#32 1.512  libnftnl                x86_64  1.2.6-2.el9                   baseos      84 k
#32 1.512  libslirp                x86_64  4.4.0-7.el9                   appstream   69 k
#32 1.512  netavark                x86_64  2:1.10.3-1.el9                appstream  4.0 M
#32 1.512  nftables                x86_64  1:1.0.9-1.el9                 baseos     424 k
#32 1.512  passt                   x86_64  0^20231204.gb86afe3-1.el9     appstream  186 k
#32 1.512  shadow-utils-subid      x86_64  2:4.9-8.el9                   baseos      87 k
#32 1.512  slirp4netns             x86_64  1.2.3-1.el9                   appstream   47 k
#32 1.512 Installing weak dependencies:
#32 1.512  aardvark-dns            x86_64  2:1.9.0-1.el9                 appstream  1.0 M

[TheRealHaoLiu]

should be fixed with #15188
can u try again (please close the issue if is fixed, thanks!)

[mensler]

Yes, it works again. Thanks!