Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman: broken network functionality for CNI plugins #7753

Closed
2 of 10 tasks
jordimassaguerpla opened this issue Jun 20, 2023 · 12 comments
Closed
2 of 10 tasks

podman: broken network functionality for CNI plugins #7753

jordimassaguerpla opened this issue Jun 20, 2023 · 12 comments
Assignees
@ilia-shipitsin
Labels
awaiting-deployment Code complete; awaiting deployment and/or deployment in progress bug report external OS: Ubuntu

Comments

@jordimassaguerpla
Copy link

Description

Since yesterday, I see this error

plugin bridge does not support config version "1.0.0" plugin portmap does not support config version "1.0.0" plugin firewall does not support config version "1.0.0" plugin tuning does not support config version "1.0.0"

I believe is a bug in the latest ubuntu update of podman https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2024394

Platforms affected

  • Azure DevOps
  • GitHub Actions - Standard Runners
  • GitHub Actions - Larger Runners

Runner images affected

  • Ubuntu 20.04
  • Ubuntu 22.04
  • macOS 11
  • macOS 12
  • macOS 13
  • Windows Server 2019
  • Windows Server 2022

Image version and build link

?

Is it regression?

yes

Expected behavior

2 days ago I had no error and network was working as expected with podman rootfull.

Actual behavior

Network does not work see https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2024394

Repro steps

podman network create -d bridge test-net
podman network ls
@ilia-shipitsin
Copy link
Contributor

interesting.

do you think it makes sense to add the following to image testing ?

podman network create -d bridge test-net
podman network ls

@ilia-shipitsin
Copy link
Contributor

btw, is it possible to downgrade podman using apt-get ? it would help to unblock users before podman fix is released

@jordimassaguerpla
Copy link
Author

I just realized that I was doing "sudo apt-get update && sudo apt-get -y install podman" . If I remove this line, the issue is not happening. I think by running apt-get I was installing the latest podman version, which is the broken one. I did not know the base image contained podman. Thus, by not running the apt-get command it works... but we do not have the latest updates from podman. However, it will buy us some time, at least until it is fixed in Ubuntu.

@jordimassaguerpla jordimassaguerpla mentioned this issue Jun 20, 2023
Merged
12 tasks
@ilia-shipitsin
Copy link
Contributor

it is similar to what we do during image generation. thus we could deploy buggy podman.

we test podman, but just "podman -v"

pbrezina added a commit to pbrezina/sssd-ci-containers that referenced this issue Jun 21, 2023
@pbrezina
action: do not install packages in setup
03a4ed3 
There is a regression in podman that fails to create networks. The
workaround is to avoid update of the podman package. We can skip
the installation since both podman and docker-compose are available
in the base image.

See: actions/runner-images#7753
@pbrezina pbrezina mentioned this issue Jun 21, 2023
Merged
pbrezina added a commit to pbrezina/sssd-ci-containers that referenced this issue Jun 21, 2023
@pbrezina
actions: do not install packages in setup
b0665a2 
There is a regression in podman that fails to create networks. The
workaround is to avoid update of the podman package. We can skip
the installation since both podman and docker-compose are available
in the base image.

See: actions/runner-images#7753
pbrezina added a commit to pbrezina/sssd-ci-containers that referenced this issue Jun 21, 2023
@pbrezina
actions: do not install packages in setup
477efa5 
There is a regression in podman that fails to create networks. The
workaround is to avoid update of the podman package. We can skip
the installation since both podman and docker-compose are available
in the base image.

See: actions/runner-images#7753
pbrezina added a commit to pbrezina/sssd-ci-containers that referenced this issue Jun 21, 2023
@pbrezina
actions: do not install packages in setup
517db86 
There is a regression in podman that fails to create networks. The
workaround is to avoid update of the podman package. We can skip
the installation since both podman and docker-compose are available
in the base image.

See: actions/runner-images#7753
pbrezina added a commit to SSSD/sssd-ci-containers that referenced this issue Jun 21, 2023
@pbrezina
actions: do not install packages in setup
515b1ce 
There is a regression in podman that fails to create networks. The
workaround is to avoid update of the podman package. We can skip
the installation since both podman and docker-compose are available
in the base image.

See: actions/runner-images#7753
@pbrezina
Copy link

Unfortunately the workaround does not work anymore in 20230619.1.0, it worked in 20230611.1.

@ilia-shipitsin
Copy link
Contributor

I can imagine that apt-get install podman=3.4.3 may work (3.4.3 is just an example, proper version ought to be specified)

@ilia-shipitsin
Copy link
Contributor

@jordimassaguerpla , can you please help with podman ?

image

is it possible to make it return non zero ?

pbrezina added a commit to pbrezina/sssd-ci-containers that referenced this issue Jun 21, 2023
@pbrezina
action: do not install packages in setup
2cba646 
There is a regression in podman that fails to create networks. The
workaround is to avoid update of the podman package. We can skip
the installation since both podman and docker-compose are available
in the base image.

See: actions/runner-images#7753
@pbrezina pbrezina mentioned this issue Jun 21, 2023
Closed
pbrezina added a commit to SSSD/sssd-ci-containers that referenced this issue Jun 21, 2023
@pbrezina
action: install older version of podman to workaround a regression
97c1e38 
There is a regression in podman that fails to create networks. The
workaround is to downgrade the podman package.

See: actions/runner-images#7753
@pbrezina
Copy link

I can imagine that apt-get install podman=3.4.3 may work (3.4.3 is just an example, proper version ought to be specified)

E: Version '3.4.3' for 'podman' was not found

@ilia-shipitsin ilia-shipitsin mentioned this issue Jun 21, 2023
Merged
4 tasks
@ilia-shipitsin
Copy link
Contributor

I can imagine that apt-get install podman=3.4.3 may work (3.4.3 is just an example, proper version ought to be specified)

E: Version '3.4.3' for 'podman' was not found

yes, that was an example.

the exact command may be constructed from launchpad issue which describes known working version

 apt install podman=3.4.4+ds1-1ubuntu1 --allow-downgrades

@ilia-shipitsin
Copy link
Contributor

@jordimassaguerpla , can you please review 43936f0 ?

pbrezina added a commit to pbrezina/sssd-ci-containers that referenced this issue Jun 22, 2023
@pbrezina
actions: workaround regression in podman
a5e34e3 
actions/runner-images#7753
@pbrezina pbrezina mentioned this issue Jun 22, 2023
Closed
rm3l added a commit to rm3l/odo that referenced this issue Jun 23, 2023
@rm3l
wip: try the workaround suggested in [1]
a0abbb8 
There seems to be an issue with Podman (libpod specifically) on Ubuntu 22.04,
which might affect networking.
See [1] and [2]

[1] actions/runner-images#7753
[2] https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2024394
@rm3l rm3l mentioned this issue Jun 23, 2023
Merged
3 tasks
openshift-merge-robot pushed a commit to redhat-developer/odo that referenced this issue Jun 23, 2023
@rm3l
Temporarily downgrade Podman on GitHub to fix CI failures for Podman …
d9d868f 
…tests (#6928)

* WIP: check Podman version

* wip: try the workaround suggested in [1]

There seems to be an issue with Podman (libpod specifically) on Ubuntu 22.04,
which might affect networking.
See [1] and [2]

[1] actions/runner-images#7753
[2] https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2024394
This was referenced Jun 23, 2023
Merged
Merged
@jordimassaguerpla
Copy link
Author

@jordimassaguerpla , can you please review 43936f0 ?

Done. Sorry I could not do it last week.

@jordimassaguerpla jordimassaguerpla mentioned this issue Jun 27, 2023
Merged
12 tasks
fauust added a commit to fauust/ansible-role-mariadb that referenced this issue Jun 28, 2023
@fauust
Downgrade podman to make the GitHub runner work again
b6faaa8 
See: actions/runner-images#7753
fauust added a commit to fauust/ansible-role-mariadb that referenced this issue Jun 28, 2023
@fauust
Downgrade podman to make the GitHub runner work again
d4867d5 
See: actions/runner-images#7753
pbrezina added a commit to pbrezina/sssd-ci-containers that referenced this issue Jun 30, 2023
@pbrezina
actions: workaround regression in podman
5681d1b 
actions/runner-images#7753
@pbrezina pbrezina mentioned this issue Jun 30, 2023
Merged
pbrezina added a commit to SSSD/sssd-ci-containers that referenced this issue Jun 30, 2023
@pbrezina
actions: workaround regression in podman
7a05943 
actions/runner-images#7753
ilia-shipitsin added a commit to ilia-shipitsin/runner-images that referenced this issue Sep 3, 2023
@ilia-shipitsin
[ubuntu] pin podmanm, add testcase for podman networking
bef4269 
Related: actions#7753
mikhailkoliada pushed a commit that referenced this issue Sep 4, 2023
@ilia-shipitsin
[ubuntu] pin podmanm, add testcase for podman networking (#7763)
1a30e20 
Related: #7753
@ilia-shipitsin ilia-shipitsin added the awaiting-deployment Code complete; awaiting deployment and/or deployment in progress label Sep 4, 2023
@mikhailkoliada
Copy link
Contributor

Workaround included into the image

@pbrezina pbrezina mentioned this issue Sep 15, 2023
Closed
fauust added a commit to fauust/ansible-role-mariadb that referenced this issue Sep 20, 2023
@fauust
actions/runner-images#7753 fixed
2887615
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ilia-shipitsin ilia-shipitsin

Labels
awaiting-deployment Code complete; awaiting deployment and/or deployment in progress bug report external OS: Ubuntu
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jordimassaguerpla @pbrezina @mikhailkoliada @ilia-shipitsin @shamil-mubarakshin