After podman 2 upgrade, systemd fails to start in containers on cgroups v1 hosts #6734
Comments
openshift-ci-robot
added
the
kind/bug
|
can you make the image in question available ? |
|
No. |
|
@baude Any idea why ports could be assigned to a second "pause" container instead of the intended one? |
|
How is the pod created? Can you provide the command that was used to launch the pod? |
|
Also, |
|
I used a
Here's the I don't see an option to run |
|
podman pod inspect |
|
any chance we can sync up on irc? freenode.net #podman |
|
btw, can couple of simple things we should have asked. apologies if i missed the information.
|
Did I miss this in the docs? It's not intuitive to have port mappings appear on a container other than the one I installed. I wasn't thrilled to see the "pause" container from a third-party service on the internet that I had no intention of pulling down content from either. |
No. I presume that means I happened to break my own container about the time I also upgraded |
|
Network mode is set to another container, which I'm assuming is the infra container (I don't see the ID in question in your first What init are you using in the container, systemd or something else? |
|
@baude One obvious thing: 1.9: 2.0: |
|
Hm. It's also ordering containers incorrectly... I'd expect sort to be by time of creation, not by ID. |
|
I'm using |
|
I built a test setup as close to yours as I could given provided information (pod with port 2222 forwarded, container in that pod with systemd as init + sshd, added a user, SSH'd in from another machine to public port, all rootless) and everything worked locally, so I think this is either environment, or some detail of the pod that is not clear from what is given here. |
|
I'm on Kubernetes Slack server now. I forgot my IRC password. |
|
@mheon Thanks for the attention. I'll test more with Docker as a control group reference and see if I can pinpoint some bug on my end that I introduced. |
|
It booted fine with The plot thickens. I'll see if I can some more useful case for you to reproduce from. |
|
I've temporarily posted my Dockerfile here: https://gist.github.com/markstos/9f7b982bc73106e4bb5a73e5524a3ec6 Once you've grabbed it, I'm going to take down the Gist. |
|
I believe the last two things I was changing before it broke were setting fish_user_paths, and looping over users to add their SSH keys to authorized_keys-- both happen in the last 20 lines of the file. |
|
Grabbed, thanks. It's a little late in the day here, but I'll pick this up tomorrow and see if I can chase something down. Might be a compose-specific bug, or might be a result of upgrading an existing 1.9 system to 2.0 |
|
I've reduced the test a case a bit. Here's a script I successfully used to launch the container with 1.9 that fails with 2.0: The result is the same-- it starts without apparent error, but I can't SSH in. This eliminates anything to do with pods. Using I'm going to try to rollback recent changes to my Dockerfile assuming that my changes broke it, not |
|
I'd recommend checking the journal within the container to see why sshd is failing. Also, checking if port forwarding works at all would be helpful - if you use 8080:80 with a simple nginx container, can you access it? |
|
Partial fix for the |
|
@mheon how I can check the journal in the container if I can't get into it? I tried this to narrow down the issue: I rewrote my I don't think port-forwarding is the issue, since |
|
@markstos |
|
@giuseppe PTAL |
|
if reverting (#6569) solves your issue, you can force a new scope wrapping podman with In your case it will be: |
Thanks for this. It's useful for Molecule users with this problem. Molecule works again with Podman 2 on Ubuntu when running |
|
Note that we've got this issue flagged as something to be fixed before we switch to podman 2.x in Fedora CoreOS. Is there any resolution or more information that we should be using to inform our decision here? Context: coreos/fedora-coreos-tracker#575 |
|
@giuseppe is this something you can look at? |
|
PR: #7339 Can anyone who is on cgroup v1 please try it? |
|
@dustymabe ^^ Mind testing this? |
create a scope everytime we don't own the current cgroup and we are running on systemd. Closes: containers#6734 Signed-off-by: Giuseppe Scrivano <[email protected]>
|
I can test if someone gives me a link to an RPM. Sorry for the delayed response. |
create a scope everytime we don't own the current cgroup and we are running on systemd. Closes: containers#6734 Signed-off-by: Giuseppe Scrivano <[email protected]>
|
I tried compiling podman today Rootless with systemd-run: Rootfull works however: |
…md-run. Presumably fixed by containers/podman#7339 for containers/podman#6734.
…md-run. Presumably fixed by containers/podman#7339 for containers/podman#6734.
…md-run. Presumably fixed by containers/podman#7339 for containers/podman#6734.
Using rootless podman on CentOS 8.3 is failing for us, due to [1][2]. We could try to force molecule to use "systemd-run --user --scope podman" instead of "podman", but running as root works as well. [1] - https://bugzilla.redhat.com/show_bug.cgi?id=1880987 [2] - containers/podman#6734 Change-Id: I30305b4396a849a4cefc4c080b3fa6be604adc79
github-actions
bot
added
the
locked - please file new issue/PR
and others
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
I was repeatedly building working containers with podman this morning when my OS (Ubuntu 20.04) notified me that podman 2.0 was available and I elected to install it.
Shortly afterword, I can no longer SSH to a newly build and launched container. I see this as output to
podman container list -a:This is frustrating: I don't any references to a container named "pause", yet one is running and listening on the ports my container had published, yet my container isn't listening on any ports at all.
I read the
podman2.0 release notes and don't see any notes about a related breaking change.I did search the project for references to "infra containers" because I sometimes see that term mentioned in error messages. I find references to "infra containers" in the code, but I can't find references in the documentation.
They seem related to this issue and it would be great if there was more accessible user documentation about "infra containers"
Steps to reproduce the issue:
Describe the results you received:
Initializing machine ID from random generator.
Failed to create /user.slice/user-1000.slice/session-8.scope/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
[!!!!!!] Failed to allocate manager object.
Describe the results you expected:
For this test, the container should boot to the point where this line appears:
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Additional environment details (AWS, VirtualBox, physical, etc.):
The text was updated successfully, but these errors were encountered: