Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman: podman rm -f doesn't leave processes #17040

Merged
merged 2 commits into from
Jan 10, 2023
Merged

podman: podman rm -f doesn't leave processes #17040

merged 2 commits into from
Jan 10, 2023

Conversation

giuseppe
Copy link
Member

@giuseppe giuseppe commented Jan 9, 2023

follow-up to 6886e80

when "podman -rm -f" is used on a container in "stopping" state, also make sure it is terminated before removing it from the local storage.

Signed-off-by: Giuseppe Scrivano [email protected]

Does this PR introduce a user-facing change?

now podman rm -f also terminates containers in "stopping" state

@openshift-ci openshift-ci bot added release-note approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Jan 9, 2023
@mheon
Copy link
Member

mheon commented Jan 9, 2023

LGTM

@baude
Copy link
Member

baude commented Jan 9, 2023

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 9, 2023
@vrothberg
Copy link
Member

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 9, 2023
vrothberg
vrothberg reviewed Jan 9, 2023
View reviewed changes
Copy link
Member

@vrothberg vrothberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@edsantiago PTAL

edsantiago
edsantiago reviewed Jan 9, 2023
View reviewed changes
test/system/055-rm.bats Outdated Show resolved Hide resolved
edsantiago
edsantiago reviewed Jan 9, 2023
View reviewed changes
test/system/055-rm.bats Outdated Show resolved Hide resolved
@rhatdan rhatdan added the 4.4 label Jan 9, 2023
@giuseppe giuseppe force-pushed the podman-rm-f-no-processes branch from b3a8353 to 3e2e650 Compare January 9, 2023 16:35
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 9, 2023
@giuseppe
Copy link
Member Author

giuseppe commented Jan 9, 2023

@edsantiago addressed the comments and pushed a new version

@edsantiago
Copy link
Member

edsantiago commented Jan 9, 2023 •
edited
Loading

Test fails on my laptop:

  # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect --format {{.State.Pid}} cDO4d51oEalqvZCcdeQ4D6dOCfWzUps
   3677837
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   running
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   running
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   running
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   stopping
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman rm -f cDO4d51oEalqvZCcdeQ4D6dOCfWzUps
   time="2023-01-09T10:13:55-07:00" level=warning msg="StopSignal SIGTERM failed to stop container cDO4d51oEalqvZCcdeQ4D6dOCfWzUps in 2 seconds, resorting to SIGKILL"
   open pidfd: No such process
   Error: cannot remove container 73b3fa6ced2586d5c6468816321f84cd52b28c76d53c160114d5dee9eb8f8bfe as it could not be stopped: sending SIGKILL to container 73b3fa6ced2586d5c6468816321f84cd52b28c76d53c160114d5dee9eb8f8bfe: container state improper: stopped
   [ rc=2 (** EXPECTED 0 **) ]

UPDATE: only fails sometimes, which is worse.

edsantiago
edsantiago suggested changes Jan 9, 2023
View reviewed changes
Copy link
Member

@edsantiago edsantiago left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test is broken (it flakes). Leaving it in your hands to fix.

@giuseppe
Copy link
Member Author

giuseppe commented Jan 9, 2023

Test fails on my laptop:

  # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect --format {{.State.Pid}} cDO4d51oEalqvZCcdeQ4D6dOCfWzUps
   3677837
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   running
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   running
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   running
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman inspect cDO4d51oEalqvZCcdeQ4D6dOCfWzUps --format {{.State.Status}}
   stopping
   # /home/esm/src/atomic/2018-02.podman/libpod/bin/podman rm -f cDO4d51oEalqvZCcdeQ4D6dOCfWzUps
   time="2023-01-09T10:13:55-07:00" level=warning msg="StopSignal SIGTERM failed to stop container cDO4d51oEalqvZCcdeQ4D6dOCfWzUps in 2 seconds, resorting to SIGKILL"
   open pidfd: No such process
   Error: cannot remove container 73b3fa6ced2586d5c6468816321f84cd52b28c76d53c160114d5dee9eb8f8bfe as it could not be stopped: sending SIGKILL to container 73b3fa6ced2586d5c6468816321f84cd52b28c76d53c160114d5dee9eb8f8bfe: container state improper: stopped
   [ rc=2 (** EXPECTED 0 **) ]

UPDATE: only fails sometimes, which is worse.

thanks, I can reproduce as well. It is another race condition/mishandling. I expect "podman rm -f" to never fail if the container was already stopped

@giuseppe
oci: check for valid PID before kill(pid, 0)
494db3e 
check that the container has a valid pid before attempting to use
kill($PID, 0) on it.  If the PID==0, it means the container is already
stopped.

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe
podman: podman rm -f doesn't leave processes
4cf06fe 
follow-up to 6886e80

when "podman -rm -f" is used on a container in "stopping" state, also
make sure it is terminated before removing it from the local storage.

Signed-off-by: Giuseppe Scrivano <[email protected]>
@giuseppe giuseppe force-pushed the podman-rm-f-no-processes branch from 3e2e650 to 4cf06fe Compare January 9, 2023 20:01
vrothberg
vrothberg reviewed Jan 10, 2023
View reviewed changes
Copy link
Member

@vrothberg vrothberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@giuseppe
Copy link
Member Author

the patch was confirmed to fix the issue, so I'll prepare the backport as soon as this is merged

flouthoc
flouthoc approved these changes Jan 10, 2023
View reviewed changes
Copy link
Collaborator

@flouthoc flouthoc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
/lgtm
/approve

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 10, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 10, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: flouthoc, giuseppe

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@giuseppe
Copy link
Member Author

/hold cancel

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 10, 2023
@openshift-merge-robot openshift-merge-robot merged commit 5b9e068 into containers:main Jan 10, 2023
@giuseppe
Copy link
Member Author

backport here: #17052

@edsantiago edsantiago mentioned this pull request Jan 25, 2023
Closed
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 15, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@vrothberg vrothberg vrothberg left review comments

@edsantiago edsantiago edsantiago requested changes

@flouthoc flouthoc flouthoc approved these changes

Assignees

@baude baude

@flouthoc flouthoc

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. release-note
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@giuseppe @mheon @baude @vrothberg @edsantiago @flouthoc @rhatdan @openshift-merge-robot