crictl images prune

[steven-sheehy]

I just switched to CRI-O and crictl and I'm trying to find an equivalent of the docker command to delete unused images to cleanup disk space. In docker, I would just run docker image prune -a -f to do this. I can't seem to find the equivalent in crictl, so is there one and if not can one be added? Something like crictl images prune or as a new param to existing crictl rmi --prune?

crictl: v1.12.0
crio: 1.11.7

[steven-sheehy]

And is there any workaround for now to accomplish the same thing with a combination of commands?

[feiskyer]

There is a cleanup function in cri-o repo: https://github.com/kubernetes-sigs/cri-o/blob/master/test/helpers.bash#L301

[steven-sheehy]

@feiskyer I tried the referenced function. It seems to try to delete all images, not unused images. Of course, the crictl rmi while fail if the images is still being used so it can be used as a workaround. A single command that prunes unused images would still be preferred. I've shorted the workaround to a one-liner:

crictl images -q | xargs -n 1 crictl rmi 2>/dev/null

[feiskyer]

@steven-sheehy Yep, there is no such single command yet as it's not part of CRI.

My concern of doing is that it may break kubelet container lifecycle, e.g. kubelet would pull an image before creating the container. If the prune happens during them, the container creation may be failed. So I think it's better to also consider image pull time when doing prune, but this is not included in CRI.

[steven-sheehy]

I think the main use case of a prune command would be to be ran manually by a user or after a helm upgrade as part of continuous deployment to free up space. So it's most likely not ran very often to encounter the scenario you mention. And if it does happen to be ran in between image pull and execution, won't the kubelet just try the pull and execute again after the back off period?

[Random-Liu]

This is possible to implement in crictl. We can:

1. List all images and iterate through them;
2. ListContainers and check whether an image is being used.

There might be some race condition, but that should be race, and we should get eventual consistency.

[Random-Liu]

I think the main use case of a prune command would be to be ran manually by a user or after a helm upgrade as part of continuous deployment to free up space. So it's most likely not ran very often to encounter the scenario you mention. And if it does happen to be ran in between image pull and execution, won't the kubelet just try the pull and execute again after the back off period?

Kubernetes keep unused images on the node, and considers image locality during scheduling. I'm not sure whether crictl rmi --prune is a pattern we recommend user to easily do on a Kubernetes node.

[steven-sheehy]

You guys are right, there is some race condition with kubelet, the container runtime and my workaround prune command above. I've since switched to containerd 1.2.0 and right after I do a helm upgrade I perform the prune and now the cluster cannot both terminate or start some pods. This is to be expected except that it never resolves itself by re-pulling the images. The errors in kubectl describe pod show:

Warning  FailedSync  13s (x48 over 1h)   kubelet, edge1  error determining status: rpc error: code = Unknown desc = failed to get image "sha256:7a344aad0fdbe8fd3ebd3ace7268d59946408503db1fe7c171bdb016a51729b7": does not exist

journalctl -fu containerd

Nov 27 17:21:19 edge2 containerd[18897]: time="2018-11-27T17:21:19.928243513Z" level=error msg="ContainerStatus for "319749e2f6c3d6004e2404549d11c09fef96b31dc98f57e0191dab3d115c73b2" failed" error="failed to get image "sha256:fb885d89ea5c35ac02acf79a398b793555cbb3216900f03f4b5f7dc31e595e31": does not exist"
Nov 27 17:21:19 edge2 containerd[18897]: time="2018-11-27T17:21:19.926931932Z" level=error msg="ContainerStatus for "8fd4a8b1cc7a4f76a1b6edc34c7d69bf596dbea79da49659568e9c543054c0b2" failed" error="failed to get image "sha256:0171babfae580e71d90e38c9dd1f8fe8b95532d95425c54c4920e98631656212": does not exist"

journalctl -fu kubelet

Nov 27 17:29:46 edge2 kubelet[18959]: E1127 17:29:46.001742   18959 remote_runtime.go:278] ContainerStatus "ae57ce88d169a01a30d571b0af8a349ebf95caff56a55559764d6aaa47c87b16" from runtime service failed: rpc error: code = Unknown desc = failed to get image "sha256:f4969b2fd68d5c6201512af2af252f415adf8e1d2c25c1640dcca43483d7b22a": does not exist
Nov 27 17:29:46 edge2 kubelet[18959]: E1127 17:29:46.001783   18959 kuberuntime_container.go:391] ContainerStatus for ae57ce88d169a01a30d571b0af8a349ebf95caff56a55559764d6aaa47c87b16 error: rpc error: code = Unknown desc = failed to get image "sha256:f4969b2fd68d5c6201512af2af252f415adf8e1d2c25c1640dcca43483d7b22a": does not exist
Nov 27 17:29:46 edge2 kubelet[18959]: E1127 17:29:46.001804   18959 kuberuntime_manager.go:873] getPodContainerStatuses for pod "egress-bc2lq_production(89681573-dfa8-11e8-ae36-00505691cd09)" failed: rpc error: code = Unknown desc = failed to get image "sha256:f4969b2fd68d5c6201512af2af252f415adf8e1d2c25c1640dcca43483d7b22a": does not exist
Nov 27 17:29:46 edge2 kubelet[18959]: E1127 17:29:46.001822   18959 generic.go:271] PLEG: pod egress-bc2lq/production failed reinspection: rpc error: code = Unknown desc = failed to get image "sha256:f4969b2fd68d5c6201512af2af252f415adf8e1d2c25c1640dcca43483d7b22a": does not exist

@Random-Liu Should I open an issue with containerd? You guys may not recommend pruning, but this issue didn't occur with CRI-O and I think cluster should have eventual consistency, as you mentioned.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[towolf]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[towolf]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[towolf]

/remove-lifecycle stale

[louygan]

command like "docker system prune"

sudo crictl ps -a | grep -v Running | awk '{print $1}' | xargs sudo crictl rm && sudo crictl rmi --prune