Question: will kind delete unused images?

[WeihanLi]

Will kind remove unused images? If not, can I delete the image manually?

What should be cleaned up or changed:

unused docker images when the kubernetes does not need that image.

for example, deploy with revisionHistoryLimit: 0, when new image used, the before image should be deleted

Why is this needed:

if not clean these unused docker images, the disk resource needed will increase crazily

[BenTheElder]

If you delete a kind cluster all resources on the host are released including disks space used by the nodes.

[WeihanLi]

I wanna delete those unused images without delete the kind cluster, I tried to docker exec kind-control-plane docker images, but got an error like unknown command docker

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"docker\": executable file not found in $PATH": unknown

[BenTheElder]

I wanna delete those unused images without delete the kind cluster, I tried to docker exec kind-control-plane docker images, but got an error like unknown command docker

Ah. So imageGC is disabled by default with kind but you could turn it on with a kubeadm config patch.
The reason this is off is to avoid GCing the core images under disk pressure and because we don't know how much disk you intend to keep free on the host.

For a production cluster with a dedicated VM / machine this would be a terrible idea, but for kind we expect that you'll run this on your workstation and it will see how much space is free there, which may be some much lower percentage than the eviction threshold for a production setup.

We will probably ease this at some point to some non-zero but small threshold (?) ... it needs thought and experimentation / data.

Now as to why docker exec kind-control-plane docker images didn't work, confusingly around kind v0.3.0 we switched to using CRI (contianerd) inside the nodes for a few reasons, so instead you need docker exec kind-control-plane crictl inspecti. We have it already configured so you don't need to specify the CRI endpoint.

More docs for crictl here 😅 https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md

[WeihanLi]

OK, Thanks @BenTheElder

[BenTheElder]

Is this an issue you're actively encountering or a hypothetical? image content storage is somewhat deduped but...

Ideally using kind you never need to docker exec yourself, but it is of course available for debugging etc. 😅

[WeihanLi]

I just met the problem yesterday... , the disk usage achieved 100%, some apps can not work like before, after some disk files cleaned and docker images removed, worked as before with system reboot.

[WeihanLi]

I guess I should config something about imageGC with kubelet

[WeihanLi]

Ah. So imageGC is disabled by default with kind but you could turn it on with a kubeadm config patch.
The reason this is off is to avoid GCing the core images under disk pressure and because we don't know how much disk you intend to keep free on the host.

Is there some more details docs? @BenTheElder

[tao12345666333]

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
# patch the generated kubeadm config with some extra settings
kubeadmConfigPatches:
- |
    apiVersion: kubelet.config.k8s.io/v1beta1
  kind: KubeletConfiguration
  metadata:
    name: config
  imageGCHighThresholdPercent: 90
  evictionHard:
    nodefs.available: "0%"
    nodefs.inodesFree: "0%"
    imagefs.available: "70%"

you can edit config file like this. I will open a PR for this ASAP. (It will probably be this evening or tomorrow. I just finished KubeCon today and I am on my way home.

EDIT: You need to format it, I have a problem with the phone format now.

[WeihanLi]

Thanks for your help @tao12345666333

[tao12345666333]

open a PR for this. #663

[RonakPrajapatiPanamax]

I guess I should config something about imageGC with kubelet
Did you use "imageGC with kubelet" configuraton patches. @BenTheElder can we use above kubelet config patch I want to keep at 50% usage of disk of node all applicatoin images should be evicated.

@BenTheElder I see your comment on other linked documenataion PR which is raise. So is it good can we use this configuration in such a way that core images should not get evicted.

I just want some automate way of configuration So after 50% of node disk usage all my application level images which is older then 24 hour should be deleted .

[BenTheElder]

So is it good can we use this configuration in such a way that core images should not get evicted.

The built in GC is not capable of doing this, it can only prevent GC for the "pause" image and everything else is at risk.

The current discussion in SIG node is to have cri-o and containerd implement something like containerd/containerd#7944 instead of Kubernetes becoming aware of this.

Depending on your use-case, this might be safe already. If you're connected to the internet and have no issues with kind pulling public images that will be slightly larger than the ones we provide, you can configure the GC to 50% of disk.

So after 50% of node disk usage all my application level images which is older then 24 hour should be deleted .

Kubernetes's image GC cannot do this. It can be configured to trigger on 50% but it will not look at image age and doesn't distinguish the type of image, it's very rudimentary.

You could write a custom tool to implement this against the containerd or CRI APIs, or script executing ctr/crictl on the nodes.

FWIW: your node(s) are all sharing the host disk space, so that's also something to watch out for. Also on some linux environments kubelet cannot see filesystem stats in the containerized environment, like #2524 (comment)