Using crictl rmi doesn't actually remove anything

[afbjorklund]

What happened:

Removing images with cri-tools doesn't seem to be working ?

What you expected to happen:

The image should be removed, or at least an error reported ?

Says Deleted:

How to reproduce it (as minimally and precisely as possible):

sudo crictl rmi

Anything else we need to know?:

Workaround:

docker rmi

Environment:

• Container runtime or hardware configuration: Docker 20.10.17
• OS (e.g: cat /etc/os-release): Ubuntu 20.04.4 LTS
• Kernel (e.g. uname -a): 5.4.0-122-generic
• Others: Kubernetes v1.24.3, crictl v1.21.0

[afbjorklund]

Seems to be known:

• crictl rmi <image> doesn't works with docker #568

The internal error is:

The image reference ... was not a digest reference

---

But removing by digest doesn't work either, even if it gives a different internal error.

The image reference foo/bar@sha256:... does not directly refer to the given image's ID sha256:..

None of these are propagated to crictl, which always returns a happy "Deleted" face.

• Removing images does not appear to be working correctly Mirantis/cri-dockerd#110

[afbjorklund]

It was a problem with the documentation, the command only expects full Id (sha256: etc)
When it gets a proper Id as input, then it actually removes all the tags referencing it.

Apparently trying to remove an image that does not exist, or an invalid Id, is not an error.
This is not only the docker implementation, it is handled the same way in containerd:

                if err == store.ErrNotExist {
                        // return empty without error when image not found.
                        return &runtime.RemoveImageResponse{}, nil
                }

The workaround is to use crictl images --no-trunc, and make sure to crictl rmi $ID
(i.e. to use the full "Image ID", 7+64 chars, and not try to use the truncated ID or tag or digest)

$ sudo crictl pull busybox:latest
Image is up to date for busybox@sha256:ef320ff10026a50cf5f0213d35537ce0041ac1d96e9b7800bafd8bc9eff6c693
$ sudo crictl images --no-trunc busybox:latest
IMAGE               TAG                 IMAGE ID                                                                  SIZE
busybox             latest              sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd   1.24MB
$ sudo crictl rmi sha256:7a80323521ccd4c2b4b423fa6e38e5cea156600f40cd855e464cc52a321a24dd
Deleted: busybox:latest
$ sudo crictl images busybox:latest
IMAGE               TAG                 IMAGE ID            SIZE

[pmengelbert]

Related to this, an image with no associated repoTags cannot be removed by imageID:

$ crictl images -o json | jq '.images[] |select(.repoTags == [])'
{
  "id": "sha256:0ebe7501b7b5ada018c460eb2519066b7c897c71214fff4aea4550513c04bfd8",
  "repoTags": [],
  "repoDigests": [
    "docker.io/library/import-2022-11-16@sha256:ffd9f8a307fab422201b95fd1de54e21a7a71de465c35ab4203f02480cc816cd"
  ],
  "size": "2109278",
  "uid": null,
  "username": "",
  "spec": null
}

$ crictl rmi sha256:0ebe7501b7b5ada018c460eb2519066b7c897c71214fff4aea4550513c04bfd8
$ echo $?
0

$ crictl images -o json | jq '.images[] | select(.id == "sha256:0ebe7501b7b5ada018c460eb2519066b7c897c71214fff4aea4550513c04bfd8")'
{
  "id": "sha256:0ebe7501b7b5ada018c460eb2519066b7c897c71214fff4aea4550513c04bfd8",
  "repoTags": [],
  "repoDigests": [
    "docker.io/library/import-2022-11-16@sha256:ffd9f8a307fab422201b95fd1de54e21a7a71de465c35ab4203f02480cc816cd"
  ],
  "size": "2109278",
  "uid": null,
  "username": "",
  "spec": null
}

[afbjorklund]

@pmengelbert I cannot reproduce, as long as you give the full id then it removes fine.

At least with cri-dockerd, so maybe something that is specific to containerd-cri or crio ?

[thibaut-lo]

@afbjorklund I do face a similar issue

[afbjorklund]

The --debug output should show the actually request/response... Does it work better, if you use ctr directly ?

[SergeyKanzhelev]

@thibaut-lo please use --debug output as suggested and also what runtime/versions do you use?