Cannot checkpoint container: "failed: could not load libcriu.so.2"

[MikelSeron]

Description
Cannot checkpoint a container with Ubuntu 22.04, CRI-O v1.29.5 (Checkpoint/restore support enabled), CRIU v3.19, K8s v1.29.

Steps to reproduce the issue:

1. sudo crictl checkpoint --export=checkpoint.tar bb04e6eb6ab0b

or

2. sudo curl -sk -X POST "https://localhost:10250/checkpoint/default/webserver/webserver"
   --key /etc/kubernetes/pki/apiserver-kubelet-client.key
   --cacert /etc/kubernetes/pki/ca.crt
   --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt

Describe the results you received:

FATA[0000] checkpointing the container "bb04e6eb6ab0b" failed: rpc error: code = Unknown desc = failed to checkpoint container bb04e6eb6ab0bb3444c1c0dbaca80a3b779a6496b24d09caa83f8e43abb1f712: 
running "/usr/bin/crio-crun" ["checkpoint" "--file-locks" "--image-path" "/var/lib/containers/storage/overlay-containers/bb04e6eb6ab0bb3444c1c0dbaca80a3b779a6496b24d09caa83f8e43abb1f712/userdata/checkpoint" 
"--work-path" "/var/lib/containers/storage/overlay-containers/bb04e6eb6ab0bb3444c1c0dbaca80a3b779a6496b24d09caa83f8e43abb1f712/userdata" 
"--leave-running" "bb04e6eb6ab0bb3444c1c0dbaca80a3b779a6496b24d09caa83f8e43abb1f712"] 
failed: `/usr/bin/crio-crun --root /run/crun --systemd-cgroup checkpoint --file-locks --image-path /var/lib/containers/storage/overlay-containers/bb04e6eb6ab0bb3444c1c0dbaca80a3b779a6496b24d09caa83f8e43abb1f712/userdata/checkpoint 
--work-path /var/lib/containers/storage/overlay-containers/bb04e6eb6ab0bb3444c1c0dbaca80a3b779a6496b24d09caa83f8e43abb1f712/userdata 
--leave-running bb04e6eb6ab0bb3444c1c0dbaca80a3b779a6496b24d09caa83f8e43abb1f712` failed: could not load `libcriu.so.2`

Describe the results you expected:
Correctly checkpoint a container

Additional information you deem important (e.g. issue happens only occasionally):

CRIU logs and information:

CRIU full dump/restore logs:

(paste your output here)

Output of `sudo criu --version`:

Version: 3.19

Output of `sudo criu check --all`:

sudo criu check --all
Looks good.

Output of `sudo crio -v`:

crio version 1.29.5
Version:        1.29.5
GitCommit:      b043c66fe29e023a4c4bd2f5b27a790c86a2ec59
GitCommitDate:  2024-06-01T00:20:55Z
GitTreeState:   clean
BuildDate:      1970-01-01T00:00:00Z
GoVersion:      go1.21.7
Compiler:       gc
Platform:       linux/amd64
Linkmode:       static
BuildTags:
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          unknown
SeccompEnabled:   true
AppArmorEnabled:  true

Output of `lsb_release -a`:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Output of `uname -r`:

6.5.0-1020-oracle

Additional environment details:

[adrianreber]

You need to install libcriu. I don't know where you get your CRIU package from. The RPMs have something called criu-libs which provides this. crun, on RPM based systems, will automatically pull libcriu.

I would recommend to use runc, however. crun has support to work with Kubernetes and checkpoint restore but it is not used commonly from what I have seen.

Ubuntu 22.04 probably means cgroupv2. That also does not work correctly yet. Checkpointing works but restore but does not work from what I have seen.

[MikelSeron]

Hi @adrianreber,

Thank you for your response. I tried installing CRIU on Ubuntu 22.04 directly from the following repository: https://launchpad.net/~criu/+archive/ubuntu/ppa. Additionally, I also attempted building it from the source code on the machine. Despite these efforts, I encountered the same issue.

Do the deb packages or building the code not include "criu-libs"?

Anyway, following your recommendation, I switched to using runc instead of crun. I am pleased to report that with runc, I am now able to successfully checkpoint the container. Thanks!

Regarding cgroupv2, what should I use instead to ensure proper functionality?

Thank you again for your assistance.

Best regards,

[adrianreber]

For restore you probably need cgroupv1. There is some code in runc which unfortunately never made it to a release (yet). Once the runc support is working and released I will change CRI-O to correctly handle cgroupv2 systems.

@rst0git Do we distribute libcriu.so in out PPA packages?

Please close this ticket if it works for you.

[rst0git]

Do we distribute libcriu.so in out PPA packages?

yes, we distribute libcriu.so:

# apt show criu
Package: criu
Version: 3.19-4
Priority: optional
Section: admin
Maintainer: Salvatore Bonaccorso <[email protected]>
Installed-Size: 3,321 kB
Depends: libgnutls30 (>= 3.7.0), python3-protobuf, python3:any, libbsd0 (>= 0.6.0), libc6 (>= 2.34), libnet1 (>= 1.1.2.1), libnftables1 (>= 1.0.2), libnl-3-200 (>= 3.2.27), libprotobuf-c1 (>= 1.0.1)
Recommends: iproute2 | iproute
Homepage: https://www.criu.org/
Download-Size: 773 kB
APT-Manual-Installed: yes
APT-Sources: http://download.opensuse.org/repositories/devel:/tools:/criu/xUbuntu_22.04  Packages
Description: checkpoint and restore in userspace
 criu contains the utilities to checkpoint and restore processes in
 userspace.

# dpkg -L criu | grep libcriu
/usr/lib/x86_64-linux-gnu/libcriu.a
/usr/lib/x86_64-linux-gnu/libcriu.so.2.0
/usr/lib/x86_64-linux-gnu/libcriu.so
/usr/lib/x86_64-linux-gnu/libcriu.so.2
# ls -al /usr/lib/x86_64-linux-gnu/libcriu.*
-rw-r--r-- 1 root root 108748 May 20 09:52 /usr/lib/x86_64-linux-gnu/libcriu.a
lrwxrwxrwx 1 root root     14 May 20 09:52 /usr/lib/x86_64-linux-gnu/libcriu.so -> libcriu.so.2.0
lrwxrwxrwx 1 root root     14 May 20 09:52 /usr/lib/x86_64-linux-gnu/libcriu.so.2 -> libcriu.so.2.0
-rw-r--r-- 1 root root  89160 May 20 09:52 /usr/lib/x86_64-linux-gnu/libcriu.so.2.0

The list of files is the same as the criu package in Ubuntu (https://packages.ubuntu.com/jammy/amd64/criu/filelist).

failed: could not load libcriu.so.2

I was able to replicate the error with crio-crun and confirm that it is not present if we build/install crun from source:

# crio-crun --debug checkpoint ubuntu
2024-06-05T13:42:07.577673Z: could not load `libcriu.so.2`

# crun --debug checkpoint ubuntu
# tail checkpoint/dump.log 
(00.295448) cg:    `- Dumping  of /ubuntu
(00.295451) cg: Writing CG image
(00.295457) unix: Dumping external sockets
(00.295458) tty: Unpaired slave 0
(00.295460) Writing image inventory (version 1)
(00.295491) Running post-dump scripts
(00.295492) Unfreezing tasks into 2
(00.295493)     Unseizing 55441 into 2
(00.295663) Writing stats
(00.295679) Dumping finished successfully

[adrianreber]

Maybe the criu headers need to be available during build time and the error message is just misleading. crun switch to dlopen() but needs criu headers during build time to enable the support.

[rst0git]

It looks like crio-crun is provided by the cri-o package:
https://github.com/cri-o/packaging/blob/main/templates/latest/cri-o/cri-o.spec
Do we need to add criu as a dependency somewhere?

[adrianreber]

@saschagrunert do you know how crio-crun is built? In this discussion we assume that the CRIU header files are missing during crun build time.

[fjellvannet]

Could this issue be reopened?

I still face exactly this issue, and do not see here how it could be solved.

I have installed criu 3.19 from the package manager like @MikelSeron and encountered exactly that error.

I also tried to install the newest version 4.0 from source as I could not find any deb-package of it anywhere. Still the same error. Might be an ubuntu issue, but would still be nice to solve this somehow.

[adrianreber]

@fjellvannet It is not a CRIU issue per se, but a packaging issue with the package you are using. Not much we can do about it here if it is not maintained by us.

[fjellvannet]

@adrianreber Or maybe it is an OS issue? Because neither the self-built package nor the pre-built dev runs, and always encounter this issue. Is there like a dependent package that I need to install to fix it, or some other way I need to configure the make install to make it work?

I already tried this (also on ubuntu) 3 months ago.

Really would like to demonstrate the kubelet checkpoint API to my colleagues in a presentation this week, but yet I have not gotten it to work because of this.

[rst0git]

@fjellvannet Would you be able to use runc instead of crun?

I have installed criu 3.19 from the package manager like @MikelSeron and encountered exactly that error.
I also tried to install the newest version 4.0 from source as I could not find any deb-package of it anywhere. Still the same error. Might be an ubuntu issue, but would still be nice to solve this somehow.

The error below appears because the package for crio-crun is built without libcriu:

failed: could not load `libcriu.so.2`

As Adrian mentioned, this is not something we can fix here. However, runc uses the criu binary instead of libcriu and it should work as expected.

[adrianreber]

I would also recommend using runc.

[fjellvannet]

Using runc was the solution! Just for others who have the same problem here the documented steps:

1. sudo apt install runc // it is included in the default ubuntu sources
2. modify /etc/crio/crio.conf, in my case it was /etc/crio/crio.conf.d/10-crio.conf. In this file, find

[crio.runtime]
default_runtime = "runc" # this is what you need to change

Then just restart the crio service and it should work.

Thank you all for your quick responses! Finally I got this to work :)

[adrianreber]

@fjellvannet thanks for the feedback. Let us know if you have any further questions. You can also find me on the Kubernetes slack.