Skip to content

Latest commit

 

History

History
220 lines (170 loc) · 7.18 KB

containerd-env-setup.md

File metadata and controls

220 lines (170 loc) · 7.18 KB
Raw

Nydus Setup for Containerd Environment

This document will walk through how to setup a nydus image service to work with containerd. It assumes that you already have containerd installed. If not, please refer to containerd documents on how to install and set it up.

Install All Nydus Binaries

Get nydus-image, nydusd, nydusify, ctr-remote and nydus-overlayfs binaries from image-service release page.

nydusd currently has two binaries, nydusd-fusedev (for fuse daemon mode) and nydusd-virtiofs (for virtiofs daemon mode), choose the one that fits your mode and rename it as nydusd.

sudo cp nydusd nydus-image /usr/local/bin
sudo cp nydusify containerd-nydus-grpc /usr/local/bin
sudo cp ctr-remote nydus-overlayfs /usr/local/bin

Get containerd-nydus-grpc binary from nydus-snapshotter release page.

sudo cp nydusify containerd-nydus-grpc /usr/local/bin

Start Containerd Snapshotter for Nydus

Nydus provides a containerd remote snapshotter containerd-nydus-grpc to prepare container rootfs with nydus formatted images. To start it, first save a nydusd configuration to /etc/nydusd-config.json:

$ sudo tee /etc/nydusd-config.json > /dev/null << EOF
{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "scheme": "http",
        "skip_verify": false,
        "timeout": 5,
        "connect_timeout": 5,
        "retry_limit": 2,
        "auth": YOUR+LOGIN+AUTH=
      }
    },
    "cache": {
      "type": "blobcache",
      "config": {
        "work_dir": "cache"
      }
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true,
  "fs_prefetch": {
    "enable": true,
    "threads_count": 4
  }
}
EOF

Note:

  • The auth is Base64 encoded username:password. It is required by nydusd to enforce HTTP basic authentication method.
  • You might have to change the scheme from http to https according to you registry configuration.
  • Please copy your docker login auth from $HOME/.docker/config.json to replace YOUR+LOGIN+AUTH=.

Then start containerd-nydus-grpc remote snapshotter:

sudo /usr/local/bin/containerd-nydus-grpc \
    --config-path /etc/nydusd-config.json \
    --shared-daemon \
    --log-level info \
    --root /var/lib/containerd/io.containerd.snapshotter.v1.nydus \
    --cache-dir /var/lib/nydus/cache \
    --address /run/containerd/containerd-nydus-grpc.sock \
    --nydusd-path /usr/local/bin/nydusd \
    --nydusimg-path /usr/local/bin/nydus-image \
    --log-to-stdout

cache-dir argument represents the blob cache root dir, if unset, it will be set to root + "/cache". It overrides the work_dir option in nydusd-config.json.

Configure and Start Containerd

Nydus uses two features of containerd:

  • remote snapshotter
  • snapshotter annotations

To set them up, first add something like the following to your containerd configuration (default to /etc/containerd/config.toml):

[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd/containerd-nydus-grpc.sock"

Next you should change default snapshotter to nydus and enable snapshot annotations like below:

For version 1 containerd config format:

[plugins.cri]
  [plugins.cri.containerd]
    snapshotter = "nydus"
    disable_snapshot_annotations = false

For version 2 containerd config format:

[plugins."io.containerd.grpc.v1.cri".containerd]
   snapshotter = "nydus"
   disable_snapshot_annotations = false

Then restart containerd, e.g.:

sudo systemctl restart containerd

Start a Local Registry Container

sudo docker run -d --restart=always -p 5000:5000 registry

Convert an Image to Nydus Format

Currently, nydus image must be created by converting from an existed OCI or docker v2 image. Please make sure you are authorized to pull and push the target registry. The basic method is to use docker pull and docker push to validate your access to the target registry.

sudo nydusify convert --nydus-image /usr/local/bin/nydus-image --source ubuntu --target localhost:5000/ubuntu-nydus

Create New Pods with Nydus Image in Kubernetes

For example, use the following nydus-sandbox.yaml and nydus-container.yaml

The nydus-sandbox.yaml looks like below:

metadata:
  attempt: 1
  name: nydus-sandbox
  namespace: default
log_directory: /tmp
linux:
  security_context:
    namespace_options:
      network: 2
annotations:
  "io.containerd.osfeature": "nydus.remoteimage.v1"

The nydus-container.yaml looks like below:

metadata:
  name: nydus-container
image:
  image: localhost:5000/ubuntu-nydus:latest
command:
  - /bin/sleep
args:
  - 600
log_path: container.1.log

To create a new pod with the just converted nydus image:

$ sudo crictl pull localhost:5000/ubuntu-nydus:latest
$ pod=`sudo crictl runp nydus-sandbox.yaml`
$ container=`sudo crictl create $pod nydus-container.yaml nydus-sandbox.yaml`
$ sudo crictl start $container
$ sudo crictl ps
CONTAINER ID        IMAGE                                CREATED             STATE               NAME                      ATTEMPT             POD ID
f4a6c6dc47e34       localhost:5000/ubuntu-nydus:latest   9 seconds ago       Running             nydus-container           0                   21b91779d551e

Try Nydus with ctr-remote

Nydus provides an enhanced ctr tool named as ctr-remote which is capable of pulling nydus image and start container based on nydus image leveraging nydus-snapshotter if it is properly configured and appends more critical labels to the nydus image which should consumed when nydus-snapshotter is setting up snapshot.

You can also use ctr-remote to run containers with converted nydus image and pull it:

$ sudo ctr-remote image rpull --plain-http localhost:5000/ubuntu-nydus:latest
fetching sha256:9523a2de... application/vnd.oci.image.manifest.v1+json
fetching sha256:a2cdb40d... application/vnd.oci.image.config.v1+json
fetching sha256:18059446... application/vnd.oci.image.layer.v1.tar+gzip

Next run container:

$ sudo ctr-remote run --rm -t --snapshotter=nydus localhost:5000/ubuntu-nydus:latest test /bin/bash
$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 13:45 pts/0    00:00:00 /bin/bash
root          10       1  0 13:46 pts/0    00:00:00 ps -ef

Try Nydus with nerdctl

To lazily start container with nerdctl, you CAN'T directly pull image using nerdctl pull or run container using nerdctl run since nydus-snapshotter relies on extra information to identify the image reference. nydus-snapshotter searches labels when preparing snapshot. Normal pull and run operations can't append such information to image when downloading and unpacking. Here, we MUST previously use ctr-remote to complete the image pulling.

Then to lazily start container:

sudo ctr-remote image rpull --plain-http localhost:5000/ubuntu-nydus:latest
sudo nerdctl --snapshotter nydus run --rm -it localhost:5000/ubuntu-nydus bash