kindest/node:v1.22.2 -> label key and value greater than maximum size (4096 bytes), key: containerd: invalid argument

[Birdrock]

What happened:
On kindest/node:v1.22.2, with images built with Pack & Paketo Buildpacks I run into the error failed to create containerd container: create container failed validation: containers.Labels: label key and value greater than maximum size (4096 bytes), key: io.buildpa: invalid argument.

What you expected to happen:
Expected containers to create successfully. Using kindest/node:v1.22.1 successfully starts containers.

How to reproduce it (as minimally and precisely as possible):

kind create cluster --image=kindest/node:v1.22.2
kubectl apply -f https://github.com/pivotal/kpack/releases/download/v0.4.1/release-0.4.1.yaml
kubectl get pods -n kpack
kubectl describe pod kpack-controller-xxxxxxxxxx-xxxxx -n kpack

Anything else we need to know?:
Images built with Paketo Buildpacks tend to have a highly populated Config.Labels and many layers. This particular issue was previously reported and had a fix applied. It is referenced in v0.9.0 breaking changes.

Environment:

• kind version: (use kind version):

kind version 0.11.1

• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-16T02:43:21Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-10-27T11:01:51Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info):

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 11
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.11.0-38-generic
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.11GiB
 Name: <redacted>
 ID: <redacted>
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: <redacted>
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

[aojea]

what is the component that raises the error? containerd? ctr?
sounds like one regression on one of those per your description

[Birdrock]

The Event is Warning Failed 12s (x5 over 53s) kubelet Error: failed to create containerd container: create container failed validation: containers.Labels: label key and value greater than maximum size (4096 bytes), key: io.paketo.: invalid argument - this suggests it is containerd to me. Where would I look for more information?

[aojea]

so, image 1.22.1 was build for kind 0.11 has containerd 1.5.2

kind/images/base/Dockerfile

Line 31 in 6a69d86

ARG CONTAINERD_VERSION="1.5.2"

the new image for 1.22.2 has containerd 1.5.7

kind/images/base/Dockerfile

Line 30 in 77e79b8

ARG CONTAINERD_VERSION="1.5.7"

if you can confirm that the problem is that containerd regressed, we should open an issue in that project

[Birdrock]

Awesome - thanks for pointing me that way. I'll try building an image with containerd 1.5.2.

[Birdrock]

After trying a few versions of containerd, the functionality changed between 1.5.5 and 1.5.6. 1.5.5 was able to create containers successfully, while 1.5.6 & 1.5.7 had the mentioned error.

[aojea]

@AkihiroSuda ^^

[aojea]

this seems related containerd/containerd#6012

[BenTheElder]

containerd/containerd@2a8dac1 is in containerd v1.6.0-beta.1 api/v1.6.0-beta.1, we could ship kind v0.12.0 using one of the containerd v1.6.0 betas. it wouldn't be the first time we've upgraded ahead of a stable tag to pick up fixes.

[aojea]

the backport is ready to merge containerd/containerd#6187
I prefer to use latest stable 1.5.x if possible, that is closest to the users and we can discover bugs like this

[aojea]

@Birdrock can you verify that the new image "kindest/node:v1.22.2@sha256:9af3ab3e36fb59890b2fb3a18000930b4792d62d10d2060e0ca701e2c392d487" solves it?

[Birdrock]

@aojea Seems to have done the trick!

  containerStatuses:
  - containerID: containerd://c855643e2ed104dd5e1dfff98e76f4d3195ce872a729bb67d0e6ac0f27a4b052
    image: sha256:0501ba3d983a69412d4c4e06e964fd17482b9710ab044d8cd96c7ddd0a98d0c3
    imageID: gcr.io/cf-build-service-public/kpack/controller@sha256:c955647b92f4a9cdfae8f690e694a71a3b48121cbc4f6d3db38fe0e126c2a280
    lastState: {}
    name: controller
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-11-04T16:24:37Z"