Create a doc to explain how to change the runtime (mostly moving away from dockershim)

[SergeyKanzhelev]

This is a Feature Request

As part of #25787

What would you like to be added

Have a content_type: task type of a documentation that describe the supported way to transition from one runtime to another. It may be a page under Dockershim deprecation tasks, upgrade doc, or an alternative location under "runtimes".

Why is this needed

With the Dockershim deprecation, having a clear step by step documentation of replacing the runtime would be needed before moving to the next step of dockershim removal.

Comments

KEP: kubernetes/enhancements#2221

See discussions:

• Migrating from dockershim documentation #25787 (comment)

/sig node
/cc @neolit123

[Aut0R3V]

Is anyone working on this? I'm interested for working on this issue.

[SergeyKanzhelev]

Is anyone working on this? I'm interested for working on this issue.

nobody working on it right now. So please go ahead. Do you want to discuss the proper placement of this document in the issue of prefer to start with the draft of a document?

[Aut0R3V]

It'll be great to hear whatever you can recommend. I'll send in a draft then

[SergeyKanzhelev]

It'll be great to hear whatever you can recommend. I'll send in a draft then

Sorry for delay. My suggestion would be to start with the document that replicates content from "Upgrade worker nodes" section here: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#upgrade-worker-nodes

And instead of kubelet update - provide instructions on changing from docker to containerd.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[neolit123]

/remove-lifecycle stale Users are already doing this. As discussed previously the steps are similar to a kubelet upgrade with node drain.

[msschl]

Any updates on this? Would be nice to have a migration guide as we plan to move our kubeadm cluster away from dockershim.

[neolit123]

i'm pretty sure the steps would be similar to this:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/#migrating-to-the-systemd-driver

...
Stop the container runtime
...
Start the container runtime <-- here is where you start a different container runtime
...

but SIG node should confirm if this is actually the right way.

you could create a test cluster and try migrating it.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Debanitrkl]

/remove-lifecycle stale
Working on this here kubernetes/kubernetes#104878

[sftim]

/triage accepted

[sftim]

/language en
/kind feature

[sftim]

/priority important-soon
Folks want to know how to migrate

[afbjorklund]

We are missing documentation on how to move docker build away from dockerd, over to buildkitd...

Moving away from Docker leaves users without any documented method of building container images.

Suggested migration approaches include using sudo nerdctl.

Or even writing your own buildctl wrapper, like minikube had to.

[sftim]

https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker/ covers why you can still use docker build.

[afbjorklund]

I was specifically referring to the (questionable) practice of building container images directly on the nodes.

It is not possible to use docker build, if dockerd has been replaced with containerd. Need to add buildkitd.

It is mostly a problem for single-node clusters, like minikube.

But it could be worth more than a little side note, like now ?

https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-deprecation-affects-you/#role-of-dockershim

"You can still pull images or build them using docker build command. But images built or pulled by Docker would not be visible to container runtime and Kubernetes. They needed to be pushed to some registry to allow them to be used by Kubernetes."

[afbjorklund]

The FAQ has been updated, and has better info about building (after switching to containerd) now:

https://kubernetes.io/blog/2020/12/02/dockershim-faq/#what-should-i-look-out-for-when-changing-cri-implementations

Another thing to look out for is anything expecting to run for system maintenance or nested inside a container when building images will no longer work. For the former, you can use the crictl tool as a drop-in replacement (see mapping from docker cli to crictl) and for the latter you can use newer container build options like img, buildah, kaniko, or buildkit-cli-for-kubectl that don’t require Docker.

We detailed the manual steps in minikube, such as installing and configuring buildkitd and buildctl:

https://minikube.sigs.k8s.io/docs/handbook/pushing/#6-pushing-directly-to-in-cluster-containerd-buildkitd

Using kubectl build is a more flexible, even if slightly more complicated, way of doing the same thing:

https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/blob/main/docs/architecture.md (under "directly")

[KelvinAkpobome]

I will like to take on this

[chris-short]

/assign @chrisnegus

[chrisnegus]

I believe documentation is done to meet the intent of this issue, including How to find out what container runtime is used on a Node, as well as instructions for migrating from using dockershim to:

• containerd
• cri-dockerd

Beyond that, @afbjorklund asked for docs describing replacements for using docker build on nodes, then points to content covering that topic in this comment. I will give everyone few days to comment on whether more is needed before I close this Issue.

[afbjorklund]

As far as I know, the information on how to build images is available in the container runtimes (outside k8s.io).

The main highlight was that it is not covered with crictl, so that you need to use something outside of CRI ?

For containerd, that means installing and configuring buildkit.

But it does not have to be addressed before closing this issue.

[chrisnegus]

@afbjorklund Thanks for your insight here. Would it be appropriate to open a separate issue for the k8s.io docs to cover the issue of building containers (that could be handled after 1.24) or it is okay to leave that to the runtime docs, outside of k8s.io?

[afbjorklund]

It is my understanding that kubernetes.io will not carry any documentation for any container runtime.

Each runtime will document, how you do docker build or podman build or nerdctl build or whatever ?

But please check with the SIG Docs members, I was just remotely following issues such as PR #32738.

For minikube the situation is slightly different, since the container runtime is provided by the project too...

[sftim]

If we wanted to explain how to build container images, that might be a topic better covered on the blog.

If someone wants to, that could either be a survey of existing articles that already cover it (there must be hundreds), or a new article that also explains how to get your built container running in Kubernetes.

I think that's a separate thing from what this issue is about: explaining how to change which container runtime your cluster uses.

[sftim]

@chrisnegus I suggest marking this as done, and tracking a new issue (based on something I just spotted):

The v1.24 docs recommend moving away from dockershim. I think we need to word it more strongly than that.

Do you agree?

[afbjorklund]

We added minikube image build and minikube image load, to help users move away from using docker.

Some users might prefer to run two VMs, one for building images and one (or more) for running the cluster.
That works fine too (just slower), and then they can have dockerd in the first and containerd in the second.

The only thing we need to explain, is that if they change their runtime - our "docker-env" stops working.

❌ Exiting due to MK_USAGE: The docker-env command is only compatible with the "docker" runtime, but this cluster was configured to use the "containerd" runtime.

Hopefully the (minikube) error message is clear enough.

[chrisnegus]

@chrisnegus I suggest marking this as done, and tracking a new issue (based on something I just spotted):

The v1.24 docs recommend moving away from dockershim. I think we need to word it more strongly than that.

Do you agree?

I agree. I think we can close this issue, since the migration content has been addressed and we are well enough covered for image building documentation. I'll open a post-1.24 Issue to consider stronger wording to people for getting off of Dockershim.

[chrisnegus]

/close

[k8s-ci-robot]

@chrisnegus: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chrisnegus]

@chrisnegus I suggest marking this as done, and tracking a new issue (based on something I just spotted):
The v1.24 docs recommend moving away from dockershim. I think we need to word it more strongly than that.
Do you agree?
I agree. I think we can close this issue, since the migration content has been addressed and we are well enough covered for image building documentation. I'll open a post-1.24 Issue to consider stronger wording to people for getting off of Dockershim.

Done. I opened Issue #32827. I think we should revisit it after 1.24.