interested in integrating with minikube

[medyagh]

hello, in minikube we support docker, containerd and cri-o runtimes. we like to keep supporting docker runtime. I would like to know if there has already been work started for a separate dockershim that we could use in minikube?

[afbjorklund]

Added an issue for it in minikube, the lack of releases and packages is an issue: kubernetes/minikube#12103

[afbjorklund]

For minikube, we would like a static binary (for the ISO) and an Ubuntu 20.04 LTS deb package (for the KIC)

[afbjorklund]

Tested with Kubernetes v1.23.0-alpha.0, but there are a lot of blocking bugs in the current cri-dockerd master

[evol262]

Releases and packages will be up on Monday

[afbjorklund]

Must have missed the announcement, dockershim will not be removed until Kubernetes 1.24

https://kubernetes.io/blog/2020/12/02/dockershim-faq/#when-will-dockershim-be-removed

[afbjorklund]

The dockershim was still present in v1.24.0-alpha.0, but now removed in v1.24.0-alpha.1

using dockershim is not supported, please consider using a full-fledged CRI implementation

[afbjorklund]

I think we will have to change minikube to "containerd", since "docker" is not tested anymore.

[evol262]

Hey Anders -

My apologies here, I've been on holiday for a bit and just got back. In the meantime, my GH account was re-created, and I'm still fighting with getting it to appropriately send me notifications. It was completely expected that upstream would remove support for docker and dockershim, which is why the codebase was handed over for maintenance.

A more or less complete refactor so we do not depend on upstream modules for more or less static const values and k8s logger infrastructure was done, in order to make cri-dockerd make more sense as a standalone product.

Admittedly, with the announcement that it would be 1.24 instead of 1.23, the codebase languished for a bit while we focused on other priorities, but it's back to the top now that upstream support has been removed completely.

Automated releases are coming this week once the Github workflows are re-engineered a little and re-enabled (they were previously stubs taken from upstream). We'd love to have cooperation with the larger community, including minikube. Indeed, minikube is what's used in the current (in-development) version of the GH actions for testing functionality.

The parts of the CRI spec which docker does not support (and by extension, cri-dockerd) are:

• Re-opening the container log (docker streams the entire log)
• Synchronous execution with a timeout

It's honestly fairly minor, though there isn't (currently) a pressing demand to add support for this.

With development picking back up to get a release out, what can we do to support you?

[afbjorklund]

It seems like k8s v1.24.0-alpha.1 is working OK with cri-dockerd 542e27d, and is detected automatically:

• Use cri-dockerd instead of dockershim where needed kubernetes/minikube#12102

Until there are releases available, we are building from source. But binaries and testing (end-to-end) is missed.

FROM golang:1.16
RUN git clone -n https://github.com/Mirantis/cri-dockerd && \
    cd cri-dockerd && git checkout 542e27dee12db61d6e96d2a83a20359474a5efa2 && \
    cd src && env CGO_ENABLED=0 go build -o cri-dockerd

The discussion in minikube is more which one should be the default, all three container runtimes are supported:

• "docker"
• "containerd"
• "crio"

Previously it was about changing the default, but the new scenario is that there is no default (only the CRI)

Users will have to bring their own CRI and their own CNI, and Kubernetes is more of a framework than a product ?

[afbjorklund]

It was completely expected that upstream would remove support for docker and dockershim, which is why the codebase was handed over for maintenance.

Removing support for dockershim was expected, removing support (like documentation and testing) for docker (via CRI) was not ?

I don't see that the two are related, there are reasons for standards (API) and there are reasons for choosing implementations...

• Dockershim removal feedback & issues kubernetes/kubernetes#106917

[evol262]

Binaries, debs, rpm’s, and E2E will land tomorrow afternoon US time. Testing complex GH actions is slow.

The short answer is that, yeah, upstream k8s has, for a long time, tried to shrink and limit core. Particularly as more feature gates make it, supporting rkt, cri-o, containerd, docker, plus hyper v containers, podman, and everything else was an enormous maintenance burden.

Then tack on weave, flannel, calico, rook, etc…

So it’s follow CNI/CRI/CSI. Users will have to bring their own. Sure, I think it’s likely that minikube users will probably have docker as part of their development workflow, and that means they’ll also have containerd. The advantage to docker (and cri-dockerd) for dev workflows is that the docker user experience is friendlier and more familiar to many than crictl, ctr, and the like.

Part or what slowed me down today in workflows was actually sussing out which parts of the audit subsystem work in GH actions to enable critest and kubetest. These are the standard tools, and they’ll be run on this repo.

The next step here is documenting commandline options, getting a workable README, and so forth. But it is coming quickly.

[afbjorklund]

Sure, I think it’s likely that minikube users will probably have docker as part of their development workflow, and that means they’ll also have containerd.

I don't think that is true, docker disables the cri plugin by default and people frown upon the old (1.4) version being used...

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

I think it is mostly because of the poor packaging and documentation of containerd/buildkitd ?

The development workflow in minikube involves interacting with the cluster, so changing container runtime is a disruption.

The main tool will still be kubectl. Possibly including kubectl build.

Kubernetes will not have any default CRI and not have any default CNI.

[evol262]

To clarify, I meant that they will have containerd installed. docker, by default, runs new containers on current versions through containerd-shim-runc, and Docker itself is speaking with the containerd socket. This can be trivially checked with ps on a system with Docker running.

Buildkitd (or Docker buildkit) is a plugin to Docker itself which is pretty much an overhaul of docker build. It's not intended to be invoked directly by end users, but just slots with the correct environment variable (older versions) or is enabled by default on newer versions.

Containerd (and ctr in particular) could definitely benefit from better docs. crictl is honestly a better way to interact with it.

I want to be explicit about my statement: it is not that minikube users will be interacting with k8s via the Docker CLI, but that minikube users (rather than k0s/microk8s/k3s/etc) are very likely to be on their workstation/laptop, and already have docker installed for some other reason. It could be podman, but the number of developers who are using raw containerd or kaniko/kubectl build is much lower than those doing docker build and pushing to a local registry and/or dockerhub to deploy in manifests or helm wharts.

The main tool to interact with k8s will of course be kubectl. The tool for interacting with containers outside of k8s on developer workstations where minikube is used is likely to be docker or maybe podman. Is that clearer?

No, k8s will not have any default CRI or CNI. As you said earlier, k8s is a framework.

Releases are now published. Outside of a patch for the README, what else can I help with?

[afbjorklund]

Sounds clear!

Containerd (and ctr in particular) could definitely benefit from better docs. crictl is honestly a better way to interact with it.

The nerdctl tool provides a more user-friendly alternative to ctr and buildctl, even if it is not available remotely (no API) and is not intended to replace Docker... It is also possible to make other custom highlevel wrappers of the lowlevel tools (so we did).

The majority of users would never have to use either, but can use "minikube" directly - including kubectl and image.

It helps them with the os, arch and version problems - and fills in the holes of the CRI abstraction (like load and build).

... are very likely to be on their workstation/laptop, and already have docker installed for some other reason. ...

Some users of minikube will prefer to avoid having to run two VMs (one for Docker, one for Kubernetes). Even Linux users prefer to not have transfer the images between the two daemons, but to have them instantly available to the single-node cluster on build.

There are even users who run minikube without kubernetes, which is somewhat ironic - but fully possible (--no-kubernetes).

Some of these have been running Docker Machine before, which has now been abandoned. And they can run minikube now.

Releases are now published.

Will update.

[evol262]

Let me ask this differently:

I haven't used Windows or MacOS in over a decade. The idea of having different VMs for Docker and Minikube is pretty alien to me. But using cri-dockerd as the CRI for minikube (or optionally supporting it) would mean a single VM for both, an no need to transfer images between daemons.

We're interested in supporting minikube use cases as much as we can, so what can I do to help you?

[afbjorklund]

We're interested in supporting minikube use cases as much as we can, so what can I do to help you?

I think that once we get the cri-dockerd sockets sorted out, we're good to go for the next minikube release.

It will check the kubernetes version parameter, and use dockershim for - 1.23 and cri-dockerd for 1.24 -

I have done updated packaging for 0.2.0, and will update again for 0.2.1 (?) with the new systemd units...

After that I think we can leave "docker" as the default runtime for minikube, and leave the rest up to the user.

Thanks for the support!

[afbjorklund]

Unfortunately it was broken in minikube 1.25.1 with kubernetes 1.24.0 alpha2, due to our cri-dockerd not updated in time.

• Support docker container runtime in Kubernetes 1.24 kubernetes/minikube#13545

Hopefully it ("docker" runtime) will work out-of-the-box with next minikube release and the kubernetes 1.24.0 beta releases ?

• https://www.kubernetes.dev/resources/release/

1.24.0-beta.0 released | Branch Manager | Wednesday 30th March 2022

Aim for fixing it already in minikube 1.26.0, though.

Version	Pre-release
v1.24.0-alpha.0	Wednesday 24th November 2021
v1.24.0-alpha.1	Wednesday 8th December 2021
v1.24.0-alpha.2	Tuesday 1st February 2022
v1.24.0-alpha.3	Tuesday 15th February 2022
v1.24.0-alpha.4	Tuesday 22nd March 2022

[afbjorklund]

After the latest PR (bumping cri-dockerd to v0.2.0-11-ga4d1895, it seems to be working OK with k8s v1.24.0-alpha.2)

• Upgrade cri-dockerd to fix the socket path kubernetes/minikube#13563

Still using dockershim with v1.23.3, I'm not sure which version of Kubernetes is the earliest one supported by cri-dockerd ?

The user command is always: minikube start --container-runtime=docker

[afbjorklund]

This has been integrated

/close