Provide kind docker image to support docker-compose setup

[gerardo]

What would you like to be added:

It would be great if kind could be provided as a docker image, so that it could be used as an extra service through docker compose. Ideally a docker-compose setup using a hypothetical kind image would look like this:

version: "2.2"
services:
  postgres:
    image: postgres:9.6
    restart: always
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow

  kubernetes:
    image: someorg/kind:latest
    privileged: true

  mycontainer:
    image: myimage:latest
    depends_on:
      - postgres
      - kubernetes

After running docker-compose run mycontainer bash (to get a shell from inside my container while having the other container services available in the same bridge network), you should be able to connect to kubernetes using the host name kubernetes.

Why is this needed:

Some use cases:

• For CI pipelines relying on docker-compose, it's more natural to have services provided as containers that are already up and running. A concrete example is Apache Airflow. We use docker-compose to provide services that are ready to be used. This also allows us to reuse this same setup for local development, without forcing devs to install services themselves. Another upside of this is that it allows us to migrate easily to a different CI provider without having to rewrite much of our pipeline.
• For use with testcontainers

Similar:

• Run a cluster as a docker run oneliner #170

[BenTheElder]

This doesn't really map particularly well to how kind works. We need to execute logic against node containers to orchestrate the cluster.

We can still run kind this way in a container if we put another layer of docker in docker and run a whole cluster inside there but then you need some mounts etc. That makes this clunky, slower, and less reliable. We're going to document how you'd go about that in a pod (tracked in another issue) which is pretty similar because we do it and need it, but I'm not sure we'd want to recommend it or try to provide a special image for it.

You can achieve this yourself today with a docker in docker image + the kind binary, which is pretty trivial to create. The rest is customizing how you want kind to run / cluster configuration, cleanup handling, possibly preloading the node image you use, timeouts for docker start / stop, logging, etc. which is going to be use-case specific.

Setting up kind across different CI providers and locally is pretty trivial today fwiw, but we normally leave setting up docker for kind to use and downloading / running the kind CLI to you. https://github.com/kind-ci/examples

[gerardo]

Thanks for the explanation. I started trying to make this work inside a docker container.

I tried 2 different scenarios:

1. I installed docker and kind inside my container, and ran such container in privileged mode. Docker starts successfully, but kind dies trying to get the control plane up and running. I decided not to keep going down that path.

2. In this second scenario, I mounted the host's docker socket (/var/run/docker.sock:/var/run/docker.sock). The docker daemon is available inside the container and kind starts the cluster successfully. But this particular setup presents a problem because of the x509 certificate:

   • If I use host.docker.internal as the cluster host in the k8s configuration, I get Unable to connect to the server: x509: certificate is valid for kind-control-plane, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, localhost, not host.docker.internal.
   • If I use the ip to which this dns name resolves (192.168.65.2 in this case), I get Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 172.17.0.2, 172.17.0.2, 127.0.0.1, not 192.168.65.2.

It seems this issue has been discussed before in #566.

What options do I have left?

[BenTheElder]

1. is what we typically do and what kind itself used to do (containerd inside now but same idea)

Note that docker in docker require some special attention including:

• docker storage should be a volume, not the container filesystem
• For running kind / another layer nested, you should use the equivalent mounts in document how to run kind in a kubernetes pod #303 (which we can't bake into an image)

For 2. it is possible to patch the kubeadm config to specify which hosts to sign the cert for, something like:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatchesJson6902:
- group: kubeadm.k8s.io
  version: v1beta2
  kind: ClusterConfiguration
  patch: |
    - op: add
      path: /apiServer/certSANs/-
      value: my-hostname-to-add

[gerardo]

Hey @BenTheElder, I'd like to revisit the kind use case inside a container. I saw #706 but I'm not quite sure I understand the required paths for making it work. Are you saying mounting /lib/modulesand /sys/fs/cgroup from the host is a requirement? What if the host is a Mac?

Also, what's the path for docker storage? /var/lib/docker in its entirety?

[BenTheElder]

Are you saying mounting /lib/modulesand /sys/fs/cgroup from the host is a requirement?

The latter is a requirement if you're adding another layer on top of kind. The former is also useful.

What if the host is a Mac?

There's no native docker on Mac, it will get those mounts from the host VM.

Also, what's the path for docker storage?

That's configurable. You're looking for the docker data root.

/var/lib/docker in its entirety?

Wherever the container filesystems will be mounted cannot also be a container filesystem.

[BenTheElder]

Note that aside from the modules which some Kubernetes related tooling will inspect, the rest of those are entirely about "nesting" containers and have no particular significance to kind.

[BenTheElder]

gcr.io/k8s-testimages/krte is what kind uses in CI

We'll probably install the stable kind binary in it soon, however this is at most semi official, and this isn't a pattern I recommend.

Ideally test setups come from the https:/)github.com/kind-ci/examples repo, we link to this in the docs and are consolidating things used for third party CI platforms there.

The Kubernetes project does not use or support most of these, so that location adjacent is a little more appropriate (and easier to enable eg Travis)

[BenTheElder]

to be clear the KRTE image contains these things, you can find the sources in https://github.com/kubernetes/test-infra/
It's meant for use by the kind project but could be reused..
For now we've focused on the binaries being easy to install. Adding the extra layer of docker is a footgun that we should avoid where possible.

[gerardo]

@BenTheElder it makes sense. I'll have a look at those. Thanks!

[abelbarrera15]

@gerardo , did you end up finding a way to get this to work? Trying to do the same at the moment :d

I copied more or less what you did in the airflow commit here: apache/airflow#5837

But it doesn't seem to work for me. I get: The connection to the server localhost:8080 was refused - did you specify the right host or port?

[ninja-]

This would be great to have so that it can work on docker native CIs like GitLab CI...
Currently I have to create a DIND setup and then pull KIND image every time which can't be cached :(

[mitar]

Yep, having a kind Docker image which one could just use as a GitLab service would be great.

[rhyek]

Trying to configure a pre-packaged dev environment for my monorepo complete with necessary infrastructure services (db, redis, etc) in a docker-compose file and it would be ideal to also include a k8s cluster service in there to make it easier to onboard new team members and it would reflect a final production environment somewhat where you have external IaaS components being accessed from within your cloud k8s cluster.

It makes total sense. If anyone has a working example they can link to that would be amazing.

[BenTheElder]

We have a contrib repo with CI examples here https://kind.sigs.k8s.io/docs/user/resources/#using-kind-in-ci

PRs welcome.