diff --git a/README.md b/README.md index 79fe73f5be..d14d662029 100644 --- a/README.md +++ b/README.md @@ -6,17 +6,17 @@ # `kind` - `k`ubernetes `in` `d`ocker -`kind` is a tool for running local Kubernetes clusters using Docker container "nodes". +`kind` is a tool for running local Kubernetes clusters using container "nodes". `kind` is primarily designed for testing Kubernetes 1.11+, initially targeting the [conformance tests]. -If you have [go] and [docker] installed `go get sigs.k8s.io/kind && kind create cluster` is all you need! +If you have [go] and [docker] or [podman] installed `go get sigs.k8s.io/kind && kind create cluster` is all you need! 2x speed `kind create cluster` demo `kind` consists of: - Go [packages][packages] implementing [cluster creation][cluster package], [image build][build package], etc. - A command line interface ([`kind`][kind cli]) built on these packages. - - Docker [image(s)][images] written to run systemd, Kubernetes, etc. + - Container [image(s)][images] written to run systemd, Kubernetes, etc. - [`kubetest`][kubetest] integration also built on these packages (WIP) `kind` bootstraps each "node" with [kubeadm][kubeadm]. For more details see [the design documentation][design doc]. @@ -30,8 +30,8 @@ You can install `kind` with `go get sigs.k8s.io/kind`. This will put `kind` in shown [here](https://golang.org/doc/code.html#GOPATH) if you encounter the error `kind: command not found` after installation. -To use `kind`, you will need to [install docker]. -Once you have docker running you can create a cluster with `kind create cluster` +To use `kind`, you will need to [install docker] or [install podman]. +Once you have installed podman or have the docker service running you can create a cluster with `kind create cluster` To delete your cluster use `kind delete cluster` @@ -84,6 +84,7 @@ Participation in the Kubernetes community is governed by the [Kubernetes Code of [go]: https://golang.org/ [docker]: https://www.docker.com/ +[podman]: https://podman.io/ [community page]: http://kubernetes.io/community/ [Kubernetes Code of Conduct]: code-of-conduct.md [Go Report Card Badge]: https://goreportcard.com/badge/sigs.k8s.io/kind @@ -106,5 +107,6 @@ Participation in the Kubernetes community is governed by the [Kubernetes Code of [#kind]: https://kubernetes.slack.com/messages/CEKK1KTN2/ [docs/roadmap.md]: ./docs/roadmap.md [install docker]: https://docs.docker.com/install/ +[install podman]: https://github.com/containers/libpod/install.md [@BenTheElder]: https://github.com/BenTheElder [@munnerz]: https://github.com/munnerz diff --git a/docs/design/base-image.md b/docs/design/base-image.md index 825cc3ec23..ff54834eb4 100644 --- a/docs/design/base-image.md +++ b/docs/design/base-image.md @@ -1,6 +1,6 @@ # The Base Image -The ["base" image][base image] is a small-ish Docker image for running +The ["base" image][base image] is a small-ish container image for running nested containers, systemd, and kubernetes components. To do this we need to set up an environment that will meet the CRI @@ -8,7 +8,7 @@ To do this we need to set up an environment that will meet the CRI step we take is inline to the image's [Dockerfile][dockerfile]), but essentially: -- we preinstall tools / packages expected by systemd / Docker / Kubernetes other +- we preinstall tools / packages expected by systemd / container engine / Kubernetes other than Kubernetes itself - we install a custom entrypoint that allows us to perform some actions before @@ -17,7 +17,7 @@ the container truly boots - we set up a systemd service to forward journal logs to the container tty - we do a few tricks to minimize unnecessary services and inform systemd that it -is in docker (see the [Dockerfile][dockerfile]) +is in a container (see the [Dockerfile][dockerfile]) This image is based on a minimal debian image (currently `k8s.gcr.io/debian-base`) due to high availability of tooling. diff --git a/docs/design/design.md b/docs/design/design.md index 3d522ba666..4b13cb0131 100644 --- a/docs/design/design.md +++ b/docs/design/design.md @@ -6,7 +6,7 @@ This is the root design documentation for `kind`. See also the project ## Overview `kind` or **k**ubernetes **in** **d**ocker is a suite of tooling for local -Kubernetes "clusters" where each "node" is a Docker container. +Kubernetes "clusters" where each "node" is a container. `kind` is targeted at testing Kubernetes. `kind` is divided into go packages implementing most of the functionality, a @@ -35,10 +35,10 @@ In practice kind looks something like this: Clusters are managed by logic in [`pkg/cluster`][pkg/cluster], which the `kind` cli wraps. -Each "cluster" is identified by an internal but well-known [docker object label](https://docs.docker.com/config/labels-custom-metadata/) key, with the cluster +Each "cluster" is identified by an internal but well-known [container object label](https://docs.docker.com/config/labels-custom-metadata/) key, with the cluster name / ID as the value on each "node" container. -We initially offload this type of state into the containers and Docker. +We initially offload this type of state into the containers. Similarly the container names are automatically managed by `kind`, though we will select over labels instead of names because these are less brittle and are properly namespaced. Doing this also avoids us needing to manage anything @@ -63,17 +63,17 @@ For more see [node-image.md][node-image.md]. ### Cluster Creation -Each "node" runs as a docker container. Each container initially boots to a +Each "node" runs as a container. Each container initially boots to a pseudo "paused" state, with [the entrypoint][entrypoint] waiting for `SIGUSR1`. -This allows us to manipulate and inspect the container with `docker exec ...` +This allows us to manipulate and inspect the container with eg `docker exec ...` and other tools prior to starting systemd and all of the components. -This setup includes fixing mounts and pre-loading saved docker images. +This setup includes fixing mounts and pre-loading saved container images. Once the nodes are sufficiently prepared, we signal the entrypoint to actually "boot" the node. -We then wait for the Docker service to be ready on the node before running +We then wait for the Docker service to be ready if we are using Docker on the node before running `kubeadm` to initialize the node. Once kubeadm has booted, we export the [KUBECONFIG][kubeconfig], then apply @@ -84,7 +84,7 @@ At this point users can test Kubernetes by using the exported kubeconfig. ### Cluster Deletion -All "node" containers in the cluster are tagged with docker labels identifying +All "node" containers in the cluster are tagged with container labels identifying the cluster by the chosen cluster name (default is "1"), to delete a cluster we can simply list and delete containers with this label. diff --git a/docs/design/node-image.md b/docs/design/node-image.md index 3a1372f632..1bde832a35 100644 --- a/docs/design/node-image.md +++ b/docs/design/node-image.md @@ -1,6 +1,6 @@ # The Node Image -The ["node" image][node image] is a Docker image for running +The ["node" image][node image] is a container image for running nested containers, systemd, and Kubernetes components. This image is built on top of the ["base" image][base image]. @@ -15,7 +15,7 @@ provides most of the tools statically needed for a kubernetes deployment (eg `systemd`), variants of this image have the following properties: - `/kind/images/` contains various `*.tar` files which are -[Docker image archives][docker image archives], +[Container image archives][container image archives], these images will be loaded by the cluster tooling prior to running `kubeadm` - `kubeadm`, `kubectl`, `kubelet` are in the path @@ -35,6 +35,6 @@ each "node" container with [kubeadm][kubeadm]. [base image]: ./base-image.md [build package]: ./../../pkg/build [cluster package]: ./../../pkg/cluster -[docker image archives]: https://docs.docker.com/engine/reference/commandline/save/ +[container image archives]: https://docs.docker.com/engine/reference/commandline/save/ [systemd service]: https://www.freedesktop.org/software/systemd/man/systemd.service.html [kubeadm]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/ diff --git a/docs/design/overview.md b/docs/design/overview.md index 4954872835..4b5d267704 100644 --- a/docs/design/overview.md +++ b/docs/design/overview.md @@ -1,6 +1,6 @@ # Overview `kind` is a command-line application for building Kubernetes clusters using -Docker containers as "nodes." + containers as "nodes." ## CLI @@ -27,7 +27,7 @@ and setup logging. ├── pkg │   ├── build # Build and manage images │   ├── cluster # Build and manage clusters -│   ├── docker # Interact with Docker +│   ├── container # Interact with container engines │   ├── exec # Execute commands │   ├── fs # Interact with the host file system │   ├── kustomize # Work with embedded kustomize commands @@ -36,7 +36,7 @@ and setup logging. ``` `kind` commands rely on the functionality of the [packages directory][pkg]. Here, you will find everything needed to build container images for `kind`; -create clusters from these images; interact with the Docker engine and file system; customize configuration files; and logging. +create clusters from these images; interact with the container engine and file system; customize configuration files; and logging. diff --git a/docs/devel/README.md b/docs/devel/README.md index 23891e6df0..d1377943e1 100644 --- a/docs/devel/README.md +++ b/docs/devel/README.md @@ -6,7 +6,7 @@ Before you being you must have: * [GitHub account][github] * `git` * `go` -* `Docker` +* `Container Engine` ## Task 1. Read the Kubernetes community guidelines Make sure to read you read the [Kubernetes community guidelines][community]. @@ -32,11 +32,13 @@ $ go version ``` This documentation is written using Go version 1.11+. -## Task 4. Install or upgrade Docker +## Task 4. Install or upgrade Docker or Podman If you haven't already, install the [Docker software using the instructions for your operating system][docker]. +or +[Podman software using the instructions for your operating system][podman]. If you have an existing installation, check your version and make sure you have -the latest Docker. +the latest Docker or Podman To check if `docker` is has been installed: ``` @@ -44,6 +46,12 @@ $ docker --version ``` This documentation is written using Docker version 18.09.0. +To check if `podman` is has been installed: +``` +$ podman --version +``` +This documentation is written using Podman version 0.12.1. + [github]: https://github.com/ @@ -51,3 +59,4 @@ This documentation is written using Docker version 18.09.0. [contributor]: https://github.com/kubernetes/community/blob/master/contributors/guide/README.md [golang]: https://golang.org/doc/install [docker]: https://docs.docker.com/install/#supported-platforms +[podman]: https://github.com/containers/libpod/install.md diff --git a/docs/todo.md b/docs/todo.md new file mode 100644 index 0000000000..b45ae585c5 --- /dev/null +++ b/docs/todo.md @@ -0,0 +1,49 @@ +# TODO + +A non-exhaustive list of tasks (in no-particular order) includes: +- [x] basic single "node" clusters +- [x] multiple clusters per host / named clusters +- [x] user guide(s) + - [x] more detailed user guides for common usage + - [x] more detailed user guides for more advanced usage +- [ ] preflight checks [WIP] +- [ ] multi-node clusters +- [x] cli command to list clusters +- [x] support for multiple kubernetes builds: + - [x] bazel build from source + - [x] docker / make build from source + - [x] apt (upstream / official release packages) + - [ ] support for selecting a non-default package version +- [ ] kubetest ingregration [WIP] +- [ ] improved logging and error handling [WIP] +- [x] continuous integration +- [x] publish pre-built images to a registry +- [x] pre-load images that are not from the build / possibly build more images + - [x] etcd + - [x] coreDNS + - [x] Kubernetes images for released builds + - [ ] overlay network images? +- [ ] support multiple overlay networks +- [x] support advanced configuration via config file + - [x] kubeadm config template override + - [x] node lifecycle hooks +- [ ] more advanced network configuration (not docker0) +- [ ] support for other CRI within the "node" containers (containerd, cri-o) +- [ ] move all container engine functionality into a common package (`pkg/container`) [WIP] + - [ ] potentially move this to using the docker client library +- [x] log dumping functionality / cli commands + - [ ] support audit logging +- [ ] fake out all internals and unit test [WIP] +- [ ] support for local registries +- [ ] support for mounting extra directories + +# Wishlist + +Longer term / continually appealing items: + +- Improved documentation +- Support for architectures / platforms other than linux / amd64 for the node images +- Support for client platforms other than docker on linux / docker for mac +- Less privileged containers or sharing a CRI via something like [containerd namespaces](https://github.com/containerd/containerd/blob/master/docs/namespaces.md), generally + better isolation +- HA kubeadm / multiple control plane nodes diff --git a/hack/build/push-base.sh b/hack/build/push-base.sh index 98dcfd2975..500023d414 100755 --- a/hack/build/push-base.sh +++ b/hack/build/push-base.sh @@ -43,5 +43,6 @@ IMAGE="kindest/base:${TAG}" (set -x; "${KIND}" build base-image --image="${IMAGE}") # push -docker push "${IMAGE}" +ENGINE=$(command -v podman 2> /dev/null || echo docker) +${ENGINE} push "${IMAGE}" diff --git a/hack/build/push-node.sh b/hack/build/push-node.sh index 0f4ea9c052..92da579bb0 100755 --- a/hack/build/push-node.sh +++ b/hack/build/push-node.sh @@ -44,10 +44,11 @@ set -x # re-tag with kubernetes version IMG="kindest/node:${TAG}" -KUBE_VERSION="$(docker run --rm --entrypoint=cat "${IMG}" /kind/version)" -docker tag "${IMG}" "kindest/node:${KUBE_VERSION}" +ENGINE=$(command -v podman 2> /dev/null || echo docker) +KUBE_VERSION="$(${ENGINE} run --rm --entrypoint=cat "${IMG}" /kind/version)" +${ENGINE} tag "${IMG}" "kindest/node:${KUBE_VERSION}" # push -docker push kindest/node:"${KUBE_VERSION}" +${ENGINE} push kindest/node:"${KUBE_VERSION}" diff --git a/pkg/build/base/base.go b/pkg/build/base/base.go index f7287a17db..c4151de0f3 100644 --- a/pkg/build/base/base.go +++ b/pkg/build/base/base.go @@ -24,6 +24,7 @@ import ( log "github.com/sirupsen/logrus" "sigs.k8s.io/kind/pkg/build/base/sources" + "sigs.k8s.io/kind/pkg/container" "sigs.k8s.io/kind/pkg/exec" "sigs.k8s.io/kind/pkg/fs" ) @@ -109,7 +110,7 @@ func (c *BuildContext) Build() (err error) { return err } - // then the actual docker image + // then the actual container image return c.buildImage(buildDir) } @@ -120,7 +121,7 @@ func (c *BuildContext) buildEntrypoint(dir string) error { entrypointDest := filepath.Join(dir, "entrypoint", "entrypoint") cmd := exec.Command(c.goCmd, "build", "-o", entrypointDest, entrypointSrc) - // TODO(bentheelder): we may need to map between docker image arch and GOARCH + // TODO(bentheelder): we may need to map between container image arch and GOARCH cmd.SetEnv("GOOS=linux", "GOARCH="+c.arch) // actually build @@ -136,14 +137,14 @@ func (c *BuildContext) buildEntrypoint(dir string) error { func (c *BuildContext) buildImage(dir string) error { // build the image, tagged as tagImageAs, using the our tempdir as the context - cmd := exec.Command("docker", "build", "-t", c.image, dir) - log.Info("Starting Docker build ...") + cmd := exec.Command(container.Engine, "build", "-t", c.image, dir) + log.Info("Starting container image build ...") exec.InheritOutput(cmd) err := cmd.Run() if err != nil { - log.Errorf("Docker build Failed! %v", err) + log.Errorf("Container image build Failed! %v", err) return err } - log.Info("Docker build completed.") + log.Info("Container image build completed.") return nil } diff --git a/pkg/build/node/node.go b/pkg/build/node/node.go index 28ccd896ea..44d43016d7 100644 --- a/pkg/build/node/node.go +++ b/pkg/build/node/node.go @@ -33,7 +33,7 @@ import ( log "github.com/sirupsen/logrus" "sigs.k8s.io/kind/pkg/build/kube" - "sigs.k8s.io/kind/pkg/docker" + "sigs.k8s.io/kind/pkg/container" "sigs.k8s.io/kind/pkg/exec" "sigs.k8s.io/kind/pkg/fs" ) @@ -51,6 +51,10 @@ const DefaultMode = "docker" // Option is BuildContext configuration option supplied to NewBuildContext type Option func(*BuildContext) +func init() { + container.Engine = DefaultMode +} + // WithImage configures a NewBuildContext to tag the built image with `image` func WithImage(image string) Option { return func(b *BuildContext) { @@ -152,7 +156,7 @@ func (c *BuildContext) Build() (err error) { return err } - // then the perform the actual docker image build + // then the perform the actual container image build return c.buildImage(buildDir) } @@ -184,7 +188,7 @@ func (c *BuildContext) getBuiltImages() (sets.String, error) { images := sets.NewString() for src, dest := range bitPaths { if imageRegex.MatchString(dest) { - tags, err := docker.GetArchiveTags(src) + tags, err := container.GetArchiveTags(src) if err != nil { return nil, err } @@ -197,9 +201,9 @@ func (c *BuildContext) getBuiltImages() (sets.String, error) { // BuildContainerLabelKey is applied to each build container const BuildContainerLabelKey = "io.k8s.sigs.kind.build" -// DockerImageArchives is the path within the node image where image archives +// ContainerImageArchives is the path within the node image where image archives // will be stored. -const DockerImageArchives = "/kind/images" +const ContainerImageArchives = "/kind/images" // private kube.InstallContext implementation, local to the image build type installContext struct { @@ -215,7 +219,7 @@ func (ic *installContext) BasePath() string { func (ic *installContext) Run(command string, args ...string) error { cmd := exec.Command( - "docker", + container.Engine, append( []string{"exec", ic.containerID, command}, args..., @@ -227,7 +231,7 @@ func (ic *installContext) Run(command string, args ...string) error { func (ic *installContext) CombinedOutputLines(command string, args ...string) ([]string, error) { cmd := exec.Command( - "docker", + container.Engine, append( []string{"exec", ic.containerID, command}, args..., @@ -240,7 +244,7 @@ func (c *BuildContext) buildImage(dir string) error { // build the image, tagged as tagImageAs, using the our tempdir as the context log.Info("Starting image build ...") // create build container - // NOTE: we are using docker run + docker commit so we can install + // NOTE: we are using ContainerEngine run + ContainerEngine commit so we can install // debians without permanently copying them into the image. // if docker gets proper squash support, we can rm them instead // This also allows the KubeBit implementations to perform programmatic @@ -249,7 +253,7 @@ func (c *BuildContext) buildImage(dir string) error { // ensure we will delete it if containerID != "" { defer func() { - exec.Command("docker", "rm", "-f", "-v", containerID).Run() + exec.Command(container.Engine, "rm", "-f", "-v", containerID).Run() }() } if err != nil { @@ -259,7 +263,7 @@ func (c *BuildContext) buildImage(dir string) error { // helper we will use to run "build steps" execInBuild := func(command ...string) error { - cmd := exec.Command("docker", + cmd := exec.Command(container.Engine, append( []string{"exec", containerID}, command..., @@ -306,7 +310,7 @@ func (c *BuildContext) buildImage(dir string) error { } // Save the image changes to a new image - cmd := exec.Command("docker", "commit", containerID, c.image) + cmd := exec.Command(container.Engine, "commit", containerID, c.image) exec.InheritOutput(cmd) if err = cmd.Run(); err != nil { log.Errorf("Image build Failed! %v", err) @@ -328,7 +332,7 @@ func (c *BuildContext) prePullImages(dir, containerID string) error { // helpers to run things in the build container execInBuild := func(command ...string) error { - cmd := exec.Command("docker", + cmd := exec.Command(container.Engine, append( []string{"exec", containerID}, command..., @@ -338,7 +342,7 @@ func (c *BuildContext) prePullImages(dir, containerID string) error { return cmd.Run() } combinedOutputLinesInBuild := func(command ...string) ([]string, error) { - cmd := exec.Command("docker", + cmd := exec.Command(container.Engine, append( []string{"exec", containerID}, command..., @@ -394,14 +398,14 @@ func (c *BuildContext) prePullImages(dir, containerID string) error { for i, image := range requiredImages { if !builtImages.Has(image) { fmt.Printf("Pulling: %s\n", image) - err := docker.Pull(image, 2) + err := container.Pull(image, 2) if err != nil { return err } // TODO(bentheelder): generate a friendlier name pullName := fmt.Sprintf("%d.tar", i) pullTo := path.Join(imagesDir, pullName) - err = docker.Save(image, pullTo) + err = container.Save(image, pullTo) if err != nil { return err } @@ -410,17 +414,17 @@ func (c *BuildContext) prePullImages(dir, containerID string) error { } // Create the /kind/images directory inside the container. - if err = execInBuild("mkdir", "-p", DockerImageArchives); err != nil { + if err = execInBuild("mkdir", "-p", ContainerImageArchives); err != nil { log.Errorf("Image build Failed! %v", err) return err } - movePulled = append(movePulled, DockerImageArchives) + movePulled = append(movePulled, ContainerImageArchives) if err := execInBuild(movePulled...); err != nil { return err } // make sure we own the tarballs // TODO(bentheelder): someday we might need a different user ... - if err = execInBuild("chown", "-R", "root:root", DockerImageArchives); err != nil { + if err = execInBuild("chown", "-R", "root:root", ContainerImageArchives); err != nil { log.Errorf("Image build Failed! %v", err) return err } @@ -430,8 +434,8 @@ func (c *BuildContext) prePullImages(dir, containerID string) error { func (c *BuildContext) createBuildContainer(buildDir string) (id string, err error) { // attempt to explicitly pull the image if it doesn't exist locally // we don't care if this errors, we'll still try to run which also pulls - _, _ = docker.PullIfNotPresent(c.baseImage, 4) - id, err = docker.Run( + _, _ = container.PullIfNotPresent(c.baseImage, 4) + id, err = container.Run( c.baseImage, []string{ "-d", // make the client exit while the container continues to run diff --git a/pkg/cluster/context.go b/pkg/cluster/context.go index d3d31a567d..ee42a6bcb7 100644 --- a/pkg/cluster/context.go +++ b/pkg/cluster/context.go @@ -34,13 +34,11 @@ import ( logutil "sigs.k8s.io/kind/pkg/log" ) -// Context is used to create / manipulate kubernetes-in-docker clusters -// See: NewContext() type Context struct { *meta.ClusterMeta } -// similar to valid docker container names, but since we will prefix +// similar to valid container names, but since we will prefix // and suffix this name, we can relax it a little // see NewContext() for usage // https://godoc.org/github.com/docker/docker/daemon/names#pkg-constants @@ -100,7 +98,7 @@ func (c *Context) ClusterName() string { return fmt.Sprintf("kind-%s", c.Name()) } -// Create provisions and starts a kubernetes-in-docker cluster +// Create provisions and starts a kind cluster func (c *Context) Create(cfg *config.Config, retain bool, wait time.Duration) error { // default config fields (important for usage as a library, where the config // may be constructed in memory rather than from disk) @@ -175,7 +173,7 @@ func (c *Context) Create(cfg *config.Config, retain bool, wait time.Duration) er return nil } -// Delete tears down a kubernetes-in-docker cluster +// Delete tears down a kind cluster func (c *Context) Delete() error { n, err := c.ListNodes() if err != nil { diff --git a/pkg/cluster/doc.go b/pkg/cluster/doc.go index af19392fde..9ef9420339 100644 --- a/pkg/cluster/doc.go +++ b/pkg/cluster/doc.go @@ -14,5 +14,5 @@ See the License for the specific language governing permissions and limitations under the License. */ -// Package cluster implements kind kubernetes-in-docker cluster management +// Package cluster implements kind cluster management package cluster diff --git a/pkg/cluster/internal/create/createcontext.go b/pkg/cluster/internal/create/createcontext.go index 9dcf7a0653..6e6b1ca180 100644 --- a/pkg/cluster/internal/create/createcontext.go +++ b/pkg/cluster/internal/create/createcontext.go @@ -27,7 +27,7 @@ import ( "sigs.k8s.io/kind/pkg/cluster/config" "sigs.k8s.io/kind/pkg/cluster/internal/meta" "sigs.k8s.io/kind/pkg/cluster/nodes" - "sigs.k8s.io/kind/pkg/docker" + "sigs.k8s.io/kind/pkg/container" logutil "sigs.k8s.io/kind/pkg/log" ) @@ -42,7 +42,7 @@ type Context struct { WaitForReady time.Duration // Wait for the control plane node to be ready. } -// Exec actions on kubernetes-in-docker cluster +// Exec actions on KIND cluster // TODO(bentheelder): refactor this further // Actions are repetitive, high level abstractions/workflows composed // by one or more lower level tasks, that automatically adapt to the @@ -103,7 +103,7 @@ func (cc *Context) EnsureNodeImages() { // attempt to explicitly pull the image if it doesn't exist locally // we don't care if this errors, we'll still try to run which also pulls - _, _ = docker.PullIfNotPresent(configNode.Image, 4) + _, _ = container.PullIfNotPresent(configNode.Image, 4) // marks the images as already pulled images[configNode.Image] = true diff --git a/pkg/cluster/logs/logs.go b/pkg/cluster/logs/logs.go index 7e36e30490..8d09271ee7 100644 --- a/pkg/cluster/logs/logs.go +++ b/pkg/cluster/logs/logs.go @@ -23,6 +23,7 @@ import ( "sync" "sigs.k8s.io/kind/pkg/cluster/nodes" + "sigs.k8s.io/kind/pkg/container" "sigs.k8s.io/kind/pkg/exec" "sigs.k8s.io/kind/pkg/util" ) @@ -56,10 +57,10 @@ func Collect(nodes []nodes.Node, dir string) error { // construct a slice of methods to collect logs fns := []errFn{ // TODO(bentheelder): record the kind version here as well - // record info about the host docker + // record info about the host container engine execToPathFn( - exec.Command("docker", "info"), - "docker-info.txt", + exec.Command(container.Engine, "info"), + "container-info.txt", ), } // add a log collection method for each node @@ -70,7 +71,7 @@ func Collect(nodes []nodes.Node, dir string) error { return coalesce( // record info about the node container execToPathFn( - exec.Command("docker", "inspect", name), + exec.Command(container.Engine, "inspect", name), filepath.Join(name, "inspect.json"), ), // grab all of the node logs @@ -91,7 +92,7 @@ func Collect(nodes []nodes.Node, dir string) error { filepath.Join(name, "docker.log"), ), // grab all container / pod logs - // NOTE: we cannot just docker cp this directory because + // NOTE: we cannot just cp this directory because // it is mostly symlinks, so we must list and resolve the symlinks func() error { // collect up file names diff --git a/pkg/cluster/nodes/create.go b/pkg/cluster/nodes/create.go index 9ba8904105..72dec9c99a 100644 --- a/pkg/cluster/nodes/create.go +++ b/pkg/cluster/nodes/create.go @@ -23,7 +23,7 @@ import ( "github.com/pkg/errors" "sigs.k8s.io/kind/pkg/cluster/internal/haproxy" "sigs.k8s.io/kind/pkg/cluster/internal/kubeadm" - "sigs.k8s.io/kind/pkg/docker" + "sigs.k8s.io/kind/pkg/container" ) // FromID creates a node handle from the node (container's) ID @@ -110,7 +110,7 @@ func createNode(name, image, clusterLabel string, extraArgs ...string) (handle * // running containers in a container requires privileged // NOTE: we could try to replicate this with --cap-add, and use less // privileges, but this flag also changes some mounts that are necessary - // including some ones docker would otherwise do by default. + // including some ones the container engine would otherwise do by default. // for now this is what we want. in the future we may revisit this. "--privileged", "--security-opt", "seccomp=unconfined", // also ignore seccomp @@ -129,13 +129,13 @@ func createNode(name, image, clusterLabel string, extraArgs ...string) (handle * // adds node specific args runArgs = append(runArgs, extraArgs...) - if docker.UsernsRemap() { + if container.UsernsRemap() { // We need this argument in order to make this command work // in systems that have userns-remap enabled on the docker daemon runArgs = append(runArgs, "--userns=host") } - id, err := docker.Run( + id, err := container.Run( image, runArgs, []string{ @@ -153,7 +153,7 @@ func createNode(name, image, clusterLabel string, extraArgs ...string) (handle * } } if err != nil { - return handle, errors.Wrap(err, "docker run error") + return handle, errors.Wrap(err, "container run error") } // Deletes the machine-id embedded in the node image and regenerate a new one. diff --git a/pkg/cluster/nodes/node.go b/pkg/cluster/nodes/node.go index 59599e7476..4b5e9121b5 100644 --- a/pkg/cluster/nodes/node.go +++ b/pkg/cluster/nodes/node.go @@ -32,7 +32,7 @@ import ( "k8s.io/apimachinery/pkg/util/version" - "sigs.k8s.io/kind/pkg/docker" + "sigs.k8s.io/kind/pkg/container" "sigs.k8s.io/kind/pkg/exec" ) @@ -41,7 +41,7 @@ import ( // It should not be manually instantiated // Node impleemnts exec.Cmder type Node struct { - // must be one of docker container ID or name + // must be one of container ID or name nameOrID string // cached node info etc. nodeCache @@ -50,10 +50,10 @@ type Node struct { // assert Node implements Cmder var _ exec.Cmder = &Node{} -// Cmder returns an exec.Cmder that runs on the node via docker exec +// Cmder returns an exec.Cmder that runs on the node via ContainerEngine exec func (n *Node) Cmder() exec.Cmder { if n.nodeCache.containerCmder == nil { - n.nodeCache.containerCmder = docker.ContainerCmder(n.nameOrID) + n.nodeCache.containerCmder = container.Cmder(n.nameOrID) } return n.nodeCache.containerCmder } @@ -80,12 +80,12 @@ func (n *Node) String() string { // SignalStart sends SIGUSR1 to the node, which signals our entrypoint to boot // see images/node/entrypoint func (n *Node) SignalStart() error { - return docker.Kill("SIGUSR1", n.nameOrID) + return container.Kill("SIGUSR1", n.nameOrID) } // CopyTo copies the source file on the host to dest on the node func (n *Node) CopyTo(source, dest string) error { - return docker.CopyTo(source, n.nameOrID, dest) + return container.CopyTo(source, n.nameOrID, dest) } // CopyFrom copies the source file on the node to dest on the host @@ -93,12 +93,15 @@ func (n *Node) CopyTo(source, dest string) error { // but this should go away when kubeadm automatic copy certs lands, // otherwise it should be refactored in something more robust in the long term func (n *Node) CopyFrom(source, dest string) error { - return docker.CopyFrom(n.nameOrID, source, dest) + return container.CopyFrom(n.nameOrID, source, dest) } // WaitForDocker waits for Docker to be ready on the node // it returns true on success, and false on a timeout func (n *Node) WaitForDocker(until time.Time) bool { + if container.Engine != "docker" { + return true + } return tryUntil(until, func() bool { cmd := n.Command("systemctl", "is-active", "docker") out, err := exec.CombinedOutputLines(cmd) @@ -120,16 +123,16 @@ func tryUntil(until time.Time, try func() bool) bool { return false } -// LoadImages loads image tarballs stored on the node into docker on the node +// LoadImages loads image tarballs stored on the node into container storage on the node func (n *Node) LoadImages() { - // load images cached on the node into docker + // load images cached on the node into Container Storage if err := n.Command( "find", "/kind/images", "-name", "*.tar", - "-exec", "docker", "load", "-i", "{}", ";", + "-exec", container.Engine, "load", "-i", "{}", ";", ).Run(); err != nil { - log.Warningf("Failed to preload docker images: %v", err) + log.Warningf("Failed to preload container images: %v", err) return } @@ -153,17 +156,17 @@ func (n *Node) LoadImages() { // TODO(bentheelder): this is a bit gross, move this logic out of bash if err := n.Command( "/bin/bash", "-c", - `docker images --format='{{.Repository}}:{{.Tag}}' | grep -v amd64 | xargs -L 1 -I '{}' /bin/bash -c 'docker tag "{}" "$(echo "{}" | sed s/:/-amd64:/)"'`, + fmt.Sprintf(`%s images --format='{{.Repository}}:{{.Tag}}' | grep -v amd64 | xargs -L 1 -I '{}' /bin/bash -c '%s tag "{}" "$(echo "{}" | sed s/:/-amd64:/)"'`, container.Engine, container.Engine), ).Run(); err != nil { - log.Warningf("Failed to re-tag docker images: %v", err) + log.Warningf("Failed to re-tag container images: %v", err) } } // FixMounts will correct mounts in the node container to meet the right -// sharing and permissions for systemd and Docker / Kubernetes +// sharing and permissions for systemd and Containers / Kubernetes func (n *Node) FixMounts() error { // Check if userns-remap is enabled - if docker.UsernsRemap() { + if container.UsernsRemap() { // The binary /bin/mount should be owned by root:root in order to execute // the following mount commands if err := n.Command("chown", "root:root", "/bin/mount").Run(); err != nil { @@ -177,7 +180,7 @@ func (n *Node) FixMounts() error { // systemd-in-a-container should have read only /sys // https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/ - // however, we need other things from `docker run --privileged` ... + // however, we need other things from `container run --privileged` ... // and this flag also happens to make /sys rw, amongst other things if err := n.Command("mount", "-o", "remount,ro", "/sys").Run(); err != nil { return err @@ -221,7 +224,7 @@ func (n *Node) IP() (ip string, err error) { return n.nodeCache.ip, nil } // retrive the IP address of the node using docker inspect - lines, err := docker.Inspect(n.nameOrID, "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}") + lines, err := container.Inspect(n.nameOrID, "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}") if err != nil { return "", errors.Wrap(err, "failed to get file") } @@ -241,7 +244,7 @@ func (n *Node) Ports(containerPort int) (hostPort int, err error) { return hostPort, nil } // retrive the specific port mapping using docker inspect - lines, err := docker.Inspect(n.nameOrID, fmt.Sprintf("{{(index (index .NetworkSettings.Ports \"%d/tcp\") 0).HostPort}}", containerPort)) + lines, err := container.Inspect(n.nameOrID, fmt.Sprintf("{{(index (index .NetworkSettings.Ports \"%d/tcp\") 0).HostPort}}", containerPort)) if err != nil { return -1, errors.Wrap(err, "failed to get file") } diff --git a/pkg/cluster/nodes/nodes.go b/pkg/cluster/nodes/nodes.go index e0d021671c..c8b6988d9d 100644 --- a/pkg/cluster/nodes/nodes.go +++ b/pkg/cluster/nodes/nodes.go @@ -23,7 +23,7 @@ import ( "github.com/pkg/errors" "sigs.k8s.io/kind/pkg/cluster/constants" - + "sigs.k8s.io/kind/pkg/container" "sigs.k8s.io/kind/pkg/exec" ) @@ -37,7 +37,7 @@ func Delete(nodes ...Node) error { ids = append(ids, node.nameOrID) } cmd := exec.Command( - "docker", + container.Engine, append( []string{ "rm", @@ -51,7 +51,7 @@ func Delete(nodes ...Node) error { } // List returns the list of container IDs for the kind "nodes", optionally -// filtered by docker ps filters +// filtered by container engine ps filters // https://docs.docker.com/engine/reference/commandline/ps/#filtering func List(filters ...string) ([]Node, error) { res := []Node{} @@ -84,7 +84,7 @@ func list(visit func(string, *Node), filters ...string) error { for _, filter := range filters { args = append(args, "--filter", filter) } - cmd := exec.Command("docker", args...) + cmd := exec.Command(container.Engine, args...) lines, err := exec.CombinedOutputLines(cmd) if err != nil { return errors.Wrap(err, "failed to list nodes") diff --git a/pkg/docker/archive.go b/pkg/container/archive.go similarity index 89% rename from pkg/docker/archive.go rename to pkg/container/archive.go index 8d6bb5a39a..a26769bf25 100644 --- a/pkg/docker/archive.go +++ b/pkg/container/archive.go @@ -14,9 +14,9 @@ See the License for the specific language governing permissions and limitations under the License. */ -// Package docker contains helpers for working with docker +// Package container contains helpers for working with container engines // This package has no stability guarantees whatsoever! -package docker +package container import ( "archive/tar" @@ -29,8 +29,8 @@ import ( "github.com/pkg/errors" ) -// GetArchiveTags obtains a list of "repo:tag" docker image tags from a -// given docker image archive (tarball) path +// GetArchiveTags obtains a list of "repo:tag" container image tags from a +// given container image archive (tarball) path // compatible with all known specs: // https://github.com/moby/moby/blob/master/image/spec/v1.0.md // https://github.com/moby/moby/blob/master/image/spec/v1.1.md diff --git a/pkg/docker/cp.go b/pkg/container/cp.go similarity index 95% rename from pkg/docker/cp.go rename to pkg/container/cp.go index 3496425002..eb0b0944c1 100644 --- a/pkg/docker/cp.go +++ b/pkg/container/cp.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "sigs.k8s.io/kind/pkg/exec" @@ -23,7 +23,7 @@ import ( // CopyTo copies the file at hostPath to the container at destPath func CopyTo(hostPath, containerNameOrID, destPath string) error { cmd := exec.Command( - "docker", "cp", + Engine, "cp", hostPath, // from the source file containerNameOrID+":"+destPath, // to the node, at dest ) @@ -33,7 +33,7 @@ func CopyTo(hostPath, containerNameOrID, destPath string) error { // CopyFrom copies the file or dir in the container at srcPath to the host at hostPath func CopyFrom(containerNameOrID, srcPath, hostPath string) error { cmd := exec.Command( - "docker", "cp", + Engine, "cp", containerNameOrID+":"+srcPath, // from the node, at src hostPath, // to the host ) diff --git a/pkg/docker/doc.go b/pkg/container/doc.go similarity index 87% rename from pkg/docker/doc.go rename to pkg/container/doc.go index a4a2b89325..8cc4540267 100644 --- a/pkg/docker/doc.go +++ b/pkg/container/doc.go @@ -14,6 +14,6 @@ See the License for the specific language governing permissions and limitations under the License. */ -// Package docker contains helpers for working with docker +// Package container contains helpers for working with container engines // This package has no stability guarantees whatsoever! -package docker +package container diff --git a/pkg/docker/exec.go b/pkg/container/exec.go similarity index 86% rename from pkg/docker/exec.go rename to pkg/container/exec.go index 5d12450274..c5eebc3d1a 100644 --- a/pkg/docker/exec.go +++ b/pkg/container/exec.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "io" @@ -22,13 +22,13 @@ import ( "sigs.k8s.io/kind/pkg/exec" ) -// containerCmder implements exec.Cmder for docker containers +// containerCmder implements exec.Cmder for containers type containerCmder struct { nameOrID string } -// ContainerCmder creates a new exec.Cmder against a docker container -func ContainerCmder(containerNameOrID string) exec.Cmder { +// Cmder creates a new exec.Cmder against a container +func Cmder(containerNameOrID string) exec.Cmder { return &containerCmder{ nameOrID: containerNameOrID, } @@ -42,7 +42,7 @@ func (c *containerCmder) Command(command string, args ...string) exec.Cmd { } } -// containerCmd implements exec.Cmd for docker containers +// containerCmd implements exec.Cmd for containers type containerCmd struct { nameOrID string // the container name or ID command string @@ -76,7 +76,7 @@ func (c *containerCmd) Run() error { args = append(args, "-e", env) } // specify the container and command, after this everything will be - // args the the command in the container rather than to docker + // args the the command in the container rather than to container engine args = append( args, c.nameOrID, // ... against the container @@ -87,7 +87,7 @@ func (c *containerCmd) Run() error { // finally, with the caller args c.args..., ) - cmd := exec.Command("docker", args...) + cmd := exec.Command(Engine, args...) if c.stdin != nil { cmd.SetStdin(c.stdin) } diff --git a/pkg/container/init.go b/pkg/container/init.go new file mode 100644 index 0000000000..c0029334a9 --- /dev/null +++ b/pkg/container/init.go @@ -0,0 +1,42 @@ +/* +Copyright 2018 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package container contains helpers for working with container engines +// This package has no stability guarantees whatsoever! +package container + +import ( + "os" + "os/exec" +) + +// Engine is the container engine to use to launch containers +// Currently we support podman and docker. Kind will use podman if it is +// found in $PATH else it fall back to docker. +var Engine string + +func init() { + if Engine = os.Getenv("KIND_ENGINE"); Engine != "" { + return + } + for _, engine := range []string{"docker", "podman"} { + var err error + if Engine, err = exec.LookPath(engine); err == nil { + return + } + } + Engine = "docker" +} diff --git a/pkg/docker/inspect.go b/pkg/container/inspect.go similarity index 93% rename from pkg/docker/inspect.go rename to pkg/container/inspect.go index e5ef40424d..f4eac42a5b 100644 --- a/pkg/docker/inspect.go +++ b/pkg/container/inspect.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "fmt" @@ -24,7 +24,7 @@ import ( // Inspect return low-level information on containers func Inspect(containerNameOrID, format string) ([]string, error) { - cmd := exec.Command("docker", "inspect", + cmd := exec.Command(Engine, "inspect", "-f", // format fmt.Sprintf("'%s'", format), containerNameOrID, // ... against the "node" container diff --git a/pkg/docker/kill.go b/pkg/container/kill.go similarity index 95% rename from pkg/docker/kill.go rename to pkg/container/kill.go index c3ce2a4675..8f634b2fe3 100644 --- a/pkg/docker/kill.go +++ b/pkg/container/kill.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "sigs.k8s.io/kind/pkg/exec" @@ -23,7 +23,7 @@ import ( // Kill sends the named signal to the container func Kill(signal, containerNameOrID string) error { cmd := exec.Command( - "docker", "kill", + Engine, "kill", "-s", signal, containerNameOrID, ) diff --git a/pkg/docker/pull.go b/pkg/container/pull.go similarity index 90% rename from pkg/docker/pull.go rename to pkg/container/pull.go index cc48945f88..280592836b 100644 --- a/pkg/docker/pull.go +++ b/pkg/container/pull.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "time" @@ -31,7 +31,7 @@ func PullIfNotPresent(image string, retries int) (pulled bool, err error) { // TODO(bentheelder): switch most (all) of the logging here to debug level // once we have configurable log levels // if this did not return an error, then the image exists locally - cmd := exec.Command("docker", "inspect", "--type=image", image) + cmd := exec.Command(Engine, "inspect", "--type=image", image) if err := cmd.Run(); err == nil { log.Infof("Image: %s present locally", image) return false, nil @@ -43,14 +43,14 @@ func PullIfNotPresent(image string, retries int) (pulled bool, err error) { // Pull pulls an image, retrying up to retries times func Pull(image string, retries int) error { log.Infof("Pulling image: %s ...", image) - err := exec.Command("docker", "pull", image).Run() + err := exec.Command(Engine, "pull", image).Run() // retry pulling up to retries times if necessary if err != nil { for i := 0; i < retries; i++ { time.Sleep(time.Second * time.Duration(i+1)) log.WithError(err).Infof("Trying again to pull image: %s ...", image) // TODO(bentheelder): add some backoff / sleep? - err = exec.Command("docker", "pull", image).Run() + err = exec.Command(Engine, "pull", image).Run() if err == nil { break } diff --git a/pkg/docker/run.go b/pkg/container/run.go similarity index 78% rename from pkg/docker/run.go rename to pkg/container/run.go index 785ab194ff..7bdf4da473 100644 --- a/pkg/docker/run.go +++ b/pkg/container/run.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "regexp" @@ -25,17 +25,17 @@ import ( "sigs.k8s.io/kind/pkg/exec" ) -// Docker container IDs are hex, more than one character, and on their own line +// Container IDs are hex, more than one character, and on their own line var containerIDRegex = regexp.MustCompile("^[a-f0-9]+$") -// Run creates a container with "docker run", with some error handling +// Run creates a container with "ContainerEngine run", with some error handling // it will return the ID of the created container if any, even on error func Run(image string, runArgs []string, containerArgs []string) (id string, err error) { args := []string{"run"} args = append(args, runArgs...) args = append(args, image) args = append(args, containerArgs...) - cmd := exec.Command("docker", args...) + cmd := exec.Command(Engine, args...) output, err := exec.CombinedOutputLines(cmd) if err != nil { // log error output if there was any @@ -44,10 +44,10 @@ func Run(image string, runArgs []string, containerArgs []string) (id string, err } return "", err } - // if docker created a container the id will be the first line and match + // if the container engine created a container the id will be the first line and match // validate the output and get the id if len(output) < 1 { - return "", errors.New("failed to get container id, received no output from docker run") + return "", errors.Errorf("failed to get container id, received no output from %s run", Engine) } if !containerIDRegex.MatchString(output[0]) { return "", errors.Errorf("failed to get container id, output did not match: %v", output) diff --git a/pkg/docker/save.go b/pkg/container/save.go similarity index 83% rename from pkg/docker/save.go rename to pkg/container/save.go index ded4737756..d275f6bb74 100644 --- a/pkg/docker/save.go +++ b/pkg/container/save.go @@ -14,13 +14,13 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "sigs.k8s.io/kind/pkg/exec" ) -// Save saves image to dest, as in `docker save` +// Save saves image to dest, as in `Engine save` func Save(image, dest string) error { - return exec.Command("docker", "save", "-o", dest, image).Run() + return exec.Command(Engine, "save", "-o", dest, image).Run() } diff --git a/pkg/docker/userns_remap.go b/pkg/container/userns_remap.go similarity index 90% rename from pkg/docker/userns_remap.go rename to pkg/container/userns_remap.go index d0c42bb33a..eb00e7a69d 100644 --- a/pkg/docker/userns_remap.go +++ b/pkg/container/userns_remap.go @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. */ -package docker +package container import ( "strings" @@ -24,7 +24,7 @@ import ( // UsernsRemap checks if userns-remap is enabled in dockerd func UsernsRemap() bool { - cmd := exec.Command("docker", "info", "--format", "'{{json .SecurityOptions}}'") + cmd := exec.Command(Engine, "info", "--format", "'{{json .SecurityOptions}}'") lines, err := exec.CombinedOutputLines(cmd) if err != nil { return false