Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update docs #710

Merged
merged 1 commit into from
Nov 1, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,24 @@ Read the proposal from https://github.com/moby/moby/issues/32925

Introductory blog post https://blog.mobyproject.org/introducing-buildkit-17e056cc5317

:information_source: If you are visiting this repo for the usage of experimental Dockerfile features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`, please refer to [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md).

### Used by

[Moby](https://github.com/moby/moby/pull/37151)
[Moby & Docker](https://github.com/moby/moby/pull/37151)

[img](https://github.com/genuinetools/img)

[OpenFaaS Cloud](https://github.com/openfaas/openfaas-cloud)

[container build interface](https://github.com/containerbuilding/cbi)

[Knative Build Templates](https://github.com/knative/build-templates)

[boss](https://github.com/crosbymichael/boss)

[Rio](https://github.com/rancher/rio) (on roadmap)

### Quick start

Dependencies:
Expand Down Expand Up @@ -79,6 +87,7 @@ See [`solver/pb/ops.proto`](./solver/pb/ops.proto) for the format definition.
Currently, following high-level languages has been implemented for LLB:

- Dockerfile (See [Exploring Dockerfiles](#exploring-dockerfiles))
- [Buildpacks](https://github.com/tonistiigi/buildkit-pack)
- (open a PR to add your own language)

For understanding the basics of LLB, `examples/buildkit*` directory contains scripts that define how to build different configurations of BuildKit itself and its dependencies using the `client` package. Running one of these scripts generates a protobuf definition of a build graph. Note that the script itself does not execute any steps of the build.
Expand Down Expand Up @@ -145,6 +154,10 @@ buildctl build --frontend=gateway.v0 --frontend-opt=source=tonistiigi/dockerfile
buildctl build --frontend gateway.v0 --frontend-opt=source=tonistiigi/dockerfile --frontend-opt=context=git://github.com/moby/moby --frontend-opt build-arg:APT_MIRROR=cdn-fastly.deb.debian.org
````

##### Building a Dockerfile with experimental features like `RUN --mount=type=(bind|cache|tmpfs|secret|ssh)`

See [`frontend/dockerfile/docs/experimental.md`](frontend/dockerfile/docs/experimental.md).

### Exporters

By default, the build result and intermediate cache will only remain internally in BuildKit. Exporter needs to be specified to retrieve the result.
Expand Down Expand Up @@ -207,15 +220,22 @@ buildctl debug workers -v

BuildKit can also be used by running the `buildkitd` daemon inside a Docker container and accessing it remotely. The client tool `buildctl` is also available for Mac and Windows.

We provide `buildkitd` container images as [`moby/buildkit`](https://hub.docker.com/r/moby/buildkit/tags/):

* `moby/buildkit:latest`: built from the latest regular [release](https://github.com/moby/buildkit/releases)
* `moby/buildkit:rootless`: same as `latest` but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md)
* `moby/buildkit:master`: built from the master branch
* `moby/buildkit:master-rootless`: same as master but runs as an unprivileged user, see [`docs/rootless.md`](docs/rootless.md)

To run daemon in a container:

```
docker run -d --privileged -p 1234:1234 tonistiigi/buildkit --addr tcp://0.0.0.0:1234
docker run -d --privileged -p 1234:1234 moby/buildkit:latest --addr tcp://0.0.0.0:1234
export BUILDKIT_HOST=tcp://0.0.0.0:1234
buildctl build --help
```

The `tonistiigi/buildkit` image can be built locally using the Dockerfile in `./hack/dockerfiles/test.Dockerfile`.
The images can be also built locally using `./hack/dockerfiles/test.Dockerfile` (or `./hack/dockerfiles/test.buildkit.Dockerfile` if you already have BuildKit).

### Opentracing support

Expand Down
10 changes: 6 additions & 4 deletions docs/rootless.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@

Requirements:
- runc `a00bf0190895aa465a5fbed0268888e2c8ddfe85` (Oct 15, 2018) or later
- Some distros such as Debian (excluding Ubuntu) and Arch Linux require `echo 1 > /proc/sys/kernel/unprivileged_userns_clone`
- `newuidmap` and `newgidmap` need to be installed on the host. These commands are provided by the `uidmap` package.
- Some distros such as Debian (excluding Ubuntu) and Arch Linux require `sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone"`.
- RHEL/CentOS 7 requires `sudo sh -c "echo 28633 > /proc/sys/user/max_user_namespaces"`. You may also need `sudo grubby --args="namespace.unpriv_enable=1 user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"`.
- `newuidmap` and `newgidmap` need to be installed on the host. These commands are provided by the `uidmap` package. For RHEL/CentOS 7, RPM is not officially provided but available at https://copr.fedorainfracloud.org/coprs/vbatts/shadow-utils-newxidmap/ .
- `/etc/subuid` and `/etc/subgid` should contain >= 65536 sub-IDs. e.g. `penguin:231072:65536`.
- To run in a Docker container with non-root `USER`, `docker run --privileged` is still required. See also Jessie's blog: https://blog.jessfraz.com/post/building-container-images-securely-on-kubernetes/

Expand Down Expand Up @@ -56,9 +57,10 @@ $ build-using-dockerfile --buildkit-addr unix:///run/user/1001/buildkit/buildkit

## Set up (using a container)

Docker image is available as [`moby/buildkit:rootless`](https://hub.docker.com/r/moby/buildkit/tags/).

```
$ docker build -t buildkit-rootless --target rootless -f hack/dockerfiles/test.Dockerfile .
$ docker run --name buildkitd -d --privileged -p 1234:1234 buildkit-rootless --addr tcp://0.0.0.0:1234
$ docker run --name buildkitd -d --privileged -p 1234:1234 moby/buildkit:rootless --addr tcp://0.0.0.0:1234
```

`docker run` requires `--privileged` but the BuildKit daemon is executed as a normal user.
Expand Down
129 changes: 129 additions & 0 deletions frontend/dockerfile/docs/experimental.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# Dockerfile frontend experimental syntaxes
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add another file that describes the dockerfile image release process? Something that could be linked from https://github.com/docker/cli/pull/1493/files#diff-4fc21624aa5dadd153e2780910da7c7dR304 . Eg. what releases happen to docker/dockerfile-upstream, the features that produce nightly builds etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could also be a second PR.


## Note for Docker users

If you are using Docker v18.06 or later, BuildKit mode can be enabled by setting `export DOCKER_BUILDKIT=1` on the client side.
Docker v18.06 also requires the daemon to be [running in experimental mode](https://docs.docker.com/engine/reference/commandline/dockerd/#description).

You need to use `docker build` CLI instead of `buildctl` CLI mentioned in this document.
See [the `docker build` document](https://docs.docker.com/engine/reference/commandline/build/) for the usage.

## Use experimental Dockerfile frontend
The features mentioned in this document are experimentally available as [`docker/dockerfile-upstream:experimental`](https://hub.docker.com/r/docker/dockerfile-upstream/tags/) image.

To use the experimental features, the first line of your Dockerfile needs to be `# syntax=docker/dockerfile-upstream:experimental`.
As the experimental syntaxes may change in future revisions, you may want to pin the image to a specific revision.

See also [#528](https://github.com/moby/buildkit/issues/528) for further information about planned `docker/dockerfile` releases.

## Experimental syntaxes

### `RUN --mount=type=bind` (the default mount type)

This mount type allows binding directories (read-only) in the context or in an image to the build container.

|Option |Description|
|---------------------|-----------|
|`target` (required) | Mount path.|
|`source` | Source path in the `from`. Defaults to the root of the `from`.|
|`from` | Build stage or image name for the root of the source. Defaults to the build context.|


### `RUN --mount=type=cache`

This mount type allows the build container to cache directories for compilers and package managers.

|Option |Description|
|---------------------|-----------|
|`id` | Optional ID to identify separate/different caches|
|`target` (required) | Mount path.|
|`ro`,`readonly` | Read-only if set.|
|`sharing` | One of `shared`, `private`, or `locked`. Defaults to `shared`. A `shared` cache mount can be used concurrently by multiple writers. `private` creates a new mount if there are multiple writers. `locked` pauses the second writer until the first one releases the mount.|


#### Example: cache Go packages

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM golang
...
RUN --mount=type=cache,target=/root/.cache/go-build go build ...
```

#### Example: cache apt packages

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM ubuntu
RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache
RUN --mount=type=cache,target=/var/cache/apt --mount=type=cache,target=/var/lib/apt \
apt update && apt install -y gcc
```

### `RUN --mount=type=tmpfs`

This mount type allows mounting tmpfs in the build container.

|Option |Description|
|---------------------|-----------|
|`target` (required) | Mount path.|


### `RUN --mount=type=secret`

This mount type allows the build container to access secure files such as private keys without baking them into the image.

|Option |Description|
|---------------------|-----------|
|`id` | ID of the secret. Defaults to basename of the target path.|
|`target` | Mount path. Defaults to `/run/secrets/` + `id`.|
|`required` | If set to `true`, the instruction errors out when the secret is unavailable. Defaults to `false`.|


#### Example: access to S3

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM python:3
RUN pip install awscli
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ...
```

```console
$ buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. \
--secret id=aws,src=$HOME/.aws/credentials
```

### `RUN --mount=type=ssh`

This mount type allows the build container to access SSH keys via SSH agents, with support for passphrases.

|Option |Description|
|---------------------|-----------|
|`id` | ID of SSH agent socket or key. Defaults to "default".|
|`target` | SSH agent socket path. Defaults to `/run/buildkit/ssh_agent.${N}`.|
|`required` | If set to `true`, the instruction errors out when the key is unavailable. Defaults to `false`.|


#### Example: access to Gitlab

```dockerfile
# syntax = docker/dockerfile-upstream:experimental
FROM alpine
RUN apk add --no-cache openssh-client
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
RUN --mount=type=ssh ssh [email protected] | tee /hello
# "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here
```

```console
$ eval $(ssh-agent)
$ ssh-add ~/.ssh/id_rsa
(Input your passphrase here)
$ buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. \
--ssh default=$SSH_AUTH_SOCK
```

You can also specify a path to `*.pem` file on the host directly instead of `$SSH_AUTH_SOCK`.
However, pem files with passphrases are not supported.