Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add more buildkit docs #1493

Merged
merged 1 commit into from
Nov 21, 2018
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 117 additions & 1 deletion docs/reference/builder.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,28 @@ registries.
When you're done with your build, you're ready to look into [*Pushing a
repository to its registry*](https://docs.docker.com/engine/tutorials/dockerrepos/#/contributing-to-docker-hub).


## BuildKit

Starting with version 18.09, Docker supports a new backend for executing your
builds that is provided by the [moby/buildkit](https://github.com/moby/buildkit)
project. The BuildKit backend provides many benefits compared to the old
implementation. For example, BuildKit can:

* Detect and skip executing unused build stages
* Parallelize building independent build stages
* Incrementally transfer only the changed files in your build context between builds
* Detect and skip transferring unused files in your build context
* Use external Dockerfile implementations with many new features
* Avoid side-effects with rest of the API (intermediate images and containers)
* Prioritize your build cache for automatic pruning
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand the "Prioritize". Is this the configurable garbage collection?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is gc and prune --keep-storage that removes unused based on priority.


To use the BuildKit backend, you need to set an environment variable
`DOCKER_BUILDKIT=1` on the CLI before invoking `docker build`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is now also configurable in the CLI configuration file? (if so, we should have a link to that configuration somewhere). Could be in a new section "configure the Docker CLI to use the BuildKit builder" (e.g.)

Wondering which one we consider the "preferred" mechanism, and/or if we should call out (dis)advantages of both ways to configure.

(I'm ok with addressing that in a follow-up)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of it was covered by #1314


To learn about the experimental Dockerfile syntax available to BuildKit-based
builds [refer to the documentation in the BuildKit repository](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also for a follow-up; would love to have this integrated in the documentation somewhere.

If we link to GitHub, it would be good to link to a specific commit / tag to prevent the link from breaking in future


## Format

Here is the format of the `Dockerfile`:
Expand Down Expand Up @@ -224,10 +246,64 @@ following lines are all treated identically:
# dIrEcTiVe=value
```

The following parser directive is supported:
The following parser directives are supported:

* `syntax`
* `escape`

## syntax

# syntax=[remote image reference]

For example:

# syntax=docker/dockerfile
# syntax=docker/dockerfile:1.0
# syntax=docker.io/docker/dockerfile:1
# syntax=docker/dockerfile:1.0.0-experimental
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we include an example with a fully-qualified reference for Docker Hub as well? (docker.io/.....)

# syntax=example.com/user/repo:tag@sha256:abcdef...

This feature is only enabled if the [BuildKit](#buildkit) backend is used.

The syntax directive defines the location of the Dockerfile builder that is used for
building the current Dockerfile. The BuildKit backend allows to seamlessly use
external implementations of builders that are distributed as Docker images and
execute inside a container sandbox environment.

Custom Dockerfile implementation allows you to:
- Automatically get bugfixes without updating the daemon
- Make sure all users are using the same implementation to build your Dockerfile
- Use the latest features without updating the daemon
- Try out new experimental or third-party features

### Official releases

Docker distributes official versions of the images that can be used for building
Dockerfiles under `docker/dockerfile` repository on Docker Hub. There are two
channels where new images are released: stable and experimental.

Stable channel follows semantic versioning. For example:

- docker/dockerfile:1.0.0 - only allow immutable version 1.0.0
- docker/dockerfile:1.0 - allow versions 1.0.*
- docker/dockerfile:1 - allow versions 1.*.*
- docker/dockerfile:latest - latest release on stable channel

The experimental channel uses incremental versioning with the major and minor
component from the stable channel on the time of the release. For example:

- docker/dockerfile:1.0.1-experimental - only allow immutable version 1.0.1-experimental
- docker/dockerfile:1.0-experimental - latest experimental releases after 1.0
- docker/dockerfile:experimental - latest release on experimental channel

You should choose a channel that best fits your needs. If you only want
bugfixes, you should use `docker/dockerfile:1.0`. If you want to benefit from
experimental features, you should use the experimental channel. If you are using
the experimental channel, newer releases may not be backwards compatible, so it
is recommended to use an immutable full version variant.

For master builds and nightly feature releases refer to the description in [the source repository](https://github.com/moby/buildkit/blob/master/README.md).

## escape

# escape=\ (backslash)
Expand Down Expand Up @@ -1627,6 +1703,38 @@ RUN echo "Hello World"
When building this Dockerfile, the `HTTP_PROXY` is preserved in the
`docker history`, and changing its value invalidates the build cache.

### Automatic platform ARGs in the global scope

This feature is only available when using the [BuildKit](#buildkit) backend.

Docker predefines a set of `ARG` variables with information on the platform of
the node performing the build (build platform) and on the platform of the
resulting image (target platform). The target platform can be specified with
the `--platform` flag on `docker build`.

The following `ARG` variables are set automatically:

* `TARGETPLATFORM` - platform of the build result. Eg `linux/amd64`, `linux/arm/v7`, `windows/amd64`.
* `TARGETOS` - OS component of TARGETPLATFORM
* `TARGETARCH` - architecture component of TARGETPLATFORM
* `TARGETVARIANT` - variant component of TARGETPLATFORM
* `BUILDPLATFORM` - platform of the node performing the build.
* `BUILDOS` - OS component of BUILDPLATFORM
* `BUILDARCH` - OS component of BUILDPLATFORM
* `BUILDVARIANT` - OS component of BUILDPLATFORM

These arguments are defined in the global scope so are not automatically
available inside build stages or for your `RUN` commands. To expose one of
these arguments inside the build stage redefine it without value.

For example:

```Dockerfile
FROM alpine
ARG TARGETPLATFORM
RUN echo "I'm building for $TARGETPLATFORM"
```

### Impact on build caching

`ARG` variables are not persisted into the built image as `ENV` variables are.
Expand Down Expand Up @@ -1935,6 +2043,14 @@ required such as `zsh`, `csh`, `tcsh` and others.

The `SHELL` feature was added in Docker 1.12.

## External implementation features

This feature is only available when using the [BuildKit](#buildkit) backend.

Docker build supports experimental features like cache mounts, build secrets and
ssh forwarding that are enabled by using an external implementation of the
builder with a syntax directive. To learn about these features, [refer to the documentation in BuildKit repository](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md).

## Dockerfile examples

Below you can see some examples of Dockerfile syntax. If you're interested in
Expand Down