[dockerfile] Proposal: External Dockerfile frontend image versioning

[tonistiigi]

This is a proposal for how to release external Dockerfile frontends so that it makes sense both for users and people interested in contributing new features.

Note that the Dockerfile itself is not versioned here. This is a proposal for an authority who agrees to release the frontend and the guidelines it will follow. It isn't really BuildKit responsibility to do that and will likely move to a Docker organization in the future. Adding proposal to here as this is where the code is currently for maintenance reasons.

I'm using tonistiigi/dockerfile in the examples as a location of the image as this is where the image is currently and will probably be for a test period. The final image source recommended by BuildKit maintainers will likely be docker.io/library/dockerfile or docker.io/docker/dockerfile.

Proposal

There are three levels of Dockerfile frontend images. Feature branches, experimental channel and stable channel.

Stable

Stable channel follows the semantic versioning logic. This is the channel you should use if you are conservative about getting new features, want to make sure you have latest bugfixes and be sure that your Dockerfiles will be usable as is for a long time.

tonistiigi/dockerfile:v$major[.$minor[.$patch]]
tonistiigi/dockerfile:v1.0.1
tonistiigi/dockerfile:v1.1
tonistiigi/dockerfile:v1

The references with missing patch, minor components point to the latest image released. An image with a patch version will never be mutated. If there are new fixes, a new image with incremented patch number will be released. If we decide to expose new features, like new Dockerfile commands or flags, we will increase the minor number. The major number is updated, should we decide to make backward incompatible changes. Currently, there is no timeline for making a change that would break an existing Dockerfile.

If major or minor releases are made from the master branch. Patch releases are done from release branches with cherry-picks. The EOL terms for releases are not defined here.

Maintainers decide if a new feature is ready for the stable branch. Generally, a feature should be proven in the experimental channel for some period and have a positive reception from the community before graduating to stable.

v1.0.0 should match the feature set in Docker 18.06-ce , assuming that we will not introduce any Dockerfile features in the legacy builder on 18.09-ce.

Feature branches

Feature branches are the default way how new features can be added to the Dockerfile frontend implementation. Every new feature should be developed using a go build tag that allows to configure it. Contributors are welcome to provide refactoring that makes this build-time plugging easier to maintain.

Contributors adding a feature can release a demo version of the image supporting it under their own accounts or after their PR has been merged in the main codebase, they can request a daily feature release with their feature enabled.

tonistiigi/dockerfile:$yyyymmdd[.$N]-$featurename
tonistiigi/dockerfile:20180720-runmount

Once an image has been released it is immutable and will not be overwritten. If more than one release is needed per-day an incrementing number is added. If there are no changes in the last day, a release will be skipped.

These images are released straight from the master branch with the specific feature and experimental build tags enabled.

It should be quite easy to get a new feature into a feature branch but maintainers should still look out that it doesn't make the overall maintenance of the project harder.

Feature branches are designed for early adopters who want to try out a very new feature to provide feedback or to test it. It should be an exception to use them in an actual project.

Experimental channel

The experimental channel provides images with a collection of new features not part of stable channel yet.

The experimental channel is defined as a collection of feature build tags that are enabled together on build time.

Maintainers decide what tags get accepted to the channel.

tonistiigi/dockerfile:v$major.$minor.$N-experimental
tonistiigi/dockerfile:v1.1.123-experimental

The major and minor numbers are from the last stable branch. Instead of a patch number, the last number just increments with every release.

There are no backward compatibility guarantees between experimental releases. A release v1.1.3-experimental may introduce a new feature that is removed in v1.1.4-experimental.

Experimental images are released from the master branch. There is no predefined interval for the releases but they happen more frequently than in stable. A new release for stable always produces a release for experimental as well.

Experimental releases are for people who want to use the latest and greatest features currently available. Because the experimental releases are immutable they will also never break over time. The downside of using experimental channel is that the Dockerfile does not automatically pick up bugfixes and the author must update the Dockerfile to use a newer version themselves if they need the fixes.

Initially, there will be only one experimental channel called "experimental". If there is a need there may be more in the future with different sets of features.

To be accepted to the experimental channel a feature should be well documented with example usage. All experimental releases should be visible in a list where it is very clear to understand what features are part of that release and links to documentation on every feature. The same list should be visible for stable releases as well.

@AkihiroSuda @tiborvass @vdemeester @thaJeztah @dgageot

[tonistiigi]

cc @andrewhsu as well

[thaJeztah]

Thanks for writing this up!

Had given this a quick once-over last weekend, and need to look into it more in depth, some quick thoughts I had when reading;

• don't mix not-yet-approved/finalised syntaxes with finalised syntaxes in the same repository in the registry; i.e. better to have a separate org or repository for non-finalised syntaxes (so that we take away any confusion as to what's still a preview/experiment, and what not)
• how to deal with air gapped setups? should the images be mirrored in an internal registry? is there a way to load them without using a registry?
• really minor nit: although discussed once as proposal, the v prefix is not part of SemVer (versions are, e.g. 1.1.1, not v1.1.1
• for the experimental features; (just a suggestion) I love the "pseudo version" format that's used by vGo (0.0.0-yyyymmddhhmmss-<git-sha>); it's both valid SemVer, and provides a lot of information about the version it was built from.
• also for experimental features; if (see first bullet) using a separate org/repo is an option, the name of the feature could be in the repository-name instead of in the tag (dockerfile-features/feature-x:0.0.0-20180724195400-deadbeef)

[tonistiigi]

don't mix not-yet-approved/finalised syntaxes with finalised syntaxes in the same repository

This will turn out to be impossible to maintain like this if we let the sources diverge. If they are behind build tags we can test all the combinations and quickly see if a bugfix conflicts with some features. If a feature is not ready yet, it should be maintained and released from a fork.

how to deal with air gapped setups?

Seems separate issue and not really part of the naming logic

for the experimental features; (just a suggestion) I love the "pseudo version" format that's used by vGo

This is slightly different. We are not inventing an automatic version for a git sha. This is an actual release and with a specific configuration (specific tags enabled). Also, the user needs to type this in so there isn't a reason for making their (and people reading the Dockerfile) lives harder by adding things that do not have any extra meaning(and removing things that have meaning like the feature name).

also for experimental features; if (see first bullet) using a separate org/repo is an option, the name of the feature could be in the repository-name instead of in the tag

I think this is risky as you can pull and push to another repository. If the tag is preserved it should be unique enough. If there is a mirror mirror:1.2.3 you shouldn't have to think about if this 1.2.3 is stable or experimental.

[tonistiigi]

Update on the proposed release process https://docs.google.com/document/d/1wP2oPvXf3duwV92c0jClhyhuijyrC6TXuKC4PA1XjhY/edit?usp=sharing

[tonistiigi]

This has been completed

[gurpreetatwal]

I might be missing something fundamental, but how does one figure out what features a given tag supports?

I see that all the tags are published here and the suggestion in the docs is to "pin the image to a specific revision. ", but I'm not entirely sure what to pin it to. I only need access to ssh mount type feature and the tags here seem to suggest there is a syntax image that only has ssh.

I'm also just kinda of generally confused about what the path to stability looks like for these features, I tried searching for some documentation or release notes but could not find anything.

Apologies for just turning this into a support request, but I've tried searching for a few hours and wasn't able to find any documentation.

[tonistiigi]

Docs are in https://github.com/moby/buildkit/blob/dockerfile/1.1.3-experimental/frontend/dockerfile/docs/experimental.md . Change the tag for different versions.

"pin the image to a specific revision. ", but I'm not entirely sure what to pin it to

That recommendation is to use full versions, eg. 1.1.3-experimental instead of latest orexperimental

I only need access to ssh mount type feature and the tags here seem to suggest there is a syntax image that only has ssh.

Automatic builds from this repo go to https://hub.docker.com/r/docker/dockerfile-upstream/tags . docker/dockerfile only contains stable and experimental channel releases that have gone through some extra testing. I would recommend using experimental channel. Single-feature releases have not proved to be very useful and may be discontinued in the future.

I'm also just kinda of generally confused about what the path to stability looks like for these features, I tried searching for some documentation or release notes but could not find anything.

Experimental channel means that future versions do not follow semver. Already released versions still guarantee backward compatibility, even on experimental channel.