feat: transparent buildx support for remote buildkit (#6732)

[reingart]

Fixes: #6732 #9197
Related: #8172 (inspired by by a prior ebekebe fork, with a more general approach)

Description

docker buildx is an enhanced builder using BuildKit that can replace traditional docker build command, with almost the same syntax (transparently).

This adds several distributed features and advanced capabilities:

• remote build-kit builders, either running standalone, docker or Kubernetes clusters
• improved cache support (registry destination, pushing full metadatada and layers)
• multi-arch support (x86_64 and arm64, emulated if no builder for that platform exist)

This features are very useful for corporate CI/CD, for example when using GitLab, where the use case requires:

• using a ephemeral remote buildkit instance (rootless)
• using a remote docker registry for cache
• using a different cache destination tag

The remote BuildKit support is useful in cases with a privileged docker daemon is not possible or desirable due security policies (eg. untrusted images or review pipelines).
Daemon-less mode is also useful to offload container building from developers notebooks, sharing and reusing remote caches more effectively.

The cache-tag is useful to point to latest or a generic cache tag, instead of the generated one for this build (via tagPolicy that would be useless in most distributed cases, as it can be invalidated by minor changes, specially if using inputDigest).
Having it as an option could allow different branches to have different cache destinations (production vs development cache)

User facing changes

To use BuildKit transparently, BuildX should be installed as default builder (this creates an alias in docker config) and create a remote instance:

docker buildx install
docker buildx create --name remote --driver remote tcp://buildkitd:2375

Then Skaffold should be configured to detect buildx and set a default cache tag:

skaffold config set -g detect-buildx true
skaffold config set -g cache-tag cache

Optionally, in skaffold.yaml the cache source and destination can be configured with cacheFrom and cacheTo with the image name:

build:
  artifacts:
  - image: my-app-image
    context: my-app-image
    docker:
      dockerfile: Dockerfile
      cacheFrom:
      - "my-app-image"
  tagPolicy:
    inputDigest: {}
  local:
    useBuildkit: true
    useDockerCLI: true
    tryImportMissing: true

• If no tag is specified for cache, the configured cache-tag will be used (in this example my-app-image:cache)
• If cache to destination is not specified, the cache-from image name and tag will be adjusted, adding type=registry,mode=max (only if push images is enabled)

This cache behavior is intended to provide transparent user experience, with similar functionality compared to traditional local docker builds, without additional boilerplate or different configuration / patches for a CI workflow.

Finally, the user can run skaffold dev or skaffold build as usual.

Follow-up Work

Future work: #2110 to include more BuildKit advanced features support like multiple build contexts, for an improved mono-repo experience for large code-bases.

TODO:

• Design document
• Unit Tests
• Integration tests

Changes summary:

• adds a new global config detect-buildx to enable buildx supports
• implements logic to detect buildx is the default builder (via docker config alias)
• removes dependency on docker daemon if using BuildX (still supported to load images if using minikube or similar)
• adds a new global config cache-tag to override default cache tagging (instead of generated tag)
• adds a cacheTo to sakffold schema for the cache destination (optional, default is to use new cacheFrom + cache tag)
• adds --cache-to support in Docker CLI build if using BuildX CLI

All changes are backward compatible, so if the developer is not using buildx, traditional docker builder will be run.

Signed-off-by: [email protected]

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[code-review-assist]

Hi team,

I'm currently reviewing this pull request and will provide feedback in a few minutes. In the meantime, here's a summary of the changes for you and other reviewers to quickly get up to speed:

This pull request, feat: transparent buildx support for remote buildkit (#6732), aims to improve Skaffold's build process by adding transparent support for docker buildx. The primary goal is to enable users to leverage the advanced features of buildx, such as remote buildkit builders and improved caching, without significant changes to their existing Skaffold configurations.

Here's a breakdown of the changes:

• New Global Configs:
  • detect-buildx: Enables automatic detection of buildx as the default builder.
  • cache-tag: Allows overriding the default cache tagging strategy, useful for managing caches across different branches or environments.
• Build Process Enhancements:
  • The build process now intelligently detects and utilizes buildx if available and configured.
  • Reduced dependency on the local Docker daemon when using buildx, enhancing security and flexibility.
  • Added cacheTo support in the Skaffold schema to specify custom cache destinations.
  • Added --cache-to support to the Docker CLI build process when using buildx.
• Backward Compatibility: All changes are designed to be backward compatible. If buildx is not detected or used, Skaffold will fall back to the traditional Docker builder.

Key File Changes:
The majority of changes are focused within the pkg/skaffold/build directory, specifically affecting the cache and Docker build logic. Significant modifications were made to cache.go, lookup.go, and docker.go to handle buildx integration and caching strategies. The schema (config.go) was also updated to include the new cacheTo field and detect-buildx option.

Note: The pull request mentions follow-up work to support additional advanced buildkit features in the future. Unit and integration tests are also listed as TODO items.

I hope this summary is helpful. I'll post my detailed review shortly.

---

And now, a little haiku to lighten the mood:

BuildKit's swift flight,
Remote caches, smooth and fast,
Code flows like a stream.

[code-review-assist]

This PR introduces transparent buildx support, which is a valuable addition. The implementation looks good overall, and addresses the issues raised in #6732 and #9197. I have a few suggestions for improvement.