[EPIC] Support for <buildx>

[neilstevenson]

Component

Kubernetes Maven Plugin

Is your enhancement related to a problem? Please describe

When building on a non-"x86" machine (eg. a Mac with an M1 chip), I cannot see how to build images for "x86".

Describe the solution you'd like

Fabric8 supports <buildx> though it's not mentioned in the Fabric8 documentation for 0.40.2.

You can have this:

<build>
    <buildx>
        <platforms>
            <platform>linux/arm64</platform>
            <platform>linux/x86_64</platform>
        </platforms>
    </buildx>

Describe alternatives you've considered

No response

Additional context

See example - https://github.com/neilstevenson/fabric8-jkube-migration

tasks

• doing investigation how it is possible

[rohanKanojia]

@neilstevenson : Are you docker user or Kubernetes user?

[neilstevenson]

Docker, just building Docker images.

[rohanKanojia]

What is your goal behind migrating to jkube? JKube offers functionality related to Kubernetes. It does involve Docker Maven Plugin's build goal, but rest of the goals are specific to Kubernetes.

[neilstevenson]

Fabric8 is deprecated and advises migrating to JKube, see https://maven.fabric8.io/

[rohanKanojia]

The website you're pointing to belongs to Fabric8 Maven Plugin, not Docker Maven Plugin. Docker Maven Plugin documentation resides in https://dmp.fabric8.io/ .

I understand there is a lot of confusion regarding Fabric8 deprecation. I've also given one StackOverflow answer regarding this: https://stackoverflow.com/questions/73705267/is-fabric8-for-kubernetes-client-deprecated

JKube is replacement for Fabric8 Maven Plugin (io.fabric8:fabric8-maven-plugin) not Docker Maven Plugin (io.fabric8:docker-maven-plugin).

Docker Maven Plugin is still an active project. However, you can use Eclipse JKube if you're only concerned with Building Image part.

[neilstevenson]

Hi, just to clarify, fabric8-maven-plugin allows buildx and recommends migrating to kubernetes-maven-plugin which doesn't allow buildx.
If kubernetes-maven-plugin had this feature it'd make the recommended migration possible for poms that use buildx. I tagged this as an enhancement.

[neilstevenson]

Above is wrong... you're right about confusion. So try again..

Fabric8's docker-maven-plugin allows buildx . It's fabric8-maven-plugin which is deprecated.

But if kubernetes-maven-plugin can build Docker images, it would be a useful enhancement if buildx was supported.

[sunix]

TODO:

• investigate if it is feasible with the rest api
• clarify what's the difference with --platform

[sunix]

related to #1157

[gansheer]

Hello @sunix !

We are also hoping for this feature on the Apache Camel-k project. I made some comparative analysis of the plugin able to build container image the jib way and really like the JKube kubernetes plugin as we could easily extend its usage for other features. I had to choose to the google jib maven plugin instead because the Jkube Kubernetes plugin was missing this feature to allow us to be able to build images for multiple architecture.

[rohanKanojia]

@gansheer : What is your use case? Do you just want to build container images?

[rohanKanojia]

@gansheer :As far as I remember buildx is not supported by Docker REST API. In Fabric8 Docker Maven Plugin, it's done via executing docker CLI

[gansheer]

In short, we have an operator container that is (between other features) in charge of building the container images the operator is later in charge of running, which means for most users building without a docker daemon. I am not talking about the operator container image, but containers builded during the operator's run time.

For now we are using as image building tools Spectrum, S2I, Kaniko and Buildah. We are introducing the JIB way to become the main building strategy and replace the Spectrum (and hopefully the Kaniko and Buildah one).
On our builds, we need to be able have some flexibility on layers declaration and want to introduce multi-architecture image as we are moving toward a more flexible model in where the image container will be executed (and that is something that has been in demand by the community).

I made some tests with the Fabric8 Docker Maven Plugin, and have missed the docker CLI dependent part, so thank you for the info.

The goal is to be able to build the container image using JIB for different architectures (linux/am64, linux/arm64, etc). Should I open another issue if is not really the solution for that problem ?

[rohanKanojia]

OpenShift Maven Plugin can also operate in JIB mode using property -Djkube.build.strategy=jib.

Would it work for your use case if we check to enable multiple architectures builds in JIB mode (This would require passing some additional configuration flags to Jib programmatically that we're doing in JibBuildService)?

[gansheer]

I checked the Fabric8 docker maven plugin again, and you where right. The flag containing the platform configurations is ignored when using the jib strategy (ans stopping the local docker daemon just in case).

What you propose would work for my use case, thank you. As best case scenario, I am expecting to add some configuration in my plugin configuration like this one (or anything that fits more your flag rules really).

Is your JIB mode code common to the kubernetes and openshift ? Having it available in both would be a nice thing.

[rohanKanojia]

I'm able to see setPlatform method being available in Jib.from method call here:
https://github.com/eclipse/jkube/blob/e573db610e21284a52782e84faeaaf68fe1c4eb7/jkube-kit/build/service/jib/src/main/java/org/eclipse/jkube/kit/service/jib/JibServiceUtil.java#L120

I think if we pass in the platform here, we might be able to build+push multi-architecture images.

Could you please create an issue about this (Support for multiple platforms in jib build strategy)?

[rohanKanojia]

As per discussion in fabric8io/docker-maven-plugin#1711 (comment)

In order to use BuildKit , we need to pass version=2 query parameter to Docker API. See Docker Rest API Docs.

Currently JKube is using classic Docker builder that has been deprecated

[manusa]

In order to use BuildKit...

Does this mean that multi-platform can be enabled for docker leveraging the rest API?

Related links

• https://github.com/moby/buildkit

[rohanKanojia]

Currently JKube is using older Docker builder that doesn't support multi arch images.

I haven't tested it myself but switching to buildkit should allow building multi platform images by passing platform query parameter.

[nightswimmings]

I'm pretty sure I used to build linux images on my mac by means of

<image>
  <buildOptions>
      <platform>linux/amd64</platform>
  </buildOptions>
 <image> 

but for some reason it is failing currently due to the docker manifest step not using the authentication as in the docker pull emulation which smartly uses contextual login. Not sure if AWS ECR changed something, it got lost in last versions of the jkube plugin or either my docker desktop last update broke something

[rohanKanojia]

@nightswimmings : Does this failure happen only for AWS ECR authentication or for some other registry as well?

[rohanKanojia]

A contributor had started work in Docker Maven Plugin to enable BuildKit by default by passing Builder-Version header but it's causing some failures related to docker compose due to which effort is on hold for now.

fabric8io/docker-maven-plugin#1794 (comment)

• enabling buildkit actually works, but builds may fail in some cases
• The reason for this is, that buildkit expects to have some sort of "callback api" to the client. To provide this callback, the client has to initiate a gRPC session (see: Initialize interactive session API) + the client has to implement several gRPC services.
• I couldn't find any documentation about these services, but the code for these seem to here here
• Creating (and maintaining) a Java implementation for these gRPC services is probably a monstrous task. I am afraid to say that it's probably easier to stop using the Docker Engine API at all and use the CLI (docker build(x)) instead. At least for now i have no clue how to deal with that.

In JKube's case, we only want to build image so this might not cause problems. But we would need to ensure that we don't break compatibility with older docker API versions.