Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-architecture support #1238

Closed
5 tasks
astefanutti opened this issue Jan 30, 2020 · 51 comments
Closed
5 tasks

Multi-architecture support #1238

astefanutti opened this issue Jan 30, 2020 · 51 comments
Labels
area/continuous integration Related to CI and automated testing status/never-stale
Milestone

Comments

@astefanutti
Copy link
Member

astefanutti commented Jan 30, 2020

At the moment, Camel K binaries and images are available for amd64 architecture only.

Following up on #1215, it would be useful to publish multi-architecture images, so that Camel K can be used on a wider range of architectures. Based on a recent PoC, I've identify the following tasks:

@github-actions
Copy link
Contributor

This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!

@robertonav20
Copy link
Contributor

Any updates for this task? Can I help with something?

@squakez
Copy link
Contributor

squakez commented May 3, 2022

@robertonav20 you may have a look at the tasks in the issue description if you're willing to have a look at the implementation. A draft PR with some PoC may be useful to experiment and reason about the pros and cons.

@robertonav20
Copy link
Contributor

Hi @squakez,
I build and execute camel k operator and a camel quarkus application inside raspberry.
I used docker buildx and specify the target platform.

I must modify makefile and dockerfile for camel-k-operator and use the correct java base image for camel quarkus application.

I apologize but I haven't strong skill with go language, I try to do my best. Can you give me an advice?

@squakez
Copy link
Contributor

squakez commented May 5, 2022

I think you can start having a look at the builder package which is the one in charge to take care for building images and pushing to registry.

@robertonav20
Copy link
Contributor

robertonav20 commented May 7, 2022

Hi @squakez,
i've tried to import BuildKit library like suggest @astefanutti inside the builder package but there is a problem of dependency

go: finding module for package github.com/go-logr/logr/funcr
github.com/apache/camel-k/pkg/builder imports
        github.com/moby/buildkit/client imports
        go.opentelemetry.io/otel/sdk/trace tested by
        go.opentelemetry.io/otel/sdk/trace.test imports
        github.com/go-logr/logr/funcr: module github.com/go-logr/logr@latest found (v1.2.3, replaced by github.com/go-logr/[email protected]), but does not contain package github.com/go-logr/logr/funcr

Now I'm trying to use buildah library instead buildKit, it's more friendly than buildkit, but it's the same I got error during build

# github.com/docker/docker/pkg/archive
/home/rob/go/pkg/mod/github.com/moby/[email protected]/pkg/archive/archive_unix.go:84:13: undefined: system.RunningInUserNS

you can find the my code here https://github.com/robertonav20/camel-k/... i will update you if i fix the problem

@robertonav20
Copy link
Contributor

robertonav20 commented May 16, 2022

Hi,
i have an update with this tasks, below the description for each point>

  • Use a multi-architecture default Java base image

I use the graalvm base image in order to use for amd64/arm64 like this
FROM --platform=$BUILDPLATFORM ghcr.io/graalvm/graalvm-ce:ol8-java11-22.1.0

I thinked to use quarkus-mandrel-base-image for arm64 but isn't available for now (quarkusio/quarkus-images@21e25a3)

  • Build and publish multi-architecture images: BuildKit could be a good candidate to ease the task of building images for all target architectures, as it automatically picks the right base image and creates the multi-architecture manifest

I modified the Makefile to use BuildKit with correct platform parameter in base of host's architecture

docker buildx rm --all-inactive --force
docker buildx create --append --name builder
docker buildx build --platform=$(IMAGE_TARGET_PLATFORM) -t $(CUSTOM_IMAGE):$(CUSTOM_VERSION) -f build/Dockerfile .

So, for the first part can i submit pr?

For the integration-kit image, I cannot find a solution to build and push a working image because there are a problem dependency (read the previous comment).
Do you have an idea how to fix?

@squakez
Copy link
Contributor

squakez commented May 18, 2022

Thanks for running those experiments and reporting the feedback. About the multi-architecture Java image, I guess we may wait for that to be officially available. We will align to Quarkus platform, so, that one will become eventually available.
As for the operator container, I think we cannot proceed so easily to change the make images. We depend on that for the release process and likely the docker buildx is not available in the github actions we use for the official release.

However, you can create a new action beside images, ie, images-arch which can be run manually by anyone who wants to run that. We may eventually use that once we have the rest of pieces in place. @oscerd @astefanutti @phantomjinx wdyt?

@oscerd
Copy link
Contributor

oscerd commented May 18, 2022

It makes sense to me. +1.

@robertonav20
Copy link
Contributor

robertonav20 commented May 18, 2022

So, @squakez suggest a command makes-arch to build for other platform, we can make the same thing with dockerfile because the base image isn't available. It makes sense if that image doesn't available soon, are you right?
I don't know if can be a good idea, maybe there are other constraints for quarkus.

I checked the GitHub action, here https://github.com/marketplace/actions/build-and-push-docker-images can be found the official docker GitHub action of docker which support buildx, so I think it's possible use it.

Anyway, It's a pleasure help the community.... For now I try to build an integration kit multi arch with buildah strategy.

It's possible just update buildah version to 1.23.3 because the latest version support arm64 arch... Stay tuned I will give some feedback soon

@robertonav20
Copy link
Contributor

robertonav20 commented May 20, 2022

As i say previously, i build and execute an integration kit inside rpi4 8gb with architecture arm64 with the use of buildah, below the modifies:

  • Update buildah at 1.23.3 version
  • Add the parameter platform
bud := []string{
        "buildah",
        "bud",
        "--storage-driver=vfs",
        "--platform",
        platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant,
        "--pull-always",
        "-f",
        "Dockerfile",
        "-t",
        task.Image,
        ".",
}

All image are available here https://hub.docker.com/repository/docker/robnav24241/camel-k operator and integration-kit

image

image

@squakez
Copy link
Contributor

squakez commented May 23, 2022

Nice work. Ideally those architecture values should be "parameter-ized" so that the user can provide that information somewhere (ie, in the IntegrationPlatform configuration).

@robertonav20
Copy link
Contributor

@squakez sorry, I don't understand why should be parameterized?
The integration kit are builded inside kubernetes so with architecture of the node amd64, arm64 etc.... this thing can't be modified by the user.

Can you describe me?

@squakez
Copy link
Contributor

squakez commented May 23, 2022

Yeah, I am thinking on a generic situation where you have multiple nodes in your cluster, and the operator is running on any non-ARM architecture. Then you have a special node in ARM architecture where you are willing to run the Integrations. In such case, having that value as a parameter would help.

@robertonav20
Copy link
Contributor

robertonav20 commented May 23, 2022

Ok Great! Thanks!
But you are limited by buildah architecture support, for others solutions it's necessary fix the dependency problem which i describe some comment ago

Anyway, i saw the IntegrationPlatform struct is based entirely by external attributes from [email protected], so i suggest add the Architeture field inside builderTrait and PublishTask like this

// The builder trait is internally used to determine the best strategy to
// build and configure IntegrationKits.
//
// +camel-k:trait=builder.
type builderTrait struct {
	BaseTrait `property:",squash"`
	// the architecture of image
	Architecture string `json:"baseImage,omitempty"`
	// Enable verbose logging on build components that support it (e.g. Kaniko build pod).
	Verbose *bool `property:"verbose" json:"verbose,omitempty"`
	// A list of properties to be provided to the build task
	Properties []string `property:"properties" json:"properties,omitempty"`
}

// PublishTask image publish configuration
type PublishTask struct {
	// can be useful to share info with other tasks
	ContextDir string `json:"contextDir,omitempty"`
	// Architecture of image
	Architecture string `json:"baseImage,omitempty"`
	// base image layer
	BaseImage string `json:"baseImage,omitempty"`
	// final image name
	Image string `json:"image,omitempty"`
	// where to publish the final image
	Registry RegistrySpec `json:"registry,omitempty"`
}

Then can be map Architecture when necessary just like this

	case v1.IntegrationPlatformBuildPublishStrategyBuildah:
		if t.Architecture == "" {
			t.Architecture = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
		}

		e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
			BaseTask: v1.BaseTask{
				Name: "buildah",
			},
			PublishTask: v1.PublishTask{
				Architecture: t.Architecture,
				Image:        getImageName(e),
				Registry:     e.Platform.Status.Build.Registry,
			},
			Verbose: t.Verbose,
		}})

PublishTask struct is shared by all builders for this reason i choosed her

@squakez @oscerd @astefanutti @phantomjinx do you agree?

@squakez
Copy link
Contributor

squakez commented May 24, 2022

You can have a look at some refactoring we've done recently here #3032 - We've introduced a generic PublishStrategyOptions in the IntegrationPlatform that can be used for any strategy extra option (we're using for Kaniko there). I think we could leverage that and from that onward cascade the parameter up to the Buildah Task. Ideally we don't want a specific parameter to slip into an abstract concept (such as the Builder). Also, feel free to create any draft PR and we can discuss over there any idea you have around this or other developments.

@robertonav20
Copy link
Contributor

robertonav20 commented May 24, 2022

Ok @squakez, i understand your advice but i dont know how to use PublishStrategyOptions.

Can you give me and example to fill that structure from cmd?

@squakez
Copy link
Contributor

squakez commented May 25, 2022

Sure, you can have a look at #3032 where we used that for the "Kaniko" options. We may mimick that and include any "Buildah" option as well.

@robertonav20
Copy link
Contributor

robertonav20 commented May 25, 2022

Hi @squakez, yes i have a look yesterday, below i describe the code:

Inside the package builder i add buildah.go file to manage buildah constants properties

package builder

const BuildahPlatform = "BuildahPlatform"

Here you can find the mapping to buildah builder

	case v1.IntegrationPlatformBuildPublishStrategyBuildah:
		var architecture string
		var found bool
		if architecture, found = e.Platform.Status.Build.PublishStrategyOptions[builder.BuildahPlatform]; !found {
			architecture = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
		}

		e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
			BaseTask: v1.BaseTask{
				Name: "buildah",
			},
			PublishTask: v1.PublishTask{
				Architecture: architecture,
				Image:        getImageName(e),
				Registry:     e.Platform.Status.Build.Registry,
			},
			Verbose: t.Verbose,
		}})

Then map architecture attribute to buildah command.

I see a difference between #3032 and this task, that task introduce a command parameter for install, but in our case we must manage the architecture of build for integration kit, so it's necessary introduce this parameter inside run command

My questions are:

  • Where is the point of the code ? I suppose run.go inside pkg/cmd and here i can add the new buildah parameter (exmaple: buildah-platform)
  • The variable platform inside run.go method createOrUpdateIntegration is empty, How to fill it ?
  • How to check if that map PublishStrategyOptions contains that parameter?

I'm hope you understand my questions.

Thanks in advance

@squakez
Copy link
Contributor

squakez commented May 26, 2022

Any build configuration should be included into the IntegrationPlatform, not exposed in the kamel run command. I think you can extend the BuildahTask to include the new needed parameters

type BuildahTask struct {

and passing them from IntegrationPlatform.PublishStrategyOptions as we're doing for Kaniko settings. Once they are in the BuildahTask, you should be able to use them.

@robertonav20
Copy link
Contributor

robertonav20 commented May 26, 2022

Ok thanks, i understand your advice and i fix the code just like this.

type BuildahTask struct {
	BaseTask    `json:",inline"`
	PublishTask `json:",inline"`
	// The platform of build image
	Platform string `json:"baseImage,omitempty"`
	// log more information
	Verbose *bool `json:"verbose,omitempty"`
}

case v1.IntegrationPlatformBuildPublishStrategyBuildah:
	var platform string
	var found bool
	if platform, found = e.Platform.Status.Build.PublishStrategyOptions[builder.BuildahPlatform]; !found {
		platform = platforms.DefaultSpec().OS + "/" + platforms.DefaultSpec().Architecture + "/" + platforms.DefaultSpec().Variant
	}

	e.BuildTasks = append(e.BuildTasks, v1.Task{Buildah: &v1.BuildahTask{
		Platform: platform,
		BaseTask: v1.BaseTask{
			Name: "buildah",
		},
		PublishTask: v1.PublishTask{
			Image:    getImageName(e),
			Registry: e.Platform.Status.Build.Registry,
		},
		Verbose: t.Verbose,
	}})

In this way you can manage the platform parameter if you want, just fill PublishStrategyOptions with the correct paramater from outside.

But, i don't understand how to works PublishStrategyOptions, for kaniko i found this code inside install.go

kanikoBuildCacheFlag := cobraCmd.Flags().Lookup("kaniko-build-cache")
if kanikoBuildCacheFlag.Changed {
	platform.Spec.Build.PublishStrategyOptions[builder.KanikoBuildCacheEnabled] = strconv.FormatBool(o.KanikoBuildCache)
}

If i undertstand well, with install command you can pass kaniko-build-cache parameter and to set the PublishStrategyOptions, right?
There is another way to fill PublishStrategyOptions?
How do you thing to manage the platform parameter for buildah?

Thanks in advance! 😄

@squakez
Copy link
Contributor

squakez commented May 27, 2022

Nice, that fits perfectly into the design. I think those parameters must be provided by the user editing the IntegrationPlatform (manually) ie kubectl edit ip. A nice addition would be to transform that kaniko-build-cache into something generic, so we can manage any PublishStrategyOptions option. I'm creating a follow up issue for that. For now, I guess that the manual editing of the IntegrationPlatform would be enough for the scope.

@robertonav20
Copy link
Contributor

Hi @barius to build integration kits arm64 it's necessary use buildah as build-strategy you can follow the pr open from this ticket

@barius
Copy link

barius commented Jun 12, 2022

@robertonav20
Okay I'll try it! Good work!

@barius
Copy link

barius commented Jun 12, 2022

Confirm that the PR works!
Now my integrations are arm64, yeah! Saw ~4x performance on my mac :)
I'll still using my local arm64 quarkus-mandrel image though, didn't find the manual command to build kamel operator to support arm architecture :(

@robertonav20
Copy link
Contributor

@barius
Copy link

barius commented Jun 13, 2022

Ah I see, great! However it uses Graal. Don't know what the difference between mandrel and graal so I'm waiting for the official release :) For now everything works for me and I don't have much time testing it out :P

@squakez
Copy link
Contributor

squakez commented Jun 20, 2022

Partially available in #3309

@robertonav20
Copy link
Contributor

Can we close this issue?

@squakez
Copy link
Contributor

squakez commented Jun 22, 2022

No, I think that although partially available with your development, this issue need some more work to be fully completed.

@barius
Copy link

barius commented Jun 25, 2022

Although Buildah is perfectly fine with regards to building kit images, when Istio auto-injection is enabled, the camel-k builder always wait for the Istio container to complete (which will never happen), and triggers the default 5 min timeout (though Buildah finished in around 1~2 min and after the timeout the build still succeeds).

Don't know if this is the expected behavior, or if I should open another issue with it.

@squakez
Copy link
Contributor

squakez commented Jun 27, 2022

@barius please better open a new issue and which version is affected (I suppose the nightly build).

@barius
Copy link

barius commented Jun 30, 2022

@squakez yes pls take a look #3408

@robertonav20
Copy link
Contributor

@squakez now the is available the multi image of ubi-quarkus-mandrel and ubi-quarkus-micro-image, so you can update the dockerfile to support arm64 natively then delete the new file Dockerfile.arch.

Here the quay.io repository
podman pull quay.io/quarkus/quarkus-micro-image:2.0
podman pull quay.io/quarkus/ubi-quarkus-mandrel:22.0.0.2-Final-java11-arm64

Ref. to the closed issue quarkusio/quarkus-images#83

@squakez
Copy link
Contributor

squakez commented Oct 24, 2022

Cool! Thanks for the heads up. Feel free to have a look yourself while this is not picked up for development.

@tobiasoort
Copy link

tobiasoort commented Nov 15, 2022

Edit: It seems I was being captain obvious. Is there anything I can help with?

For future googlers, the following error in your pod's logs:
standard_init_linux.go:228: exec user process caused: exec format error
means you're running the wrong arch (in this specific case, running an amd64 image on arm64).

@squakez
Copy link
Contributor

squakez commented Nov 16, 2022

Edit: It seems I was being captain obvious. Is there anything I can help with?

For future googlers, the following error in your pod's logs: standard_init_linux.go:228: exec user process caused: exec format error means you're running the wrong arch (in this specific case, running an amd64 image on arm64).

I guess the last message from @robertonav20 could be a good point from where to continue with the development of this issue.

@tobiasoort
Copy link

tobiasoort commented Nov 16, 2022

So, based on the assumptions:

  • ubi-quarkus-mandrel v22.0.0.2-Final-java11 is the version to target (v21 doesnt have arm builds) ✅
  • hosting the application in the new quarkus container is enough - the Maven build doesnt need to change (yay jvm?) ✅
  • I know how buildx works ✅

I have submitted #3820

@tobiasoort
Copy link

  • I know how buildx works

Foreshadowing quote was exactly that. I'm afraid i'll break the build because with multi-arch images we can't rely on the local image store (see docker/buildx#59). I've worked around it and hope that none of the processes rely on 'the image just being there'.

Also, I see a 'docker build' step for a 'bundle'. What is that 'bundle', and do I need to fix that too?

@tobiasoort
Copy link

tobiasoort commented Nov 17, 2022

Edit: this was wrong.

@tobiasoort
Copy link

tobiasoort commented Nov 17, 2022

So, the issue i'm currently blocked by is:

  • The IMAGE_NAME is set in the secrets.E2E_CLUSTER_CONFIG secret to kind-registry:5000/apache/camel-k (I think, its a secret!)
  • The IMAGE_NAME contains the hostname kind-registry or that name comes from somewhere
  • This hostname is not resolving at all (it is supposed to be 127.0.0.1) and thus it errors out.

How can we validate this? Can someone share the outline of the secrets.E2E_CLUSTER_CONFIG so I can override it in my own fork?
Edit: it was networking related. Buildx lives inside of a container and to resolve the kind-registry we need hostnetworking. Added a small change, lets see if the pipeline-deities are now happy.

@tobiasoort
Copy link

tobiasoort commented Nov 17, 2022

Integration tests are failing due to:

 Error: p.go:274:     > {"level":"error","ts":1668698213.7254713,"logger":"camel-k.maven.build","msg":"\t[error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.IllegalStateException: Out of date version of GraalVM detected: native-image 22.0.0.2-Final Mandrel Distribution (Java Version 11.0.14+9). Quarkus currently supports 22.3.0. Please upgrade GraalVM to this version."

I don't really understand this - I cant find a version reference of GraalVM in the source.

@squakez
Copy link
Contributor

squakez commented Nov 17, 2022

g":"\t[error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.IllegalStateException: Out of date version of GraalVM detected: native-image 22.0.0.2-Final Mandrel Distribution (Java Version 11.0.14+9). Quarkus currently supports 22.3.0. Please upgrade GraalVM to this version."

Known issue #3817

@tobiasoort
Copy link

Current status: ready for polishing. All pipeline errors are due to #3817 or timeouts that are also already known. Next up: parameterize the build archs.

@tobiasoort
Copy link

tobiasoort commented Nov 24, 2022

Current status: retracted the PR because of the amount of noise. The main branch is broken at the moment so i'll wait for things to green up again and then i'll continue getting this thing done.

@squakez
Copy link
Contributor

squakez commented Feb 2, 2023

As we're planning to rework the way we build the images, I think we should take the opportunity to include the multi-architecture support as well.

@squakez squakez mentioned this issue Feb 2, 2023
@squakez
Copy link
Contributor

squakez commented Mar 21, 2023

I am closing this issue in favor of #4148 and #4149 where we can track separately the evolution of Operator images and Integration images.

@squakez squakez closed this as not planned Won't fix, can't repro, duplicate, stale Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/continuous integration Related to CI and automated testing status/never-stale
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants