Make project generation explicit and generic

[squakez]

As we're discussing around Camel K 2.0 and the possibility to make it runtime-generic, and as we're discovering we may need full source at build time in order to build natively, here a possible proposal to enable such features.

Disclaimer: this is a very high level design, any development will require a further and low level assessment. I'd like to discuss the "what" without thinking too much in the "how" (at least at this stage).

At this stage the project generation is done just before the build of the project. But the build is generic as the original scope was to have a base maven project containing the set of dependencies (jars) required by the IntegrationKit to build the container image required at runtime. The source developed by the user was later "injected" into the Camel context by Camel K Runtime project. This approach is limiting us to enable anything else but Camel K Runtime applications. Moreover with this approach the pod build strategy requires a circular reference to the Camel K container image to be able to reuse an hidden kamel builder command.

It follows a diagram with my proposal:

I propose to separate the project generation and the build so that we can generate the project or even upload the project (ie, from a git repository or from a camel export). We can try to reuse (and eventually maintain) the same Camel JBang export in order to have the same output as the user had locally. We can introduce a reconciliation loop on a new Project CR: once the project is generated/uploaded (Project status Ready) we'd move to the Build phase that has to know the project (probably the location which may be in Project.Status).

The Build should be more or less how it is now, and the rest of the flow probably similar as well. Probably we'll need to start supporting the kind of project we support now (Camel K runtime/Camel Quarkus) and verifying the compatibility of the incremental image/reuse of previous IntegrationKits. Once this is smooth enough we can start supporting any further generic Camel application (main, SB) leveraging the same mechanism of IntegrationKits inheritance.

[squakez]

cc @davsclaus @oscerd @lburgazzoli @claudio4j @essobedo @gansheer @tadayosi

[essobedo]

Very interesting idea, here are my first questions/remarks

1. What would be the expected content of your git project, to be able to launch it with the kamel run command?
2. Don't you believe that allowing github URLs could be seen as an open door to security issues?
3. Does it mean that the dependency management is delegated to Camel JBang? This would mean, we won't have dependencies in IntegrationKit and Build, is that correct?
4. Having the project workspace in a local directory could be an issue in a cluster, indeed, if the master dies after generating the project you could have a problem later resuming the build from another node. Maybe it should rather be compressed and stored into a ConfigMap, shouldn't it?
5. Why have a dedicated CR for this purpose instead of eventually a new trait?

[squakez]

Very interesting idea, here are my first questions/remarks

1. What would be the expected content of your git project, to be able to launch it with the `kamel run` command?

We should agree on the expected format, for now I can think about the outcome of the camel export

2. Don't you believe that allowing github URLs could be seen as an open door to security issues?

Not really. What we need to analyze is how to secure the access to a Git repo (github was an example, it could be any enterprise source code repo).

3. Does it mean that the dependency management is delegated to Camel JBang? This would mean, we won't have dependencies in IntegrationKit and Build, is that correct?

We still need both IntegrationKit and Build. What I'd like to analyze is the feasibility of camel to produce a Maven project from a Camel route (that's what we're already doing in Camel K, but I'd like to see if we can delegate that part)

4. Having the project workspace in a local directory could be an issue in a cluster, indeed, if the master dies after generating the project you could have a problem later resuming the build from another node. Maybe it should rather be compressed and stored into a ConfigMap, shouldn't it?

This is already the way it works (we store everything in /tmp/): if the operator crash and it's rescheduled, it should restart a Project. However the possibility to use a stateful storage is something we should evaluate, because it intersect with the requirement to build with pod strategy (a bit of a discussion in #3831). At this stage I'd prefer not to go in low level analysis, it would be some implementation detail we should understand. I'd love to use something like a Configmap, but the problem is the size limitation we have as it can only store 1 MB data.

5. Why have a dedicated CR for this purpose instead of eventually a new trait?

I think it would simplify the generation and the error that can happen when a Project is created. Also it would simplify the kick off of a Build which should watch for a Project to be Ready.

[essobedo]

I'm getting confused, what would be the interest of keeping inferring the dependencies and storing them in the IK and Build if the maven project is generated by Camel JBang or retrieved from a git clone?

[squakez]

I'm getting confused, what would be the interest of keeping inferring the dependencies and storing them in the IK and Build if the maven project is generated by Camel JBang or retrieved from a git clone?

Kit reusability and incremental image. We don't want to rebuild a container image every time.

[essobedo]

In case, we provide a GitHub URL, I'm wondering what would we have in the Kit?

With this generic approach, since the main Camel logic is delegated to Camel JBang, we could even imagine having a Camel-K agnostic to the Java Application launched. The project could then be proposed in the incubator with a different name and Camel-K would then be an extension of this new project for Camel where we would have the project builder based on Camel JBang and a specific Camel mechanism that would extract the specific info to put into the kit for reusability and incremental image. Maybe I'm going too far but at some point, it could be something to consider.

[oscerd]

Since it's stuff coming from Camel it will eventually go under the Apache Camel umbrella, not passing through incubator.

[essobedo]

IMHO, If you keep it under the Camel umbrella, people will get confused and the project will hardly get the deserved success. So far everything under the Camel umbrella is Camel related.

[oscerd]

I think you are going a bit too far.

[essobedo]

If we wanted to have Camel-K agnostic to the Java Application launched, would you mind listing what we would need to add?
Just to allow me to understand how far we are from this potential objective?

[claudio4j]

I am all in for this proposal to have the standard maven project with source code instead of injecting it, it sounds more natural and allows flexibility in the future to tweak the build tool (mvnd?)
I don't know the original motivation to have camel-k-runtime to inject the source code (perhaps, to not depend on the pod operator file system).
With this move, there will be less code to maintain in camel-k-runtime.

[lburgazzoli]

I am all in for this proposal to have the standard maven project with source code instead of injecting it, sound more natural and allows flexibility in the future to tweak the build tool (mvnd?) I don't know the original motivation to have camel-k-runtime to inject the source code (perhaps, to not depend on the pod operator file system). With this move, there will be less code to maintain in camel-k-runtime

I really don't read this proposal this way, and if it were then it would break any major camel-k use case as you would need to trigger a build every time a kamelet binding/integration changes. We had the option to include the sources back in the day and I'm all for supporting this, but it must be an opt in feature (note that including a yaml/xml/js/groovy/kotlin route is not different from attaching it at runtime and such functionality is implemented by camel core, the camel-k-runtime adds supports for advanced use cases required such has knative, cron, master)

[lburgazzoli]

@squakez when talking about git, what does that mean exactly ? In my mind, cloning a project from git could and should be supported, pushing to git as a "storage" is very hard to be done right and I don't see the real advantage vs doing an ephemeral project generation in a pipeline.

[squakez]

If we wanted to have Camel-K agnostic to the Java Application launched, would you mind listing what we would need to add? Just to allow me to understand how far we are from this potential objective?

Although we may think on something very generic, my idea is to stick focusing on Camel project aspects exclusively. There are many operations beside the deployment that still need to be very Camel specific.

[squakez]

I am all in for this proposal to have the standard maven project with source code instead of injecting it, sound more natural and allows flexibility in the future to tweak the build tool (mvnd?) I don't know the original motivation to have camel-k-runtime to inject the source code (perhaps, to not depend on the pod operator file system). With this move, there will be less code to maintain in camel-k-runtime

I really don't read this proposal this way, and if it were then it would break any major camel-k use case as you would need to trigger a build every time a kamelet binding/integration changes. We had the option to include the sources back in the day and I'm all for supporting this, but it must be an opt in feature (note that including a yaml/xml/js/groovy/kotlin route is not different from attaching it at runtime and such functionality is implemented by camel core, the camel-k-runtime adds supports for advanced use cases required such has knative, cron, master)

Exactly, this would be an optional feature that could simplify onboarding other Camel runtimes. Also, mind that we need to evaluate the feasibility with the actual design in place.

[squakez]

@squakez when talking about git, what does that mean exactly ? In my mind, cloning a project from git could and should be supported, pushing to git as a "storage" is very hard to be done right and I don't see the real advantage vs doing an ephemeral project generation in a pipeline.

Just pulling.

[essobedo]

Back to the GitHub URL feature, don't you believe that losing control of what is built, could allow the execution of malicious code at build time that could, for example, use the K8s API to collect sensitive data especially if the build is made from the operator's pod?

[squakez]

Back to the GitHub URL feature, don't you believe that losing control of what is built, could allow the execution of malicious code at build time that could, for example, use the K8s API to collect sensitive data especially if the build is made from the operator's pod?

I'm not particularly worried, consider any pipeline tool (ie Tekton), they have tasks to pull from git and build accordingly. At some point we may even think to integrate them from any given pipeline technology.

[essobedo]

I'm not particularly worried, consider any pipeline tool (ie Tekton), they have tasks to pull from git and build accordingly. At some point we may even think to integrate them from any given pipeline technology.

Yes, but in the case of a pipeline, you can isolate the runner and limit its access rights. Here even if all the builds are delegated to a specific pod, the pod still needs to be able to update the status of the build, doesn't it?

[squakez]

Parking this for the moment as it may be impacted by the development proposed in #4281

[github-actions]

This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!