[Discussion] Camel K 2023 roadmap

[squakez]

As 2023 has started, I have collected some ideas on what we can do during this year based on what is left from 2022 roadmap and on the requirements that are emerging lately from the community. We will possibly end up having a Camel K 2 (meaning we can consider introducing breaking changes if the design requires to do so).

Items

Simplify release process

• Remove Kamelets from Camel K release cycle cadence
• Move Camel K runtime into Camel Quarkus
• Keep working on CI automation via Github actions

We already started in 2022 to move in the direction to make a Camel K release simpler. Kamelets have become a core part of Camel, so, it does not make sense any longer to depends on Camel K. Camel K runtime has also become a very thin layer which may be beneficial to be in Camel Quarkus as a provider and let Camel K directly use it from there. In theory we could benefit from this decoupling having Camel K being able to run any runtime (more to follow on the "build" section).
We have done a lot of work on automation and now it's paying off with a certain level of stability in the build and release processes. I suggest to keep maintaining high the attention and automate any manual developer task.

Build

• Remove dependency from Quarkus required tooling (Mandrel image)
• Implement JIB and deprecate Spectrum, Kaniko, Buildah and S2I (only if JIB is good enough with Openshift)
• Evaluate how to plug any external pipeline tool (ie, Tekton) for building
• Enable GitOps

In general the idea is that we should have a more "enterprise" building system.
There are several improvements we should be doing in this area. The first one intersect directly with the release process and the fact that now, at build time, we require the same tooling needed at runtime. We should work to remove such a dependency and make the operator not to inherit from the build tooling (more details in #3831).
JIB (#1656) looks as a good alternative on what we're doing now with different building systems. We're not able to maintain and keep every build strategy up to date, so, I advocate to deprecate/remove everything else and focus our effort around JIB.
We should also analyze how we can have external building tools being delegated with the build task (for instance a Tekton pipeline may take care of providing a build of a project).
About GitOps we should explore possibilities how to enable that and have it available in the operator workflow.

Support for Camel 4

Probably this is more affecting the runtime, and with the idea to decouple the runtime from the operator, then, this should be transparent. However it makes sense to keep an eye on it.

Multitenancy

• Remove the single namespace limitation

The new multitenancy model development has still some aspect to polish. One of them is the ability to have more than one operator running in the same namespace. At that stage, I guess that we may even remove or deprecate the global operator feature.

Observability

• Move from opentracing to opentelemetry
• Provide SLO/SLI/SLA metrics

As we are focusing on "enterprise" features, probably we should have something more on the Observability aspect, such as providing Service Level whatever available in the exposed metrics.

CLI

• Move CLI logic in Camel JBang
• Enhance source code route introspection

A lot of work is already done in Camel JBang. We should join effort and move the logic of kamel cli in camel cli instead. One very important feature we should enhance is the possibility to do a deep code introspection. Right now we can discover simple components and capabilities and it would be a great help to the user experience to have it automatic.

Kustomize parity features with kamel install

As soon as we have parity between Kustomize and kamel install we may deprecate the latter.

Kamelets

• Headers, Schema, Data types

We can provide more features on the Kamelets side in order to be able to use headers, automatic data formats, etc.

Strengthen Keda, KNative offering

Not sure what we should do, but KNative and Keda are an important complement to the serverless capability of Camel K. Probably we should check the latest developments of both projects and add any missing feature to Camel K. Plus, promoting with examples.

Technical marketing: “how to” blogs

Last year we've done a series of blogs that were well received by the community on crucial aspect of Camel K operations. I invite to keep this as part of our roadmap and maintain a certain cadence so that the community can discover features that otherwise may be kept hidden in the code.

Documentation: keep in line with latest development

Also this one, although adds nothing to the roadmap, is a reminder to the development team to update the documentation beside any change done to the code (in particular if we're introducing breaking changes).

Please, feel free to comment in order to add more topics or to give more arguments to the topic I've added above. Thanks in advance!

[oscerd]

For the Kamelets releases we are already doing more or less like that.

What I'd like to explore too, is the ability to avoid using a tag from Camel Kamelets, but instead create some kind of container image, containing only the Kamelets to be used in Camel K.

[squakez]

For the Kamelets releases we are already doing more or less like that.

What I'd like to explore too, is the ability to avoid using a tag from Camel Kamelets, but instead create some kind of container image, containing only the Kamelets to be used in Camel K.

I was evaluating the possibility to use instead the kamelet.jar that is released (and contain the kamelets). However, I've later discovered that the original design was to include a flat (git) repository, reason why I moved on the usage of a tag. As an example, think on the kamel kamelet add-repo github:owner/repo[/path_to_kamelets_folder][@version] command.

I think it is wise to keep this in order to simplify the Kamelet development. In the contrary, we may need to create a maven project (or any other way to contain the Kamelets) and release them somewhere. In any case it's not a strong opinion, it would be nice to collect some users feedback here.

[lburgazzoli]

I'd second @oscerd idea to explore using container images as lot of other tools/projects are moving to support oci registries to store some kind of artefacts (help, fluxcd, crossplane, etc)

Related issues:

• use the OCI containers to distribute camel-k artifacts  #2732

Of course we can support both but in general using OCI registries seems to be very useful

[tadayosi]

Under the CLI refactoring, we can also consider including this task for Camel K 2.

• Split Operator binary from Kamel CLI binary #3175

[squakez]

I'd second @oscerd idea to explore using container images as lot of other tools/projects are moving to support oci registries to store some kind of artefacts (help, fluxcd, crossplane, etc)

Related issues:

* [use the OCI containers to distribute camel-k artifacts  #2732](https://github.com/apache/camel-k/issues/2732)

Of course we can support both but in general using OCI registries seems to be very useful

Okey. @oscerd @lburgazzoli Just to understand the possible proposal, are you thinking that Camel K bundles the Kamelets into a container image or should we expect that Kamelet release process do that and we reuse it in Camel K?

[squakez]

Adding a comment to provide some notes after some discussion I had privately with other commiters. Some interesting ideas would be to leverage the work we're willing to carry on with #3831 and allow Camel K operator to be able to manage any Camel based workload. This would open the possibility to have any other runtime (not only Camel Quarkus) to be managed by the operator.

An other feature we discussed is the ability to let the user build/deploy the Camel application (manually, via a pipeline tool) and allow the operator to intercept and manage certain operations (for instance, observability). This option probably goes in the direction of enabling GitOps.

As a side note, we'll keep this issue open for a couple of more weeks in order to let any community user to add any further feedback or idea. By the end of January we'll close it and we'll publish the roadmap as a blog post.

[davsclaus]

Good suggestions in this thread, and I suggest to moving all of this into Camel K 2.x, and keep 1.x as-is.

Here is some of my brain-dump

Let Camel K 2 be based on:

• Camel v4
• run->deploy via build pipelines as primary (gitops ... favour code in git not in CRD)
• build using standard Quarkus / Spring Boot (via their maven package)
• make application.properties|yaml first-class (and not modeline / complex confusing traits) - then you can use standard Q/SB way of configuring (also k8s specifics)
• If Camel K needs to argument the build then have plugins/customizations to Q/SB build / application.properties /jbang script etc)
• Maybe only allow Kamelet Binding in CRD (for small event based rules)
• camel-jbang for local development
• if you need to develop directly in k8s, then camel-jbang can possible be enhanced to do that
• Let Camel K operator be able to manage also any kind of Camel deployments (also traditional CSB/CQE - for example via label selectors). Can be done later in 2.x.
• Better unit testing (also needed in camel-jbang) (could maybe be yaks with jbang ?)
• Cross-over from traditional CEQ/CSB to Camel-K would then be much smoother and similar UX, and vice-versa, eg you start with Camel K and then want to do a full standard maven based project, and move over to CSB/CQE

[oscerd]

I personally don't think we need to throw away everything from v1. So I would mainly focus on improving/enhancing what we have, but not restarting from scratch or completely move to a different architecture.

[squakez]

Thanks @davsclaus for your latest feedback, it's a valid vision of future goals for sure. IMO we're not in the position to do a full redesign of the project (we may require a lot of development resource we don't have ATM), although, trying to resume your points, we should aim to make Camel K the deployment tool for Kubernetes Camel applications (whichever runtime the user want to choose). For this reason the main aim is to decompose as much as we can the building part, to make it more explicit and be able in a following iteration to hook with anything Camel related (being a generic Maven project, a git repo, etc...). For this reason, we should not be afraid of introducing some breaking change (therefore having a major Camel K 2.0). However, as pointed out by @oscerd we still have a valid design, but we definitely need to think ahead of time of what we eventually want to have, so that Camel K 2.0 could be a more flexible design to simplify future Camel K version developments. The build part is quite strategic and focusing on that together with decoupling from Camel runtime is going a big part of these changes.

[squakez]

Thanks all for the great feedback received during this latest week. I think we have a fair idea of the possibility in front of us for 2023. I will wrap up this in a blog post to announce the ideas discussed here so that we can provide a plan for future development.

[mgubaidullin]

@squakez I have just a few questions about we should aim to make Camel K the deployment tool for Kubernetes Camel applications. Camel application is just a common Java application with Camel libraries in it.
The mostly used Camel Runtimes: Quarkus and SpringBoot.

1. Quarkus has its own extension for deployment
2. SpringBoot usually used with jkube

Why do we need to develop a Camel specific deployment tool?
What value does it bring to developers?

[Delawen]

I like the idea of extending Camel K as a tool that handles all potential Camel runtimes around K8. At least for me it makes sense from the point of view of an integration developer. One command line to interact with the cluster in all Camel sense. Install, deploy, monitor,...

Not to mention it also makes sense to me as a way to easily connect other libraries and applications (obviously thinking on the editor Kaoto.io here).

JBang can be adapted to use K8, alright, but we already have Camel K that does it for many DSLs. Why should we cut down features we already have in Camel k? Enhancing JBang doesn't mean cutting down Camel K which already works. Maybe even the adaptation of JBang to interact with the cluster could be done via camel-k. That would make sense to me, reusing existing resources.

Plus, JBang is Java related, while Camel K is kind of agnostic. Not all Camel developers are Java developers, right?

[squakez]

@squakez I have just a few questions about we should aim to make Camel K the deployment tool for Kubernetes Camel applications. Camel application is just a common Java application with Camel libraries in it. The mostly used Camel Runtimes: Quarkus and SpringBoot.

1. Quarkus has its own extension for deployment

2. SpringBoot usually used with jkube

Why do we need to develop a Camel specific deployment tool? What value does it bring to developers?

I think that, beside simplifying the deployment model (from a simple route deep down the whole Kubernetes world of configuration to tune), Camel K add a series of operational aspects, such as monitoring, scaling, etc, that may be possible without CK, but definitely more difficult to achieve. The main values I can list on top of my head for Camel K are:

• Automating the parsing of a route and composing a project with all required components/dependencies (still room for improvement here, and likely we'll join the Camel JBang to fully achieve it as stated in 2023 roadmap)
• Inherit previous Camel K kit containers and reduce the time to build/run an application
• Manage the container registry on behalf of the user
• First class citizen for Serverless in conjunction with KNative
• Interaction with other operators (Keda, Knative, ServiceBinding)
• Enable a managed service model and/or an EDA via Kamelet/KameletBinding
• Promote a Camel application across environments
• Injecting resources (properties, configmaps, ...) to be used easily from the route
• Fine tune Kubernetes with simple traits (ie, you don't have to create a Service if the route expose an http port)
• Simplify the upgrade process when a new Camel version is available

I guess I forget something. In general I think that, although a Camel application may look like any other Java application, in reality, there are more opinionated aspects typical of a framework that Camel K manage for the user.

[valdar]

Why do we need to develop a Camel specific deployment tool?

I think we can flip the perspective here: camel k is a k8s specific tool, and to manage entities on k8s something more similar to an operator than to a maven plugin is needed.

What value does it bring to developers?

In the light of the perspective flip, I think part of the value added is more obvious: a comprehensive k8s experience.

[mgubaidullin]

With camel-quarkus I have quite a comprehensive k8 experience. What I have with Camel-K that I can not have with camel-quarkus?

[oscerd]

Faster build when you have a kit built before for the same application.

Serverless integration ootb.

Just to say a couple of things.

I don't quite get what we are discussing here. Camel k is important for the camel ecosystem and it has a lot to give and we have to improve it.

[orpiske]

* Let Camel K operator be able to manage also any kind of Camel deployments (also traditional CSB/CQE - for example via label selectors). Can be done later in 2.x.

+1 for this. I think it can potentially help / improve Camel in other areas too.

[tadayosi]

• Let Camel K operator be able to manage also any kind of Camel deployments (also traditional CSB/CQE - for example via label selectors). Can be done later in 2.x.

I like this idea, too. hawtio-online has been providing such feature on K8s/OpenShift, but it requires a Camel application to be Jolokia-enabled. I'm interested in how much functionality can be transferred from hawtio-online to the Camel K operator.

[lburgazzoli]

• Let Camel K operator be able to manage also any kind of Camel deployments (also traditional CSB/CQE - for example via label selectors). Can be done later in 2.x.

I like this idea, too. hawtio-online has been providing such feature on K8s/OpenShift, but it requires a Camel application to be Jolokia-enabled. I'm interested in how much functionality can be transferred from hawtio-online to the Camel K operator.

I think the main advantage are:

• discoverability (no need to scrape pods and container port, just look at integration CR)
• the status reporting of an Integration can include information for tooling (management consoles, cli, etc) to access the management interface
• tools (management consoles, cli, etc) can use the integration' status reporting to provide information about the status of the integration without having to deal with internals of camel (i.e. scraping pods health check probes) which greatly reduces grants required by 3th party applications and avoid doing the same thing on a number of tools.
• camel-k operator can set-up the integration automatically to enable secure access to the management endpoint (SSL, AuthN, AuthZ, etc)
• would make kubectl a first class citizen for camel k operations

Unrelated to camel-k, I think we can try to make the new camel-console the way to interact with camel leveraging i.e. spring boot actuators and future dedicated management support in quarkus to expose such functionality and get rid of JMX (at least as the protocol to interact with the running instances).