fix(#592): Introduce build order strategy

[christophd]

The new build order strategy simply decides which of the already submitted builds gets picked first for execution (e.g. FIFO). This already allows to run the submitted builds in parallel to each other as long as max running builds limit has not been reached.

• Run builds on same operator namespace with user defined strategy
• Default strategy is "sequential" running only one single build at a time
• Also support "fifo" strategy where builds are run/queued based on their creation timestamp. This strategy allows parallel builds as long as individual build dependency lists are not colliding
• Users may adjust/overwrite the build order strategy via install command option, in the (local) integration platform settings or via the builder trait option
• Max number of running builds limitation is not affected by the build order strategy (ensure to always obey the limitation)

Release Note

feat: Introduce build order strategy

[squakez]

I think the most interesting feature would be to have a strategy that run in parallel unless there is any possible dependency between builds (which I'd propose as the default choice). The structure looks fine, so, looking forward to see such implementation in place.

[christophd]

@squakez yes, this has been my priority, too. Allowing parallel builds where possible. And then I realized that such a strategy is almost already in place.

When Integrations are independent to each other in terms of dependencies and the used Camel DSL the operator will create separate integrationkits and submit the builds in parallel to each other already. The last missing piece was to remove the single running build per namespace limitation that strictly queued all builds for a namespace (e.g. the operator namespace). Now that we have the max running builds limitation we can safely run builds in parallel once they are submitted.

The new build order strategy simply decides which of the already submitted builds gets picked first for execution (e.g. FIFO). And this already allows to run the submitted builds in parallel to each other as long as max running builds limit has not been reached.

[christophd]

The default build order strategy is set to sequential at the moment. Sequential strategy will queue the builds for a namespace which is exactly like it has been implemented before in Camel K v1. I wonder if we should use fifo as a new default strategy in Camel K v2 instead in order to run builds in parallel to each other as long as max running build limit has not been reached.

WDYT?

[squakez]

What I'd like to see as a default is the following behavior:

Given an Integration X with dependencies: a,b
Given an Integration Y with dependencies: a,b,c
Given an Integration Z with dependencies: d,e,f

When the Integrations are required to be built at the same time, the operator should be able to start a parallel build for X and Z only so that the Y will be able to reuse the Kit from X when this is finished (incremental build). But also we want to have a possible parallelism of Z which has no dependencies from other running builds..

I think this is what we actually miss.

[christophd]

Sorry, but I do not see how Y will benefit from build X. I had some local testing and I think that build Y is independent of build X. The integrationkit and the image built for Y is not building on top of X from what I can see. Maybe I am missing a piece.

[squakez]

Sorry, but I do not see how Y will benefit from build X. I had some local testing and I think that build Y is independent of build X. The integrationkit and the image built for Y is not building on top of X from what I can see. Maybe I am missing a piece.

As discussed offline, the incremental image build is a feature, so we should understand how to leverage that.

[christophd]

Yeah, I was missing a piece and image layering is working as expected. I need to improve the order strategy to actually work with the layering

[christophd]

@squakez I have added another build order strategy dependencies that uses the list of dependencies to queue builds that may reuse image layers of already scheduled or running builds

[squakez]

@squakez I have added another build order strategy dependencies that uses the list of dependencies to queue builds that may reuse image layers of already scheduled or running builds

Nice. It would be good to have an e2e test to validate the strategy, ie, an integration that depends on another really queues up until the depending kit is ready.

[squakez]

LGTM.

[squakez]

It would be nice if we could return the reason why a build is allowed or not, so that we include it in the condition. In particular, we may show the user that a build is queued waiting for a depending build to complete. However I'm not sure how much complicated is to bubble up this information, you please evaluate if it makes sense to include it or defer to a later development. I think it also would simplify the e2e test (ie, we just need to check that a build is queued up with the proper condition without waiting it to complete)