Split / Join in Pipelines

[wkulhanek]

Feature request

It appears there is currently no way to split / join a Pipeline. This is a basic feature that is missing - and will prevent broad adoption of Tekton Pipelines. Almost every Pipeline that I build (besides simple demo ones) requires this capability.

Note that this is different from parallel execution of tasks where all tasks will run.

Use case

The use case is actually quite simple.

In my pipeline I have Task A.

After Task A I either need to execute Task B1 or Task B2 - but not both -> Split. This could also be B1/B2/B3/B4/... In my example I have a mutually exclusive when condition on tasks B1 and B2 (is condition / is not condition) guaranteeing that one task gets executed.

After either task finishes I would like to continue the pipeline with task C -> Join.

I tried to set this up with runAfter clauses on the tasks - but the pipeline just ends after either Task B1 or B2 is executed - Task C is never reached.

[vdemeester]

/cc @pritidesai @jerop
I think it might be covered by TEP-0059 but I wasn't sure as I didn't see described in the motivation/use cases.

[jerop]

@vdemeester yes, it'll be solved by TEP-0059; we're discussing the proposal and alternatives in tektoncd/community#407

@wkulhanek thanks for the detailed description of the scenario - if possible, please share real-world use case or example of how you need to use this feature so that we can consider it in TEP-0059

[afrittoli]

/cc @pritidesai @jerop
I think it might be covered by TEP-0059 but I wasn't sure as I didn't see described in the motivation/use cases.

TEP-0059 does not help with mutually exclusive conditions but it helps (a bit) with joining back.

The split can be achieved today using when expressions but it's a bit verbose.
For example:

                Task A
        Implement Condition
        Has result called "check"
   Writes 'true' or 'false' to result
         |                         \
      Task B                     Task C
  when expression            when expression
 "check" in ["true"]       "check" in ["false"]

After TEP-0059 it will be possible for both (B) and (C) to be marked with "Task" scope, and thus for a following Task D to run.

This is true only as long as Task D is marked as runAfter['B', 'C'] but has no dependency to either B or C.

I think a complete solution for this use case should allow to:

• express the condition only once, instead of requiring two opposite condition
• allow for a results to be provided by either task B or task C

We could event consider a N split and join, where the condition may generate N different results - a "switch" kind of construct.

[wkulhanek]

@jerop The simple case that I fell over was that I built a generic pipeline that builds and deploys a Java based application.

But the actual deployment was happening either via a "Deployment" or via a "Knative Service" based on a boolean flag on the PipelineRun.

So all my steps in the pipeline are the same (git checkout, maven build, image build, tag, ...). The only thing that's different is how to update the "application" in my namespace.

[wkulhanek]

@afrittoli This is exactly what I had. Except I would suggest that just using the "result" of Task A would be too narrow. In my case the condition is a boolean input parameter on the PipelineRun itself rather than being generated by Task A.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[wkulhanek]

/remove-lifecycle stale I'm kind of surprised this isn't getting any attention. Seems like this is the most basic capability that's missing...

[pritidesai]

Hi @wkulhanek the simple use case you have provided in the description is now supported with branch scoped when expressions (released in 0.27). Please refer to the release notes or the documentation on how to configure it.

We have introduced a new feature flag scope-when-expressions-to-task (set to false by default). Setting it to true will enable branch scoped when expressions i.e. task C will be executed after either B1 or B2.

when expressions guards C (and never reached) if it relies on the result of either B1 or B2, a new feature is coming to address this use case where C can continue executing without actual results - demoed in the API WG.

Hope it helps, let me know if there is anything missing 🙏

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.