Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: PipelineLoop: iteration counter #886

Closed
Udiknedormin opened this issue Mar 22, 2022 · 4 comments
Closed

Feature: PipelineLoop: iteration counter #886

Udiknedormin opened this issue Mar 22, 2022 · 4 comments

Comments

@Udiknedormin
Copy link
Contributor

/kind feature

Description:
[A clear and concise description of what your proposal. What problem does it solve?]
Currently, PipelineLoop enables the user to choose iterationParam, which contains the element the loop loops over. It's a functional equivalent of the following Python code:

for el in iterable:
  ...

However, the input is always sequential, so the number of each iteration is known --- as proved by the fact that iteration_number is populated in the execution tracking yaml. That information, however, is not available at runtime, so no task nor custom-task may use it e.g. when reporting the value. It's therefore impossible to do the functional equivalent of:

for i, el in enumerate(iterable):
  print(f"iteration {i}, value: {el}")

Additional information:
I would suggest adding a new parameter, iterationNumberParam, pointing at a parameter used to denote which iteration it is in.

From the DSL perspective, I'd suggest tekton.Loop to simply get a new field --- iteration_number, of type TektonLoopIterationNumber(dsl.PipelineParam). It would be a much simpler cousin to LoopArguments(dsl.PipelineParam), as no sub-vars, to_list nor from_pipeline_param would be needed --- just a PipelineParam of type int that will be populated by the parameter listed in iterationNumberParam.

If backward-compatibility for not generating iterationNumberParam would be an issue, iteration_number can always be made a property which, on access, sets a flag iteration_number_used so that the compiler knows if the DSL used it at least once or not. If not --- then it will generate the very same yaml it does today, so no incompatibility can occur.

Example:

producer = ListProducerOp(...)
loop = tekton.Loop(producer.output)
with loop as el:
  PrintOp(f"iteration {loop.iteration_number}, element: {el}")

If that's bothersome (it may be, considering how iteration_number would be accessible outside of the loop body, which isn't really what is expected), some simple helper method may be added:

producer = ListProducerOp(...)
with tekton.Loop(producer.output).enumerate() as (i, el):
  PrintOp(f"iteration {i}, element: {el}")
@Tomcli
Copy link
Member

Tomcli commented Mar 24, 2022

Action items:

  • Update PipelineLoop controller to add new Tekton variable iterationNumberParam for parameters that points to the iteration counter. Similar to the iterateNumeric for numeric iteration.
  • Update SDK with a new extension function .enumerate(). It returns two PipelineParams that points to the iteration counter and the actual value.

The SDK part will be tricky because it changes the original design for parallelFor DSL. We need to first implement this feature on the controller level first to figure out what spec is easier to map back to the DSL format.

@Tomcli
Copy link
Member

Tomcli commented Mar 24, 2022

fyi @yhwang @ScrapCodes

@Udiknedormin
Copy link
Contributor Author

I verified that loop counter works as expected. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants