From 5e77d63a0caa22bbf39fecc6b050e32b1b6398f9 Mon Sep 17 00:00:00 2001 From: Meg McRoberts <meg.mcroberts@dynatrace.com> Date: Thu, 24 Aug 2023 22:34:37 -0700 Subject: [PATCH 1/7] add info to implementing/tasks about sequential execution Signed-off-by: Meg McRoberts <meg.mcroberts@dynatrace.com> --- .../en/docs/implementing/tasks/_index.md | 55 +++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/content/en/docs/implementing/tasks/_index.md b/docs/content/en/docs/implementing/tasks/_index.md index 5e692c7cca..f037ba13e0 100644 --- a/docs/content/en/docs/implementing/tasks/_index.md +++ b/docs/content/en/docs/implementing/tasks/_index.md @@ -84,6 +84,61 @@ See the [KeptnTaskDefinition](../../yaml-crd-ref/taskdefinition.md) reference page for the synopsis and examples for each runner. +## Executing sequential tasks + +All `KeptnTaskDefinition` resources at the same level +(either pre-deployment or post-deployment) execute in parallel. +This is by design, because Keptn is not a pipeline engine. +Task sequences that are not part of the lifecycle workflow +should not be handled by KLT +but should instead be handled by the pipeline engine tools being used +such as Jenkins, Argo Workflows, Flux, and Tekton. + +For task sequences that are part of the lifecycle workflow +can be defined sequentially in a `KeptnTaskDefinition` resource, +which can execute a virtually unlimited number of programs and actions. +You have great flexibility in how you define your `KeptnTaskDefinition` resources, +allowing you to define the ideal mix of tasks that execute sequentially +and tasks (or sets of tasks) that execute in parallel. + +If you define a `container-runtime` runner container +for your `KeptnTaskDefinition`, +you can do anything allowed by the configuration you define for that container. + +If you use either the `deno-runtime` or `python-runtime` runner, +you can specify the actions to take by coding the actual calls +inline in the manifest, +by calling scripts on a remote webserver, +or by calling other `KeptnTaskDefinition` resources you have defined. + +As an example, consider the `KeptnTaskDefinition` resources +you need for testing your `KeptnApp`: + +- Perhaps you need to run a set of integration tests, + a set of performance tests, and a set of regression tests. +- Perhaps you need to run these on Linux, MacOS, and Windows. +- Perhaps you need to run these for Java 11, Java 17, and Java 21. + +You have several options: + +- Define one `KeptnTaskDefinition` resource that runs + integration tests, then regression test, then performance tests + in order. + + - You could use the `functionRef` syntax + and code the calling sequences for all tests + inside your `KeptnTaskDefinition` resource + or you could use the `httpRef` syntax + to call scripts from an external webserver. + - You could code separate `KeptnTaskDefinition` resources + for integration tests, regression tests, and performance tests. + These three test sets could then run in parallel. + - You could create a `KeptnTaskDefinition` resource + that uses the `functionRef` syntax to call the + `KeptnTaskDefinition` resources for each type of testing. + Executing that resource would execute the three types of tests + in sequential order. + ## Context A Kubernetes context is a set of access parameters From 421185937416941d5e1b2f638d89be304702fb8c Mon Sep 17 00:00:00 2001 From: Meg McRoberts <meg.mcroberts@dynatrace.com> Date: Thu, 31 Aug 2023 04:27:49 -0700 Subject: [PATCH 2/7] start rewrite Signed-off-by: Meg McRoberts <meg.mcroberts@dynatrace.com> --- .../en/docs/implementing/tasks/_index.md | 36 +++++++++++++------ 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/docs/content/en/docs/implementing/tasks/_index.md b/docs/content/en/docs/implementing/tasks/_index.md index f037ba13e0..cc3aa0ef36 100644 --- a/docs/content/en/docs/implementing/tasks/_index.md +++ b/docs/content/en/docs/implementing/tasks/_index.md @@ -7,28 +7,34 @@ hidechildren: false # this flag hides all sub-pages in the sidebar-multicard.htm A [KeptnTaskDefinition](../../yaml-crd-ref/taskdefinition.md/) -resource defines tasks that the Keptn Lifecycle Toolkit runs +resource defines one or more "executables" +(functions, programs, scripts, etc) +that the Keptn Lifecycle Toolkit runs as part of the pre- and post-deployment phases of a [KeptnApp](../../yaml-crd-ref/app.md) or [KeptnWorkload](../../crd-ref/lifecycle/v1alpha3/#keptnworkload). -A Keptn task executes as a runner in an application +A +[KeptnTask](../../crd-ref/lifecycle/v1alpha3/#keptntask) +executes as a runner in an application [container](https://kubernetes.io/docs/concepts/containers/), which runs as part of a Kubernetes [job](https://kubernetes.io/docs/concepts/workloads/controllers/job/). -A `KeptnTaskDefinition` includes a function -that defines the action taken by that task. +A `KeptnTaskDefinition` includes calls to executables to be run. -To implement a Keptn task: +To implement a `KeptnTask`: - Define a [KeptnTaskDefinition](../../yaml-crd-ref/taskdefinition.md) resource that defines the runner to use for the container + and the executables to be run +pre- and post-deployment - Apply [basic-annotations](../integrate/#basic-annotations) to your workloads to integrate your task with Kubernetes -- Add your task to the [KeptnApp](../../yaml-crd-ref/app.md) - resource that associates your `KeptnTaskDefinition` - with the pre- and post-deployment tasks that should run in it; +- Annotate the appropriate + [KeptnApp](../../yaml-crd-ref/app.md) + resource to associate your `KeptnTaskDefinition` + with the pre- and post-deployment tasks that should run it; see [Pre- and post-deployment tasks and checks](../integrate/#pre--and-post-deployment-checks) for more information. @@ -36,6 +42,10 @@ To implement a Keptn task: This page provides information to help you create your tasks: - Code your task in an appropriate [runner](#runners-and-containers) +- How to control the + [execution order](#executing-sequential-tasks) + of functions, programs, and scripts + since all `KeptnTask` resources at the same level run in parallel - Understand how to use [Context](#context) that contains a Kubernetes cluster, a user, a namespace, the application name, workload name, and version. @@ -53,6 +63,9 @@ to define the task. The `spec` section of the `KeptnTaskDefinition` defines the runner to use for the container: +KLT provides a general Kubernetes that you can configure +to do almost anything you want: + - The `container-runtime` runner provides a pure custom Kubernetes application container that you define to includes a runtime, an application @@ -86,11 +99,12 @@ reference page for the synopsis and examples for each runner. ## Executing sequential tasks -All `KeptnTaskDefinition` resources at the same level +All `KeptnTask` resources that are defined by +`KeptnTaskDefinition` resources at the same level (either pre-deployment or post-deployment) execute in parallel. This is by design, because Keptn is not a pipeline engine. -Task sequences that are not part of the lifecycle workflow -should not be handled by KLT +**Task sequences that are not part of the lifecycle workflow +should not be handled by KLT** but should instead be handled by the pipeline engine tools being used such as Jenkins, Argo Workflows, Flux, and Tekton. From 893b63f504a0029eb22a590faefbfda1a8aa80dd Mon Sep 17 00:00:00 2001 From: Meg McRoberts <meg.mcroberts@dynatrace.com> Date: Thu, 31 Aug 2023 22:41:51 -0700 Subject: [PATCH 3/7] rewrite in response to Adam's comments Signed-off-by: Meg McRoberts <meg.mcroberts@dynatrace.com> --- .../en/docs/implementing/tasks/_index.md | 81 ++++++++++++++----- 1 file changed, 60 insertions(+), 21 deletions(-) diff --git a/docs/content/en/docs/implementing/tasks/_index.md b/docs/content/en/docs/implementing/tasks/_index.md index cc3aa0ef36..54bdc734bc 100644 --- a/docs/content/en/docs/implementing/tasks/_index.md +++ b/docs/content/en/docs/implementing/tasks/_index.md @@ -108,32 +108,67 @@ should not be handled by KLT** but should instead be handled by the pipeline engine tools being used such as Jenkins, Argo Workflows, Flux, and Tekton. -For task sequences that are part of the lifecycle workflow -can be defined sequentially in a `KeptnTaskDefinition` resource, -which can execute a virtually unlimited number of programs and actions. -You have great flexibility in how you define your `KeptnTaskDefinition` resources, -allowing you to define the ideal mix of tasks that execute sequentially -and tasks (or sets of tasks) that execute in parallel. - -If you define a `container-runtime` runner container -for your `KeptnTaskDefinition`, -you can do anything allowed by the configuration you define for that container. +If your lifecycle workflow includes +a sequence of executables that need to be run in order, +you can put them all in one `KeptnTaskDefinition` resource, +which can execute a virtually unlimited number +of programs, scripts, and functions, +as long as they all need the same runner, such as Python. If you use either the `deno-runtime` or `python-runtime` runner, you can specify the actions to take by coding the actual calls inline in the manifest, -by calling scripts on a remote webserver, +by calling scripts from a remote webserver, or by calling other `KeptnTaskDefinition` resources you have defined. - -As an example, consider the `KeptnTaskDefinition` resources -you need for testing your `KeptnApp`: - -- Perhaps you need to run a set of integration tests, - a set of performance tests, and a set of regression tests. -- Perhaps you need to run these on Linux, MacOS, and Windows. -- Perhaps you need to run these for Java 11, Java 17, and Java 21. - -You have several options: +This provides great flexibility in +how you define your `KeptnTaskDefinition` resources, +allowing you to define the ideal mix of executables that run sequentially +and executables (or sets of executables) that run in parallel. + +As an example, let's say you need to run a set of integration tests, +a set of performance tests, and a set of regression tests. + +* You can create one `KeptnTaskDefinition` that calls all the tasks, + in order, either by putting the actual calls + in the `KeptnTaskDefinition` resource (`inline` syntax) + or by calling scripts from a remote webserver (`httpRef` syntax). + +* You can create separate `KeptnTaskDefinition` resources + for integration tests, performance tests, and regression tests. + + - If you annotate the `KeptnApp` resource to call + each of these `KeptnTask` resources, + the three sets of tests run in parallel. + + - You can create a "parent" `KeptnTaskDefinition` resource + that uses the `functionref` syntax + to call the `KeptnTaskDefinition` resources + for integration tests, performance tests, and regression tests. + If you annotate the `KeptnApp` resource + to run this parent `KeptnTask` resource, + all tests run sequentially. + + This approach also allows you to run the test sequence if, for example, + the integration tests require the `deno-runtime` runner + but the performance and regression tests + require the `python-runtime` runner. + The parent `KeptnTaskDefinition` can run `KeptnTask` resources + that use different runners, + although the "parent" definition runtime is used + for the container that runs all the tests. + In other words, the parent `KeptnTaskDefinition` resources + is not a merge of other `KeptnTaskDefinition` resources + but rather the code/container of the parent runner. + +* If you need to test your deployment for different platforms + (such as Linux, MacOS, and Windows) + or for different software versions + (such as Java 11, Java 17, and Java 21), + parallel testing can improve performance. + In this case, you can construct different `KeptnTaskDefinition` resources + for each platform, + perhaps defining different input parameters + (such as different secrets or environment variables). - Define one `KeptnTaskDefinition` resource that runs integration tests, then regression test, then performance tests @@ -153,6 +188,10 @@ You have several options: Executing that resource would execute the three types of tests in sequential order. +If you define a `container-runtime` runner container +for your `KeptnTaskDefinition`, +you can do anything allowed by the configuration you define for that container. + ## Context A Kubernetes context is a set of access parameters From 8926eb4e2aed99569c37949c9530c0a289979322 Mon Sep 17 00:00:00 2001 From: Meg McRoberts <meg.mcroberts@dynatrace.com> Date: Thu, 31 Aug 2023 22:44:01 -0700 Subject: [PATCH 4/7] markdownlint-fix Signed-off-by: Meg McRoberts <meg.mcroberts@dynatrace.com> --- docs/content/en/docs/implementing/tasks/_index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/content/en/docs/implementing/tasks/_index.md b/docs/content/en/docs/implementing/tasks/_index.md index 54bdc734bc..5b8ccc1616 100644 --- a/docs/content/en/docs/implementing/tasks/_index.md +++ b/docs/content/en/docs/implementing/tasks/_index.md @@ -128,12 +128,12 @@ and executables (or sets of executables) that run in parallel. As an example, let's say you need to run a set of integration tests, a set of performance tests, and a set of regression tests. -* You can create one `KeptnTaskDefinition` that calls all the tasks, +- You can create one `KeptnTaskDefinition` that calls all the tasks, in order, either by putting the actual calls in the `KeptnTaskDefinition` resource (`inline` syntax) or by calling scripts from a remote webserver (`httpRef` syntax). -* You can create separate `KeptnTaskDefinition` resources +- You can create separate `KeptnTaskDefinition` resources for integration tests, performance tests, and regression tests. - If you annotate the `KeptnApp` resource to call @@ -160,7 +160,7 @@ a set of performance tests, and a set of regression tests. is not a merge of other `KeptnTaskDefinition` resources but rather the code/container of the parent runner. -* If you need to test your deployment for different platforms +- If you need to test your deployment for different platforms (such as Linux, MacOS, and Windows) or for different software versions (such as Java 11, Java 17, and Java 21), From 9f07a0ed06adeea856b4fa94b1e7da0a3dbf39d1 Mon Sep 17 00:00:00 2001 From: Meg McRoberts <meg.mcroberts@dynatrace.com> Date: Fri, 1 Sep 2023 01:50:31 -0700 Subject: [PATCH 5/7] Update docs/content/en/docs/implementing/tasks/_index.md Signed-off-by: Meg McRoberts meg.mcroberts@dynatrace.com Co-authored-by: Giovanni Liva <giovanni.liva@dynatrace.com> Signed-off-by: Meg McRoberts <meg.mcroberts@dynatrace.com> --- docs/content/en/docs/implementing/tasks/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/en/docs/implementing/tasks/_index.md b/docs/content/en/docs/implementing/tasks/_index.md index 5b8ccc1616..f1eb345130 100644 --- a/docs/content/en/docs/implementing/tasks/_index.md +++ b/docs/content/en/docs/implementing/tasks/_index.md @@ -131,7 +131,7 @@ a set of performance tests, and a set of regression tests. - You can create one `KeptnTaskDefinition` that calls all the tasks, in order, either by putting the actual calls in the `KeptnTaskDefinition` resource (`inline` syntax) - or by calling scripts from a remote webserver (`httpRef` syntax). + or by calling scripts from a remote webserver (`httpRef` syntax), or by providing a container to run. - You can create separate `KeptnTaskDefinition` resources for integration tests, performance tests, and regression tests. From c7a34f2c1ce4efb0157399e32f47d78d86b9dbc9 Mon Sep 17 00:00:00 2001 From: Meg McRoberts <meg.mcroberts@dynatrace.com> Date: Fri, 1 Sep 2023 01:53:20 -0700 Subject: [PATCH 6/7] Update docs/content/en/docs/implementing/tasks/_index.md Signed-off-by: Meg McRoberts meg.mcroberts@dynatrace.com Co-authored-by: Giovanni Liva <giovanni.liva@dynatrace.com> Signed-off-by: Meg McRoberts <meg.mcroberts@dynatrace.com> --- docs/content/en/docs/implementing/tasks/_index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/content/en/docs/implementing/tasks/_index.md b/docs/content/en/docs/implementing/tasks/_index.md index f1eb345130..9be833db21 100644 --- a/docs/content/en/docs/implementing/tasks/_index.md +++ b/docs/content/en/docs/implementing/tasks/_index.md @@ -114,6 +114,7 @@ you can put them all in one `KeptnTaskDefinition` resource, which can execute a virtually unlimited number of programs, scripts, and functions, as long as they all need the same runner, such as Python. +Another option is to encode all your steps in the language of your choice and build a container that Keptn will execute. If you use either the `deno-runtime` or `python-runtime` runner, you can specify the actions to take by coding the actual calls From a4573a38573f126782a2aa9c0e9d6deeed065d82 Mon Sep 17 00:00:00 2001 From: Meg McRoberts <meg.mcroberts@dynatrace.com> Date: Fri, 1 Sep 2023 02:00:54 -0700 Subject: [PATCH 7/7] Update docs/content/en/docs/implementing/tasks/_index.md Signed-off-by: Meg McRoberts meg.mcroberts@dynatrace.com Signed-off-by: Meg McRoberts <meg.mcroberts@dynatrace.com> --- docs/content/en/docs/implementing/tasks/_index.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/content/en/docs/implementing/tasks/_index.md b/docs/content/en/docs/implementing/tasks/_index.md index 9be833db21..40534306e6 100644 --- a/docs/content/en/docs/implementing/tasks/_index.md +++ b/docs/content/en/docs/implementing/tasks/_index.md @@ -114,7 +114,10 @@ you can put them all in one `KeptnTaskDefinition` resource, which can execute a virtually unlimited number of programs, scripts, and functions, as long as they all need the same runner, such as Python. -Another option is to encode all your steps in the language of your choice and build a container that Keptn will execute. + +Another option is to encode all your steps in the language of your choice +and build a container that Keptn executes. +This is often the best solution if you need to execute complex sequences. If you use either the `deno-runtime` or `python-runtime` runner, you can specify the actions to take by coding the actual calls