Skipping Strategies

This change implements skipping strategies to give users the flexibility to skip a single guarded Task only and unblock execution of its dependent Tasks. Today, WhenExpressions are specified within Tasks but they guard the Task and its dependent Tasks. To provide flexible skipping strategies, we want to change the scope of WhenExpressions from guarding a Task and its dependent Tasks to guarding the Task only. If a user wants to guard a Task and its dependent Tasks, they can: - cascade the WhenExpressions to the dependent Tasks - compose the Task and its dependent Tasks as a sub-Pipeline that's guarded and executed together using Pipelines in Pipelines (but this is still an experimental feature) Changing the scope of WhenExpressions to guard the Task only is backwards-incompatible, so to make the transition smooth: - we'll provide a feature flag, scope-when-expressions-to-task, which: - will default to false to guard a Task and its dependent Tasks - can be set to true to guard a Task only - after migration, we'll change the global default for the feature flag to true to guard a Task only by default - eventually, we'll remove the feature flag and guard a Task only going forward Implements [TEP-0059: Skipping Strategies](https://github.com/tektoncd/community/blob/main/teps/0059-skipping-strategies.md) Closes tektoncd#2127
jerop · Jul 20, 2021 · 379ba8d · 379ba8d
1 parent 0e9d9e6
commit 379ba8d
Show file tree

Hide file tree

Showing 13 changed files with 636 additions and 53 deletions.
diff --git a/config/config-feature-flags.yaml b/config/config-feature-flags.yaml
@@ -82,3 +82,6 @@ data:
   # Setting this flag will determine which gated features are enabled.
   # Acceptable values are "stable" or "alpha".
   enable-api-fields: "stable"
+  # Setting this flag to "true" scopes WhenExpressions to guard a Task only
+  # instead of a Task and its dependent Tasks.
+  scope-when-expressions-to-task: "false"
diff --git a/docs/deprecations.md b/docs/deprecations.md
@@ -25,3 +25,5 @@ being deprecated.
 | [`Conditions` CRD is deprecated and will be removed. Use `WhenExpressions` instead.](https://github.com/tektoncd/community/blob/main/teps/0007-conditions-beta.md)       | [v0.16.0](https://github.com/tektoncd/pipeline/releases/tag/v0.16.0)         | Alpha                                                                                                     | Nov 02 2020                         |
 | [The `disable-home-env-overwrite` flag will be removed](https://github.com/tektoncd/pipeline/issues/2013)                                     | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0)         | Beta                                                                                                     | February 10 2022                    |
 | [The `disable-working-dir-overwrite` flag will be removed](https://github.com/tektoncd/pipeline/issues/1836)                                  | [v0.24.0](https://github.com/tektoncd/pipeline/releases/tag/v0.24.0)         | Beta                                                                                                     | February 10 2022                    |
+| [The `scope-when-expressions-to-task` flag will be flipped from `false` to `true`](https://github.com/tektoncd/pipeline/issues/1836)          | [v0.26.0](https://github.com/tektoncd/pipeline/releases/tag/v0.26.0)         | Beta                                                                                                     | February 10 2022                    |
+| [The `scope-when-expressions-to-task` flag will be removed](https://github.com/tektoncd/pipeline/issues/1836)                                 | [v0.26.0](https://github.com/tektoncd/pipeline/releases/tag/v0.26.0)         | Beta                                                                                                     | February 10 2022                    |
diff --git a/docs/install.md b/docs/install.md
@@ -366,6 +366,10 @@ use of custom tasks in pipelines.
 most stable features to be used. Set it to "alpha" to allow alpha
 features to be used.
 
+- `scope-when-expressions-to-task`: set this flag to "true" to scope `when` expressions to guard a `Task` only. Set it
+  to "false" to guard a `Task` and its dependent `Tasks`. It defaults to to "false". For more information, see [guarding
+  `Task` execution using `when` expressions](pipelines.md#guard-task-execution-using-whenexpressions).
+
 For example:
 
 ```yaml

diff --git a/docs/pipelines.md b/docs/pipelines.md
@@ -465,6 +465,133 @@ There are a lot of scenarios where `WhenExpressions` can be really useful. Some
 - Checking if the name of a CI job matches
 - Checking if an optional Workspace has been provided
 
+#### Guarding a `Task` and its dependent `Tasks`
+
+When  `when` expressions evaluate to `False`, the `Task` and its dependent `Tasks` will be skipped by default while the 
+rest of the `Pipeline` will execute. Dependencies between `Tasks` can be either ordering ([`runAfter`](https://github.com/tektoncd/pipeline/blob/main/docs/pipelines.md#using-the-runafter-parameter))
+or resource (e.g. [`Results`](https://github.com/tektoncd/pipeline/blob/main/docs/pipelines.md#using-results)) 
+dependencies, as further described in [configuring execution order](#configuring-the-task-execution-order). The global 
+default scope of `when` expressions is set to a `Task` and its dependent`Tasks`; `scope-when-expressions-to-task` field 
+in [`config/config-feature-flags.yaml`](install.md#customizing-the-pipelines-controller-behavior) defaults to "false".
+
+**Note:** Scoping `when` expressions to a `Task` and its dependent `Tasks` is deprecated. To guard a `Task` and its 
+dependent `Tasks`, cascade `when` expressions to the specific dependent `Tasks` to be guarded as well.
+
+```
+                                     tests
+                                       |
+                                       v
+                                 manual-approval
+                                 |            |
+                                 v        (approver)
+                            build-image       |
+                                |             v
+                                v          slack-msg
+                            deploy-image
+```
+
+Taking the use case above, a user who wants to guard `manual-approval` and its dependent `Tasks` can design the 
+`Pipeline` as such:
+
+```yaml
+tasks:
+...
+- name: manual-approval
+  runAfter:
+    - integration-tests
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  taskRef:
+    name: manual-approval
+
+- name: slack-msg
+  params:
+    - name: approver
+      value: $(tasks.manual-approval.results.approver)
+  taskRef:
+    name: slack-msg
+
+- name: build-image
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  runAfter:
+    - manual-approval
+  taskRef:
+    name: build-image
+
+- name: deploy-image
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  runAfter:
+    - build-image
+  taskRef:
+    name: deploy-image
+```  
+
+#### Guarding a `Task` only
+
+To guard a `Task` only and unblock execution of its dependent `Tasks`, set the global default scope of `when` expressions
+to `Task` using the `scope-when-expressions-to-task` field in [`config/config-feature-flags.yaml`](install.md#customizing-the-pipelines-controller-behavior)
+by changing it to "true".
+
+```
+                                     tests
+                                       |
+                                       v
+                                 manual-approval
+                                 |            |
+                                 v        (approver)
+                            build-image       |
+                                |             v
+                                v          slack-msg
+                            deploy-image
+```
+
+Taking the use case above, a user who wants to guard `manual-approval` only can design the `Pipeline` as such:
+
+```yaml
+tasks:
+...
+- name: manual-approval
+  runAfter:
+    - tests
+  when:
+    - input: $(params.git-action)
+      operator: in
+      values:
+        - merge
+  taskRef:
+    name: manual-approval
+
+- name: slack-msg
+  params:
+    - name: approver
+      value: $(tasks.manual-approval.results.approver)
+  taskRef:
+    name: slack-msg
+
+- name: build-image
+  runAfter:
+    - manual-approval
+  taskRef:
+    name: build-image
+
+- name: deploy-image
+  runAfter:
+    - build-image
+  taskRef:
+    name: deploy-image
+```
+
 ### Guard `Task` execution using `Conditions`
 
 **Note:** `Conditions` are [deprecated](./deprecations.md), use [`WhenExpressions`](#guard-task-execution-using-whenexpressions) instead.
@@ -691,10 +818,13 @@ so that one will run before another and the execution of the `Pipeline` progress
 without getting stuck in an infinite loop.
 
 This is done using:
-
-- [`from`](#using-the-from-parameter) clauses on the [`PipelineResources`](resources.md) used by each `Task`
-- [`runAfter`](#using-the-runafter-parameter) clauses on the corresponding `Tasks`
-- By linking the [`results`](#configuring-execution-results-at-the-pipeline-level) of one `Task` to the params of another
+- _resource dependencies_:
+  - [`from`](#using-the-from-parameter) clauses on the [`PipelineResources`](resources.md) used by each `Task`
+  - [`results`](#configuring-execution-results-at-the-pipeline-level) of one `Task` being pa `params` or
+    `when` expressions of another
+
+- _ordering dependencies_:
+  - [`runAfter`](#using-the-runafter-parameter) clauses on the corresponding `Tasks`
 
 For example, the `Pipeline` defined as follows
 

diff --git a/pkg/apis/config/feature_flags.go b/pkg/apis/config/feature_flags.go
@@ -37,6 +37,7 @@ const (
 	enableTektonOCIBundles                  = "enable-tekton-oci-bundles"
 	enableCustomTasks                       = "enable-custom-tasks"
 	enableAPIFields                         = "enable-api-fields"
+	scopeWhenExpressionsToTask              = "scope-when-expressions-to-task"
 	DefaultDisableHomeEnvOverwrite          = true
 	DefaultDisableWorkingDirOverwrite       = true
 	DefaultDisableAffinityAssistant         = false
@@ -45,6 +46,7 @@ const (
 	DefaultRequireGitSSHSecretKnownHosts    = false
 	DefaultEnableTektonOciBundles           = false
 	DefaultEnableCustomTasks                = false
+	DefaultScopeWhenExpressionsToTask       = false
 	DefaultEnableAPIFields                  = StableAPIFields
 )
 
@@ -59,6 +61,7 @@ type FeatureFlags struct {
 	RequireGitSSHSecretKnownHosts    bool
 	EnableTektonOCIBundles           bool
 	EnableCustomTasks                bool
+	ScopeWhenExpressionsToTask       bool
 	EnableAPIFields                  string
 }
 
@@ -105,6 +108,9 @@ func NewFeatureFlagsFromMap(cfgMap map[string]string) (*FeatureFlags, error) {
 	if err := setFeature(requireGitSSHSecretKnownHostsKey, DefaultRequireGitSSHSecretKnownHosts, &tc.RequireGitSSHSecretKnownHosts); err != nil {
 		return nil, err
 	}
+	if err := setFeature(scopeWhenExpressionsToTask, DefaultScopeWhenExpressionsToTask, &tc.ScopeWhenExpressionsToTask); err != nil {
+		return nil, err
+	}
 	if err := setEnabledAPIFields(cfgMap, DefaultEnableAPIFields, &tc.EnableAPIFields); err != nil {
 		return nil, err
 	}

diff --git a/pkg/apis/config/feature_flags_test.go b/pkg/apis/config/feature_flags_test.go
@@ -38,6 +38,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				DisableHomeEnvOverwrite:          false,
 				DisableWorkingDirOverwrite:       false,
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
+				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 				EnableAPIFields:                  "stable",
 			},
 			fileName: config.GetFeatureFlagsConfigName(),
@@ -51,6 +52,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				RequireGitSSHSecretKnownHosts:    true,
 				EnableTektonOCIBundles:           true,
 				EnableCustomTasks:                true,
+				ScopeWhenExpressionsToTask:       true,
 				EnableAPIFields:                  "alpha",
 			},
 			fileName: "feature-flags-all-flags-set",
@@ -66,6 +68,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				DisableHomeEnvOverwrite:          true,
 				DisableWorkingDirOverwrite:       true,
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
+				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 			},
 			fileName: "feature-flags-enable-api-fields-overrides-bundles-and-custom-tasks",
 		},
@@ -78,6 +81,7 @@ func TestNewFeatureFlagsFromConfigMap(t *testing.T) {
 				DisableHomeEnvOverwrite:          true,
 				DisableWorkingDirOverwrite:       true,
 				RunningInEnvWithInjectedSidecars: config.DefaultRunningInEnvWithInjectedSidecars,
+				ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 			},
 			fileName: "feature-flags-bundles-and-custom-tasks",
 		},
@@ -98,6 +102,7 @@ func TestNewFeatureFlagsFromEmptyConfigMap(t *testing.T) {
 		DisableHomeEnvOverwrite:          true,
 		DisableWorkingDirOverwrite:       true,
 		RunningInEnvWithInjectedSidecars: true,
+		ScopeWhenExpressionsToTask:       config.DefaultScopeWhenExpressionsToTask,
 		EnableAPIFields:                  "stable",
 	}
 	verifyConfigFileWithExpectedFeatureFlagsConfig(t, FeatureFlagsConfigEmptyName, expectedConfig)
@@ -141,6 +146,8 @@ func TestNewFeatureFlagsConfigMapErrors(t *testing.T) {
 		fileName: "feature-flags-invalid-boolean",
 	}, {
 		fileName: "feature-flags-invalid-enable-api-fields",
+	}, {
+		fileName: "feature-flags-invalid-scope-when-expressions-to-task",
 	}} {
 		t.Run(tc.fileName, func(t *testing.T) {
 			cm := test.ConfigMapFromTestFile(t, tc.fileName)

diff --git a/pkg/apis/config/testdata/feature-flags-all-flags-set.yaml b/pkg/apis/config/testdata/feature-flags-all-flags-set.yaml
@@ -25,4 +25,5 @@ data:
   require-git-ssh-secret-known-hosts: "true"
   enable-tekton-oci-bundles: "true"
   enable-custom-tasks: "true"
+  scope-when-expressions-to-task: "true"
   enable-api-fields: "alpha"
diff --git a/pkg/apis/config/testdata/feature-flags-invalid-scope-when-expressions-to-task.yaml b/pkg/apis/config/testdata/feature-flags-invalid-scope-when-expressions-to-task.yaml
@@ -0,0 +1,21 @@
+# Copyright 2021 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: feature-flags
+  namespace: tekton-pipelines
+data:
+  scope-when-expressions-to-task: "im-not-a-boolean"
diff --git a/pkg/reconciler/pipelinerun/pipelinerun.go b/pkg/reconciler/pipelinerun/pipelinerun.go
@@ -493,10 +493,11 @@ func (c *Reconciler) reconcile(ctx context.Context, pr *v1beta1.PipelineRun, get
 	// Build PipelineRunFacts with a list of resolved pipeline tasks,
 	// dag tasks graph and final tasks graph
 	pipelineRunFacts := &resources.PipelineRunFacts{
-		State:           pipelineRunState,
-		SpecStatus:      pr.Spec.Status,
-		TasksGraph:      d,
-		FinalTasksGraph: dfinally,
+		State:                      pipelineRunState,
+		SpecStatus:                 pr.Spec.Status,
+		TasksGraph:                 d,
+		FinalTasksGraph:            dfinally,
+		ScopeWhenExpressionsToTask: config.FromContextOrDefaults(ctx).FeatureFlags.ScopeWhenExpressionsToTask,
 	}
 
 	for _, rprt := range pipelineRunFacts.State {
@@ -647,7 +648,7 @@ func (c *Reconciler) runNextSchedulableTask(ctx context.Context, pr *v1beta1.Pip
 	}
 
 	for _, rprt := range nextRprts {
-		if rprt == nil || rprt.Skip(pipelineRunFacts) || rprt.IsFinallySkipped(pipelineRunFacts) {
+		if rprt == nil || rprt.Skip(pipelineRunFacts).IsSkipped || rprt.IsFinallySkipped(pipelineRunFacts).IsSkipped {
 			continue
 		}
 		if rprt.ResolvedConditionChecks == nil || rprt.ResolvedConditionChecks.IsSuccess() {