Task with more than one workspace waiting forever to start

[r0bj]

Expected Behavior

Task with more than one workspace starts.

Actual Behavior

Task with more than one workspace waiting forever to start because it cannot attach volumes.

Steps to Reproduce the Problem

1. There is a task (taskA) in pipeline which uses 2 workspaces
2. Pipelinerun starts, creates 2 different affinity-assistants
3. Because affinity-assistants use anti-affinity they are started on 2 different nodes
4. Each affinity-assistant mount volume on node where it runs, so volumes are mounted on different nodes
5. Task (taskA) cannot start because it uses affinity to one of the affinity-assistants and tries to attach 2 volumes which is impossible with ReadWriteOnce volumes
6. Task (taskA) waits forever to start

Is it possible to handle this situation with affinity-assistant enabled?

Additional Info

• Tekton Pipeline version:

Client version: 0.10.0
Pipeline version: v0.13.2
Triggers version: v0.5.0

[vdemeester]

/kind bug
/cc @jlpettersson

[jlpettersson]

Is it possible to handle this situation with affinity-assistant enabled?

No - by intention. When the Affinity Assistant was introduced, the docs was also updated so that we recommend at most one writeable workspace per Task. If you really need to use two PVC volumes in a single Task - you should disable the Affinity Assistant.

Each affinity-assistant mount volume on node where it runs, so volumes are mounted on different nodes

And this will happen if e.g. a pipeline starts with two parallel tasks - even without the Affinity Assistant.

In the worst case - these two volumes are located in two different Availability Zones - and your Pipeline is deadlocked - with or without the Affinity Assistant. For this reason I had to remove an end-to-end-test using two volumes - that sometimes ended up in to different Availability Zones in a Regional cluster: #2546

If you really want to use two volumes in a single Task - you must disable the Affinity Assistant, but you will must likely run into problems in regional clusters - but it depends on how your storage work.

There are two alternatives, either disable the Affinity Assistant (but parallelism in pipeline will be harder) or design Tasks so that they use at most one workspace.

[jlpettersson]

We should probably validate and emit an Event or use the Admission Webhook to validate these situation to inform the user in a better way.

[jlpettersson]

@r0bj is there any specific reason to why you need two workspaces in the same Task?

It is possible to have a single workspace volume but mount different parts of it using subPath, there is an example of that: https://github.com/tektoncd/pipeline/blob/master/examples/v1beta1/pipelineruns/pipelinerun-using-different-subpaths-of-workspace.yaml

Is that a way to solve your use case?

[r0bj]

@jlpettersson thanks for explanation. In my case using subPath resolves issue. Using two workspaces is not a strong requirement, just an option I was considering.

[ghost]

Now that we have #2885 merged, should we close this issue?

[vdemeester]

I would think so yes. If anyone feels it should be re-open, please comment 👼
/close

[tekton-robot]

@vdemeester: Closing this issue.

In response to this:

I would think so yes. If anyone feels it should be re-open, please comment 👼
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.