Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TriggerTemplate does not create PVC #476

Closed
disposab1e opened this issue Mar 10, 2020 · 7 comments
Closed

TriggerTemplate does not create PVC #476

disposab1e opened this issue Mar 10, 2020 · 7 comments

Comments

@disposab1e
Copy link

Expected Behavior

PVC is created.

Actual Behavior

{"level":"error","logger":"eventlistener","caller":"sink/sink.go:153","msg":"couldn't create resource with group version kind \"/v1, Resource=persistentvolumeclaims\": resource /v1, Resource=persistentvolumeclaims not supported","knative.dev/controller":"eventlistener","/triggers-eventid":"lvgvl","/trigger":"pipeline-trigger","stacktrace":"github.com/tektoncd/triggers/pkg/sink.Sink.processTrigger\n\tgithub.com/tektoncd/triggers/pkg/sink/sink.go:153\ngithub.com/tektoncd/triggers/pkg/sink.Sink.HandleEvent.func1\n\tgithub.com/tektoncd/triggers/pkg/sink/sink.go:95"}

Steps to Reproduce the Problem

  1. Create a TriggerTemplae with PVC.
....
 resourcetemplates:
 - apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: $(uid)
   spec:
     resources:
       requests:
         storage: 5Gi
     accessModes:
       - ReadWriteMany
 - apiVersion: tekton.dev/v1alpha1
   kind: PipelineRun
   metadata:
     generateName: pipeline-run-
     annotations:
       argocd.argoproj.io/compare-options: IgnoreExtraneous            
   spec:
     ....
  1. Trigger TriggerTemplate.
  2. View erros in EventListener.

Additional Info

From the docs: TriggerTemplates can create any Kubernetes resource(s)

@disposab1e
Copy link
Author

I see: As on now, only Tekton resources can be defined within a TriggerTemplate.

As in Tekton Pipelines it's a good idea to make the docs more clear!

@ncskier
Copy link
Member

ncskier commented Mar 12, 2020

Thanks for your feedback @disposab1e. I created a PullRequest (#477) to address your issue over the TriggerTemplate documentation. Please let me know if you think my PullRequest will adequately solve your issue for new users who read through the Triggers documentation.

@ncskier
Copy link
Member

ncskier commented Mar 12, 2020

As a second note, @disposab1e, would you find it very helpful if Tekton Triggers was able to create Kubernetes resources other than Tekton resources? Are you unable to use Tekton Triggers without this feature?

@disposab1e
Copy link
Author

disposab1e commented Mar 12, 2020

@ncskier Yes I think this can be very helpful?

We use for GitOps in OpenShift 3.11/Kubernetes 1.14 with:

  • Argo CD 1.4.2 (Operator)
  • Tekton Pipeline (v0.10.1)
  • Tekton Triggers (v0.3.1)
  • SelaedSecrets (v0.9.6)

Actually we use a single EventListener, TriggerBinding and TriggerTemplate for all off our CI/CD Pipelines serving many differnet projects in many different namespaces. All of our custom Tekton Pileline/Trigger Kubernetes Resources are in a single namespace.

apiVersion: tekton.dev/v1alpha1
kind: EventListener
metadata:
  name: pipeline-el
  namespace: awesome-pipelines
spec:
  serviceAccountName: pipeline-sa
  triggers:
    - name: pipeline-trigger
      bindings:
        - name: pipeline-tb
      template:
        name: pipeline-tt
---
apiVersion: tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
  name: pipeline-tb
  namespace: awesome-pipelines
spec:
  params:
  - name: gitRevision
    value: $(body.git.revision)
  - name: gitUrl
    value: $(body.git.url)
  - name: pipelineName
    value: $(body.pipeline.name)
  - name: params
    value: $(body.pipeline.params)
---
apiVersion: tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: pipeline-tt
  namespace: awesome-pipelines
spec:
  params:
  - name: pipelineName
    description: Pipeline Name (build, deploy, verify)    
 resourcetemplates:
  - apiVersion: tekton.dev/v1alpha1
    kind: PipelineRun
    metadata:
      generateName: pipeline-run-
    spec:
      serviceAccountName: pipeline-sa
      pipelineRef:
        name: pipeline-$(params.pipelineName)
      params:
      - name: uid
        value: $(uid)
      resources:
      - name: git-source
        resourceSpec:
          type: git
          params:
          - name: revision
            value: $(params.gitRevision)
          - name: url
            value: $(params.gitUrl)
      workspaces:
      - name: pipeline-workspace
        persistentVolumeClaim:
          claimName: `pipeline-workspace`
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pipeline-workspace
spec:
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteMany

We share a single PVC (big enough) between all of our PipelineRuns. A PipelineRun is triggered through a Webhook/EventListener with JSON payload.

To share this PVC we use the $(uid) from TriggerTemplate and dynamically create a directory inside the PVC as a minimum of isolation :-)
So we simple create a directory $(uid) inside the PVC at start and remove it at the end of our Pipelines.
To be honest it's not a nice solution... but it's a GitOps solution!

The same problem exists after migrating PipelineResources to Tasks and workspaces.

So we would like to have a possibilty to dynamically create a PVC with the TriggerTemplate.

apiVersion: tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
 name: pipeline-tt
 namespace: awesome-pipelines
spec:
 params:
 - name: pipelineName
   description: Pipeline Name (build, deploy, verify)    
resourcetemplates:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: $(uid)
  spec:
    resources:
      requests:
        storage: 0.2Gi
    accessModes:
      - ReadWriteMany
 - apiVersion: tekton.dev/v1alpha1
   kind: PipelineRun
   metadata:
     generateName: pipeline-run-
   spec:
     serviceAccountName: pipeline-sa
     pipelineRef:
       name: pipeline-$(params.pipelineName)
     workspaces:
     - name: pipeline-workspace
       persistentVolumeClaim:
         claimName: `pipeline-workspace`

What I really miss is Error Handling tektoncd/pipeline#1684 and Lifecycle Management tektoncd/pipeline#1460

I'm also aware of the Auto Workspaces discussion: tektoncd/pipeline#1986

I don't think this will break any design decissions but can be very helpfull in some dynamic scenarios.

@ncskier
Copy link
Member

ncskier commented Mar 13, 2020

@disposab1e Thank you for elaborating on your use-case. If you have the time, could you please turn your last comment, with your use-case, into a new Triggers issue? We'll probably want to add a feature for removing the TriggerTemplate limitation of only creating Tekton resources. I think that it will be helpful to discuss this feature with the community in a new issue 👍

(I can create the issue if you don't have time, but the issue will probably be more accurate/effective coming from you)

@disposab1e
Copy link
Author

@ncskier Pleasure! #482

@ncskier
Copy link
Member

ncskier commented Mar 16, 2020

Thanks!

jlpettersson added a commit to jlpettersson/pipeline that referenced this issue Apr 4, 2020
An existing PersistentVolumeClaim can currently be used as a Workspace
volume source. There is two ways of using an existing PVC as volume:

 - Reuse an existing PVC
 - Create a new PVC before each PipelineRun.

There is disadvantages by reusing the same PVC for every PipelineRun:

 - You need to clean the PVC at the end of the Pipeline
 - All Tasks using the workspace will be scheduled to the node where
   the PV is bound
 - Concurrent PipelineRuns may interfere, an artifact or file from one
   PipelineRun may slip in to or be used in another PipelineRun, with
   very few audit tracks.

There is also disadvantages by creating a new PVC before each PipelineRun:

 - This can not (easily) be done declaratively
 - This is hard to do programmatically, because it is hard to know when
   to delete the PVC. The PipelineRun can not be set as OwnerReference since
   the PVC must be created first

 This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This
 has several advantages:

 - The syntax is used in k8s StatefulSet and other k8s projects so it is
   familiar in the kubernetes ecosystem
 - It is possible to declaratively declare that a PVC should be created for each
   PipelineRun, e.g. from a TriggerTemplate.
 - The user can choose storageClass (or omit to get the cluster default) to e.g.
   get a faster SSD volume, or to get a volume compatible with e.g. Windows.
 - The user can adapt the size to the job, e.g. use 5Gi for apps that contains
   machine learning models, or 1Gi for microservice apps. It can be changed on
   demand in a configuration that lives in the users namespace e.g. in a
   TriggerTemplate.
 - The size affects the storage quota that is set on the namespace and it may affect
   billing and cost depending on the cluster environment.
 - The PipelineRun or TaskRun with the template is created first, and is used
   as OwnerReference on the PVC. That means that the PVC will have the same lifecycle
   as the PipelineRun.

 Related to tektoncd#1986

 See also:
  - tektoncd#2174
  - tektoncd#2218
  - tektoncd/triggers#476
  - tektoncd/triggers#482
  - kubeflow/kfp-tekton#51
jlpettersson added a commit to jlpettersson/pipeline that referenced this issue Apr 9, 2020
An existing PersistentVolumeClaim can currently be used as a Workspace
volume source. There is two ways of using an existing PVC as volume:

 - Reuse an existing PVC
 - Create a new PVC before each PipelineRun.

There is disadvantages by reusing the same PVC for every PipelineRun:

 - You need to clean the PVC at the end of the Pipeline
 - All Tasks using the workspace will be scheduled to the node where
   the PV is bound
 - Concurrent PipelineRuns may interfere, an artifact or file from one
   PipelineRun may slip in to or be used in another PipelineRun, with
   very few audit tracks.

There is also disadvantages by creating a new PVC before each PipelineRun:

 - This can not (easily) be done declaratively
 - This is hard to do programmatically, because it is hard to know when
   to delete the PVC. The PipelineRun can not be set as OwnerReference since
   the PVC must be created first

 This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This
 has several advantages:

 - The syntax is used in k8s StatefulSet and other k8s projects so it is
   familiar in the kubernetes ecosystem
 - It is possible to declaratively declare that a PVC should be created for each
   PipelineRun, e.g. from a TriggerTemplate.
 - The user can choose storageClass (or omit to get the cluster default) to e.g.
   get a faster SSD volume, or to get a volume compatible with e.g. Windows.
 - The user can adapt the size to the job, e.g. use 5Gi for apps that contains
   machine learning models, or 1Gi for microservice apps. It can be changed on
   demand in a configuration that lives in the users namespace e.g. in a
   TriggerTemplate.
 - The size affects the storage quota that is set on the namespace and it may affect
   billing and cost depending on the cluster environment.
 - The PipelineRun or TaskRun with the template is created first, and is used
   as OwnerReference on the PVC. That means that the PVC will have the same lifecycle
   as the PipelineRun.

 Related to tektoncd#1986

 See also:
  - tektoncd#2174
  - tektoncd#2218
  - tektoncd/triggers#476
  - tektoncd/triggers#482
  - kubeflow/kfp-tekton#51
tekton-robot pushed a commit to tektoncd/pipeline that referenced this issue Apr 14, 2020
An existing PersistentVolumeClaim can currently be used as a Workspace
volume source. There is two ways of using an existing PVC as volume:

 - Reuse an existing PVC
 - Create a new PVC before each PipelineRun.

There is disadvantages by reusing the same PVC for every PipelineRun:

 - You need to clean the PVC at the end of the Pipeline
 - All Tasks using the workspace will be scheduled to the node where
   the PV is bound
 - Concurrent PipelineRuns may interfere, an artifact or file from one
   PipelineRun may slip in to or be used in another PipelineRun, with
   very few audit tracks.

There is also disadvantages by creating a new PVC before each PipelineRun:

 - This can not (easily) be done declaratively
 - This is hard to do programmatically, because it is hard to know when
   to delete the PVC. The PipelineRun can not be set as OwnerReference since
   the PVC must be created first

 This commit adds 'volumeClaimTemplate' as a volume source for workspaces. This
 has several advantages:

 - The syntax is used in k8s StatefulSet and other k8s projects so it is
   familiar in the kubernetes ecosystem
 - It is possible to declaratively declare that a PVC should be created for each
   PipelineRun, e.g. from a TriggerTemplate.
 - The user can choose storageClass (or omit to get the cluster default) to e.g.
   get a faster SSD volume, or to get a volume compatible with e.g. Windows.
 - The user can adapt the size to the job, e.g. use 5Gi for apps that contains
   machine learning models, or 1Gi for microservice apps. It can be changed on
   demand in a configuration that lives in the users namespace e.g. in a
   TriggerTemplate.
 - The size affects the storage quota that is set on the namespace and it may affect
   billing and cost depending on the cluster environment.
 - The PipelineRun or TaskRun with the template is created first, and is used
   as OwnerReference on the PVC. That means that the PVC will have the same lifecycle
   as the PipelineRun.

 Related to #1986

 See also:
  - #2174
  - #2218
  - tektoncd/triggers#476
  - tektoncd/triggers#482
  - kubeflow/kfp-tekton#51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants