TEP-0083: Scheduled and Polling runs in Tekton #517

sm43 · 2021-09-13T07:00:41Z

This TEP introduces an idea for a feature in triggers which would allows user to

Schedule a pipelinerun/taskrun at a certain time/ at certain interval
Setup a poll which looks for changes on a repository and triggers pipelinerun/taskrun.

Signed-off-by: Shivam Mukhade [email protected]

afrittoli · 2021-09-13T16:19:10Z

/assign @sbwsg
/assign @savitaashture

savitaashture

This is a cool feature which adds more usecases to Triggers
Thank you @sm43 @vdemeester

teps/0083-scheduled-pipelinerun--tekton-polling.md

savitaashture · 2021-09-14T06:04:57Z

teps/0083-scheduled-pipelinerun--tekton-polling.md

+  template: pipeline-template
+```
+
+- This takes GitHub Repo URL and then check for changes at frequeny defined by user.


I see right now it supports only Github
Is there a plan to support different scm like bitbucket, gitlab etc...?
If yes then can we add somewhere in TEP the supported and non supported one?

The repo happens to be GitHub, but there is nothing specific to GitHub. The PoC is limited to git though, but we could rework a bit the structure to allow more support for other VCS (like mercurial, …)

savitaashture · 2021-09-14T06:07:20Z

teps/0083-scheduled-pipelinerun--tekton-polling.md

+-->
+
+## Design Details
+


It would be nice if we have design details mentioning some points like

what are the optional and required field in SyncRepo CRD

does both polling and scheduled supported together

If yes then describing which attributes for which functionality(polling, scheduling)

This initial TEP is mainly on the problem statement. the SyncRepo PoC is only based on the poll part of this TEP, but digging more, we might rework that CRD altogether, hence not getting in too much detail too quickly 😛

teps/0083-scheduled-pipelinerun--tekton-polling.md

ghost · 2021-09-14T16:40:50Z

Thanks for opening the TEP, I understand the end goal but here are the questions I've come away with after reading:

We identify CronJobs as an existing solution for scheduling. Why aren't they good enough on their own?
We identify an alternative project that exists today for polling GitHub. Is there a strong reason to favor a new solution and if so what is it? Is there a strong reason to favor a project owned by Tekton and if so what is it?

It would be great to get a bit more detail in the problem statement covering these if possible.

vdemeester · 2021-09-14T16:43:03Z

Thanks for opening the TEP, I understand the end goal but here are the questions I've come away with after reading:

We identify CronJobs as an existing solution for scheduling. Why aren't they good enough on their own?

We identify an alternative project that exists today for polling GitHub. Is there a strong reason to favor a new solution and if so what is it? Is there a strong reason to favor a project owned by Tekton and if so what is it?

It would be great to get a bit more detail in the problem statement covering these if possible.

Yes I think we need to answer those 2 questions in the TEP indeed 👼🏼

afrittoli · 2021-09-20T15:25:57Z

Nice idea, thank you!
This reminds me of the concept of event source, commonly used in serverless architectures.

A couple of examples:

It seems to me that both cron and polling a git repo could be seen as event sources.

One option is to have the triggers controller able to reconcile an event source CRD.
Another solution could be to have a generic CRD - and have dedicated controllers, one for each kind of event source.
Those controllers would be responsible for setting up a dedicated trigger CRD and to send events to it over HTTP.
This would be similar approach I was thinking of using for the implementation of notifications in Tekton.

pritidesai · 2021-09-20T16:10:39Z

/assign @afrittoli

dibyom · 2021-09-21T15:43:33Z

/assign

afrittoli · 2021-09-23T15:53:30Z

Nice idea, thank you!
This reminds me of the concept of event source, commonly used in serverless architectures.

A couple of examples:

https://knative.dev/docs/developer/eventing/sources/

https://argoproj.github.io/argo-events/

It seems to me that both cron and polling a git repo could be seen as event sources.

One option is to have the triggers controller able to reconcile an event source CRD.
Another solution could be to have a generic CRD - and have dedicated controllers, one for each kind of event source.
Those controllers would be responsible for setting up a dedicated trigger CRD and to send events to it over HTTP.
This would be similar approach I was thinking of using for the implementation of notifications in Tekton.

My comments above ^^^ are more relevant for the design discussion - I agree with the goals of being able to schedule runs and polling to avoid exposing a public endpoint.

dibyom · 2021-09-23T18:31:39Z

Problem statement/use cases sounds good!

Also agree with @afrittoli 's comment that there are a lot of projects in this space and we should consider how we can integrate with them or use them seamlessly vs inventing something of our own. (But's that a discussion for the design phase!)

/approve

tekton-robot · 2021-09-23T18:31:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dibyom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~teps/OWNERS~~ [dibyom]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sm43 · 2021-09-27T16:22:10Z

Thanks for opening the TEP, I understand the end goal but here are the questions I've come away with after reading:
* We identify CronJobs as an existing solution for scheduling. Why aren't they good enough on their own?

* We identify an alternative project that exists today for polling GitHub. Is there a strong reason to favor a new solution and if so what is it? Is there a strong reason to favor a project _owned by Tekton_ and if so what is it?
It would be great to get a bit more detail in the problem statement covering these if possible.

@sbwsg I have addressed the question in the TEP. ptal :)
cc @dibyom

ghost · 2021-09-27T16:52:36Z

Thanks for providing responses to those questions. At this point you need an approve from a non-Google reviewer.

ghost · 2021-10-12T20:06:53Z

teps/0083-scheduled-pipelinerun--tekton-polling.md

+
+```
+apiVersion: triggers.tekton.dev/v1alpha1
+kind: SyncRepo


@sm43 @dibyom @chmouel It looks like something similar to a Repo CRD is a shared problem across this TEP, the Workflows project, pipelines as code, remote resource resolution

Could we potentially extract a common element here and deliver that on its own? I could see a few ways this could help. It might just be common vcs configuration (a name, the type of VCS, and the repo's URL?) or it could extend all the way to supporting polling, content fetching, content caching, rbac, etc etc with one or multiple controllers watching for RepoCRD objects and layering functionality on top of them.

Not sure, what do folks think?

Yeah I think this is definitely an idea worth exploring!!

Great, I will put together some ideas for discussion at the next Workflows WG 👍

afrittoli · 2021-10-18T11:22:53Z

teps/0083-scheduled-pipelinerun--tekton-polling.md

+- (sbwsg) We identify an alternative project that exists today for polling GitHub. Is there a strong reason to favor a new solution 
+  and if so what is it? Is there a strong reason to favor a project owned by Tekton and if so what is it?


#517 (comment)

afrittoli · 2021-10-18T11:26:38Z

teps/0083-scheduled-pipelinerun--tekton-polling.md

+- To allow users to schedule a pipelinerun/taskrun at a certain time or at certain interval
+
+    Ex. I want to run a pipeline every day at 8 am. Currently, I can do this by setting up a cronjob, but having as a part of Trigger could be a good idea. So, Triggers can start a pipelinerun at the time mentioned by me.
+
+- To allow users to use triggers without need to setup a webhook. Triggers could have a feature which allow user to setup a polling feature for a repository which would look for changes in repository and trigger a pipelinerun or taskrun.


It would be nice to have a few words explaining why the two features are proposed together.
I assume the reason is that their implementation may rely on a shared new feature? If so should the TEP only be about that base feature and scheduled pipeline / polling be added as use cases?

Right, that might make sense.

afrittoli · 2021-10-18T11:43:54Z

teps/0083-scheduled-pipelinerun--tekton-polling.md

+    Cronjob do cover a use case which we are proposing which is triggering run at certain interval but to check it
+    something is actually changed to trigger a pipelinrun/taskrun then we will need a script to check the condition.
+    Everytime we setup this for a repository, the user will have to write a script. This logic can be abstracted into 
+    triggers and an interface can be exposed to user which would be simple to configure.


It would be good to capture the requirement of doing "conditional" scheduled runs in the summary or use cases or requirements. Abstracting the filtering logic to a Trigger can be done when using CronJobs too - the CronJob can send an event to an EventListener that can then decide - through the triggers - whether to run a pipeline or not. Recently added TriggerGroups also may help in making this setup easier to maintain.

The main downside for me in using CronJobs is that they are not well integrated with Tekton and it requires an event listener which could be avoided for an "internal" trigger. The HTTP POST needs to be scripted in the CronJob and the target URL needs to be derived from the event listener.

I would prefer an approach where we could define a Cron trigger more declaratively - see TriggerRuns and tektoncd/triggers#504 for further context.

bobcatfish · 2021-10-25T16:10:28Z

Comments from API WG today: need to explore the alternatives a bit more in the proposal; in some setups you may not need explicit support for this (e.g. if you have knative)

teps/0083-scheduled-pipelinerun--tekton-polling.md

vdemeester · 2021-10-25T15:22:02Z

teps/0083-scheduled-pipelinerun--tekton-polling.md

+- To allow users to schedule a pipelinerun/taskrun at a certain time or at certain interval
+
+    Ex. I want to run a pipeline every day at 8 am. Currently, I can do this by setting up a cronjob, but having as a part of Trigger could be a good idea. So, Triggers can start a pipelinerun at the time mentioned by me.
+
+- To allow users to use triggers without need to setup a webhook. Triggers could have a feature which allow user to setup a polling feature for a repository which would look for changes in repository and trigger a pipelinerun or taskrun.


Right, that might make sense.

sm43 · 2021-10-27T16:32:11Z

Comments from API WG today: need to explore the alternatives a bit more in the proposal; in some setups you may not need explicit support for this (e.g. if you have knative)

@bobcatfish we created this PR to introduce the idea and use cases basically. we are exploring possible solutions and suggestion on the pr 😅
shall we merge this pr and follow up in a new with possible solutions and alternatives?

This TEP introduces an idea for a feature in triggers which would allows user to - Schedule a pipelinerun/taskrun at a certain time/ at certain interval - Setup a poll which looks for changes on a repository and triggers pipelinerun/taskrun. Signed-off-by: Shivam Mukhade <[email protected]>

pritidesai · 2021-11-01T16:08:34Z

API WG - waiting for more feedback ...

dibyom · 2021-11-09T18:52:44Z

@savitaashture @afrittoli (also @khrm and @vdemeester ) - This problem statement still needs a non Google approver

khrm · 2021-11-17T16:04:12Z

/lgtm

tekton-robot requested review from pradeepitm12 and vdemeester September 13, 2021 07:00

tekton-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Sep 13, 2021

vdemeester added the kind/tep Categorizes issue or PR as related to a TEP (or needs a TEP). label Sep 13, 2021

tekton-robot assigned savitaashture and ghost Sep 13, 2021

savitaashture reviewed Sep 14, 2021

View reviewed changes

vdemeester reviewed Sep 14, 2021

View reviewed changes

teps/0083-scheduled-pipelinerun--tekton-polling.md Outdated Show resolved Hide resolved

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 17, 2021

sm43 force-pushed the triggers-polling branch from 11514ba to fd7a3b3 Compare September 20, 2021 08:19

tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 20, 2021

tekton-robot assigned afrittoli Sep 20, 2021

tekton-robot assigned dibyom Sep 21, 2021

tekton-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Sep 23, 2021

sm43 force-pushed the triggers-polling branch from fd7a3b3 to 380600a Compare September 24, 2021 12:55

tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 24, 2021

sm43 force-pushed the triggers-polling branch from beb7617 to 4b769e0 Compare September 24, 2021 13:00

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 5, 2021

ghost reviewed Oct 12, 2021

View reviewed changes

afrittoli reviewed Oct 18, 2021

View reviewed changes

vdemeester reviewed Oct 26, 2021

View reviewed changes

sm43 force-pushed the triggers-polling branch from 4b769e0 to f29f16c Compare October 26, 2021 06:38

tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 26, 2021

sm43 changed the title ~~TEP-0083: Scheduled Pipelinerun & Tekton Polling~~ TEP-0083: Scheduled and Polling runs in Tekton Oct 26, 2021

tekton-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2021

sm43 force-pushed the triggers-polling branch from f29f16c to 7e8ce8d Compare October 29, 2021 04:32

tekton-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 29, 2021

ghost mentioned this pull request Nov 1, 2021

TEP-0060: Update proposal and mark Remote Resolution implementable #547

Merged

tekton-robot assigned khrm Nov 17, 2021

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 17, 2021

tekton-robot merged commit 6810d57 into tektoncd:main Nov 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEP-0083: Scheduled and Polling runs in Tekton #517

TEP-0083: Scheduled and Polling runs in Tekton #517

sm43 commented Sep 13, 2021 •

edited

Loading

afrittoli commented Sep 13, 2021

savitaashture left a comment

savitaashture Sep 14, 2021

vdemeester Sep 14, 2021

savitaashture Sep 14, 2021

vdemeester Sep 14, 2021

ghost commented Sep 14, 2021

vdemeester commented Sep 14, 2021

afrittoli commented Sep 20, 2021

pritidesai commented Sep 20, 2021

dibyom commented Sep 21, 2021

afrittoli commented Sep 23, 2021

dibyom commented Sep 23, 2021

tekton-robot commented Sep 23, 2021

sm43 commented Sep 27, 2021

ghost commented Sep 27, 2021

ghost Oct 12, 2021

dibyom Oct 13, 2021

ghost Oct 15, 2021

afrittoli Oct 18, 2021

sm43 Oct 26, 2021

afrittoli Oct 18, 2021

vdemeester Oct 25, 2021

afrittoli Oct 18, 2021

bobcatfish commented Oct 25, 2021

vdemeester Oct 25, 2021

sm43 commented Oct 27, 2021

pritidesai commented Nov 1, 2021

dibyom commented Nov 9, 2021

khrm commented Nov 17, 2021

		- (sbwsg) We identify an alternative project that exists today for polling GitHub. Is there a strong reason to favor a new solution
		and if so what is it? Is there a strong reason to favor a project owned by Tekton and if so what is it?

TEP-0083: Scheduled and Polling runs in Tekton #517

TEP-0083: Scheduled and Polling runs in Tekton #517

Conversation

sm43 commented Sep 13, 2021 • edited Loading

afrittoli commented Sep 13, 2021

savitaashture left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Sep 14, 2021

vdemeester commented Sep 14, 2021

afrittoli commented Sep 20, 2021

pritidesai commented Sep 20, 2021

dibyom commented Sep 21, 2021

afrittoli commented Sep 23, 2021

dibyom commented Sep 23, 2021

tekton-robot commented Sep 23, 2021

sm43 commented Sep 27, 2021

ghost commented Sep 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobcatfish commented Oct 25, 2021

Choose a reason for hiding this comment

sm43 commented Oct 27, 2021

pritidesai commented Nov 1, 2021

dibyom commented Nov 9, 2021

khrm commented Nov 17, 2021

sm43 commented Sep 13, 2021 •

edited

Loading