Enable recurring schedule of Notebooks and Pipelines

[lresende]

Initial requirements:

• Add support for both Airflow and KFP

  • ui
  • cli (low priority)

• Update submit dialog and add a field to specify the recurrence {once (default), hourly, daily, weekly, monthly, yearly }

  • This should be updated for all submission options (notebooks and scripts)
  • on the notebook side, the last schedule option should be persisted and shown as the new default on the next submission
    • The persisted value should be designed to allow for a more complex ui in the future, where you could specify other options from the cron format or what is supported from the backends

From Airflow:

For KFP, some UI examples:

Looks like Kubeflow has enabled a recurring schedule of experiments, and we could use to enable notebooks and pipelines to be scheduled

[omarsumadi]

Hey @lresende - it seems like we got really far on this #2371 and decided to de-prioritize. I'm very new to the project and was wondering what it would take to get this prioritized.

In the meantime, I will try to find some support to see if we can make this happen from a product side with my team of engineers if I can.

[D3f0]

For the KFP side, I've located the UI where we can define a cron schedule. I'm gonna take a look at the SDK/API now.

kfp_schedule.mp4