Runtime for Pipelines

[rabernat]

Right now, the pipelines are very coupled to their "runtime," by which I mean environment context that varies between location.

Example: https://github.com/pangeo-forge/terraclimate-feedstock/blob/master/recipe/pipeline.py

We want to refactor this so that the runtime is specified separately from the pipeline.

Things attached to the runtime:

• Prefect stuff:
  • execution environment (e.g. DaskKubernetesEnvironment, recipe/job.yaml, recipe/worker_pod.yaml)
  • flow storage (e.g. Docker)
  • flow registration (everything in the __main__)
• Data storage stuff:
  • cache location
• Credentials
  • docker
  • cloud storage
  • prefect token

What am I missing?

[rabernat]

Make we want to make the runtime a cli, like conda smithy.

pangeo smithy run pipeline.py