Issue 15219 - Cloud Run Jobs Launcher

[baumann-t]

I've created a new Cloud Launcher, allowing to create, monitor and cancel Dagster runs on Google Cloud Run Jobs
(based on issue #15219 )

A Cloud Run Job configuration must be created in a GCP project, using a Docker image containing the user code needed to materialize the assets. The Cloud Run Job name must be specified in the instance configuration (formatted like projects/{project_id}/locations/{location}/jobs/{job_id})

I'm very new to Dagster, so I would appreciate any comments as well as guidance on the next steps for this Cloud Launcher (testing, example development...)

[baumann-t]

I’ve added two scripts that allow a user to :

1. deploy a simple GCE Instance hosting the Dagster Webserver and Deamon
2. deploy a new cloud run job configuration, allowing to launch Dagster runs on Cloud Run

To deploy the example:

• Postgres Instance:
  • Create a Postgres instance on GCP, with a public ip, a new database, a user and password
  • Set Postgres DB secrets in GCP secret manager (one secret per variable: DAGSTER_PG_HOST, DAGSTER_PG_USERNAME, DAGSTER_PG_PASSWORD, DAGSTER_PG_DB)
  • Allow network connections from the vm to the DB instance
• VM:
  • Create a service account for the VM with the rights to launch a cloud run job and to access secrets from secret manager
  • Set the variables in thedeploy_vm.shscript for your GCP PROJECT_ID and SERVICE_ACCOUNT_EMAIL
  • Create a firewall rule allowing incoming traffic for port 3000 on your network
• Cloud Run:
  • Activate the Cloud Run api in your GCP project
  • Your Cloud run default service account must have secret accessor permissions (also possible to add your own service account)
  • Set PROJECT_ID and REGION variable in deploy_cloud_run_job.sh
• dagster.yaml
  • Change YOUR_PROJECT_ID placeholders for your own project id (id must be in numbers, not project name)
  • If you changed the name of your cloud run job, change it in cloud_run_job_name as well
• User code:
  • Edit code in defs.py if needed

Once all these changes have been applied:

• Ensure you are logged in to gcloud
• Run the script to deploy the new cloud run job : sh deploy_cloud_run_job.sh
• Once the script has finished running, run the script to deploy the vm: sh deploy_vm.sh

You should be able to access your Dagster webserver at your public VM url, port 3000.
If you launch a run from the web server, it should start a cloud run job.

Please let me know if you have any comments regarding the deployment process or the cloud run job launcher!

[tacastillo]

Thank you for contributing this run launcher. Apologies for the delay on reviewing this! We're working internally on who would be the domain expert to review this and will update you once it's figured out!

[baumann-t]

Closing this PR as some of this work will be used in: #21864