More Cloud RunLaunchers

[PascalStehling]

What's the use case?

At the moment the only RunLauncher for a Cloud Vendor is Amazon with the ECS RunLauncher. Are their Plans/Ideas to support other Cloud Vendors like Azure or GCP? The Goal would be to have the same feature Set, just for different Clouds.

Ideas of implementation

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[dpeng817]

I don't think it's on the radar, mostly because we haven't received a ton of asks for that - but it's certainly a reasonable ask.

[mkleinbort-ic]

I've wanted to run our pipelines in Azure for a while.
Our plan was to wait for you to implement it to avoid scrapping our interim solution once you build an Azure RunLauncher.

[PascalStehling]

Same here, an Azure RunLauncher would be much appreciated.

[gai6948]

GCP Cloud Run launcher would be awesome

[francocalvo]

Sorry to revive this, but yeah, a GCP launcher would be cool.
I'm looking to have a minimal VM for orchestration, but storage and computation handled elsewhere.

I'd like to use a 'serverless' approach, where the launcher would spin up a container, execute the job and spin it down afterwards.

Is there any recommended approach for this in GCP until there is a launcher?

[kujenga]

Just wanted to echo that adding a Cloud Run launcher for GCP would be great addition, it would make getting up and running in GCP significantly easier!

There was a bit of discussion on it here that made it sound not all that difficult: https://discuss.dagster.io/t/2644960/thanks-u016c4e5cp8-i-ll-take-a-look-through-things-if-you-kn

[francocalvo]

I have a ton of work currently, but as soon as it winds down and if this hasn't been done, I'll try to get it running.
Hope someone else gets in before me tho lol

[baumann-t]

I'm currently working on a launcher for GCP Cloud Run Job, I should have a working first version soon!

[baumann-t]

I've opened a draft (#21509) for the first version of a Cloud Run Jobs Launcher.
I would really appreciate any comments or guidance to help me complete this (testing, example code...) !

[baumann-t]

The pull request is now ready to be tested for the Cloud Run Job Launcher (#21509)
I've added example code on how to deploy the webserver on a GCE VM and a new cloud run job configuration, allowing to launch Dagster runs on Cloud Run.

[garethbrickman]

@baumann-t Making you aware of this other PR that just got posted: #21864

Discussion is here with that contributor for how to move forward: #18958 (reply in thread)

[garethbrickman]

Work on the GCP Cloud Runner has been migrated to our community-integrations repo: dagster-io/community-integrations#26