Terraform module for a Dagster deployment using Google Cloud managed services
This module defines a production-ready Dagster deployment which is nearly fully managed/serverless, with the exception of a lightweight Compute Engine instance running the Dagster daemon and the Cloud SQL database. It supports multiple code locations. The module is quite opinionated - additional configurability may be added in the future.
This module deploys:
- A Cloud Run webserver
- For each code location:
- A Cloud Run service for the gRPC code server
- A Cloud Run job for the run worker
- A Compute Engine instance for the Dagster daemon
- A Cloud SQL database and user for Dagster (the instance itself is not provisioned by this module)
- All required IAM for the above resources
- The webserver, daemon and code servers share a primary service account with minimal permissions
- A distinct service account is used for each run worker to facilitate fine-grained permissions between code locations
- Storage for IO and logs
Deploying this module requires a number of supporting resources, including prebuilt Docker images for the different jobs/services. Refer to dagster-gcp-demo for a worked example.
Generally speaking, the following additional components are recommended:
- Identify-Aware Proxy (IAP) for the Cloud Run webserver
- CI/CD pipelines to update the code servers and run workers when Dagster definitions are modified
- Migration support for upgrading Dagster (e.g. a Cloud Run job that runs
dagster instance migrate)