Dagster Cloud Hybrid Deployment Quickstart

This template lets you get started using Dagster Cloud with a Hybrid agent.

Create a new repository from this template

Click the Use this Template button and provide details for your new repo.

Add registry to dagster_cloud.yaml

The dagster_cloud.yaml file defines the configuration for building and deploying your code locations. Here, you will need to specify the Docker registry to push your code location to in the registry key.

For more information on the possible configuration options, see the Dagster Cloud docs.

Modify GitHub Workflow

Edit the GitHub Workflows at .github/workflows/deploy.yml and .github/workflows/branch_deployments.yml to set up Docker registry access. Uncomment the step associated with your registry (ECR, DockerHub, GCR etc.), and take note of which secrets will need to be defined for your particular platform.

Set up secrets

Set up secrets on your newly created repository by navigating to the Settings panel in your repo, clicking Secrets on the sidebar, and selecting Actions. Then, click New repository secret.

Name	Description
DAGSTER_CLOUD_API_TOKEN	An agent token, for more details see the Dagster Cloud docs.
ORGANIZATION_ID	The organization ID of your Dagster Cloud organization, found in the URL. For example, pied-piper if your organization is found at https://dagster.cloud/pied-piper or https://pied-piper.dagster.cloud/.
Docker access secrets	Depending on which Docker registry you are using, you must define the credentials listed in the workflow file.

Verify builds are successful

At this point, the Workflow should complete successfully. If builds are failing, ensure that your secrets are properly set up the Workflow properly sets up Docker regsitry access.

Adding or modfiying code locations

To add new code locations to be built or to modify the existing location definition, change the input matrix at the top of each action file.

For example:

matrix:
  # Here, define the code locations that should be built and deployed
  location:
    - name: foo_location
      # Dockerfile location
      build_folder: my_package/foo_location
      # Docker registry URL
      registry: https://364536301934.dkr.ecr.us-west-2.amazonaws.com/foo-location
      # Path to file containing location definition
      location_file: cloud_workspace.yaml
    - name: bar_location
      build_folder: my_package/bar_location
      registry: https://364536301934.dkr.ecr.us-west-2.amazonaws.com/bar-location
      location_file: cloud_workspace.yaml

The location_file specified can either contain a single location's definition, or a list of multiple locations' definitions.

Enable branch deployments

To enable Branch Deployments for your Hybrid agent, you will need to:

1. Ensure your agent is set up to run Branch Deployments.
2. Uncomment the triggers in the .github/workflows/branch_deployments.yml workflow file:

# Uncomment to enable branch deployments [run on pull request]
on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

# Comment this out once you have enabled branch deployments
# on: workflow_dispatch