Skip to content

Commit

Permalink
Convert dep_dsl to full example
Browse files Browse the repository at this point in the history
Test Plan: manual

Reviewers: schrockn

Differential Revision: https://dagster.phacility.com/D2970
  • Loading branch information
helloworld committed May 18, 2020
1 parent 6b0ebea commit bdd564d
Show file tree
Hide file tree
Showing 7 changed files with 92 additions and 8 deletions.
9 changes: 9 additions & 0 deletions .gitpod.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM gitpod/workspace-full

USER gitpod

# Install wxPython dependencies
RUN sudo apt-get -q update

ENV DAGSTER_HOME="$HOME/dagster_home"
RUN mkdir -p $DAGSTER_HOME
16 changes: 16 additions & 0 deletions .gitpod.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
image:
file: .gitpod.Dockerfile

# This will expose all necessary ports needed for your VNC image
ports:
- port: 3000
onOpen: open-preview

tasks:
- command: |
cd examples/dagster_examples/$EXAMPLE
pip3 install -r requirements.txt
[ -e README.md ] && gp open README.md
clear && dagit
- openMode: split-right
command: echo SplitTerminal && clear
54 changes: 54 additions & 0 deletions examples/dagster_examples/dep_dsl/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---
title: Dependency DSL
description: How to create a custom DSL for defining pipelines
---

# Dependency DSL

This demo example is to demonstrate that it is possible to construct
the dependencies of a pipeline definition from a YAML file or similar.
This is not the "blessed" file format nor is it our recommendation for
doing things. However we've been asked a number of times if this is possible
and this should serve as a reasonable template to build upon.

For example:

```YAML
pipeline:
name: some_example
description: blah blah blah
solids:
- def: add_one
alias: A
- def: add_one
alias: B
deps:
num:
solid: A
- def: add_two
alias: C
deps:
num:
solid: A
- def: subtract
deps:
left:
solid: B
right:
solid: C
```
# Open in Playground
Open up this example in a playground using [Gitpod](https://gitpod.io)
[![Open in Gitpod](https://gitpod.io/button/open-in-gitpod.svg)](https://gitpod.io/#EXAMPLE=dep_dsl/https://github.com/dagster-io/dagster)
# Download Manually
Download the example:
```
curl https://codeload.github.com/dagster-io/dagster/tar.gz/master | tar -xz --strip=3 dagster-master/examples/dagster_examples/dep_dsl
cd dep_dsl
```
Original file line number Diff line number Diff line change
@@ -1,13 +1,7 @@
'''
This demo example is to demonstrate that it is possible to construct
the dependencies of a pipeline definition from a yaml file or similar.
This is not the "blessed" file format nor is it our recommendation for
doing things. However we've been asked a number of times if this is possible
and this should serve as a reasonable template to build upon.
'''
from dagster import (
DependencyDefinition,
PipelineDefinition,
RepositoryDefinition,
SolidInvocation,
check,
file_relative_path,
Expand Down Expand Up @@ -60,3 +54,9 @@ def define_dep_dsl_pipeline():
return construct_pipeline_with_yaml(
file_relative_path(__file__, 'example.yaml'), [add_one, add_two, subtract]
)


def define_repository():
return RepositoryDefinition(
"my_repository", pipeline_dict={'dsl_pipeline': define_dep_dsl_pipeline}
)
3 changes: 3 additions & 0 deletions examples/dagster_examples/dep_dsl/repository.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
repository:
file: repo.py
fn: define_repository
2 changes: 2 additions & 0 deletions examples/dagster_examples/dep_dsl/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
dagster
dagit
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from dagster_examples.dep_dsl.pipeline import define_dep_dsl_pipeline
from dagster_examples.dep_dsl.repo import define_dep_dsl_pipeline

from dagster import execute_pipeline

Expand Down

0 comments on commit bdd564d

Please sign in to comment.