Dagster Pipeline Integration Tests

[quazi-h]

Description/Context

We had a situation where a Dagster resource was refactored during the development of a new pipeline. This caused changes in how the resource behaved, which then broke some of our other pipelines.

In this case, the ResultsDir resource was refactored to use the Pythonic interface which affected how files were passed from the resource within different ops. This needed to be addressed with a bugfix but we would like to avoid having our pipelines go down in such an event.

Some integration tests or smoke tests would help solve that issue. It should be fairly easy to mock out our resources and generate some test data to use as an input for the tests. We should run these tests as either part of the precommit hook or the automated tests that run on the branch after commits are pushed.

Plan/Design

We should aim for a simple test structure for each of our pipelines in our first iteration:

• Mock out dagster resources
• Create test input data
• Automate testing