Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove all calls to get_or_create_run from dagster-airflow #2412

Closed
helloworld opened this issue Apr 28, 2020 · 2 comments
Closed

Remove all calls to get_or_create_run from dagster-airflow #2412

helloworld opened this issue Apr 28, 2020 · 2 comments
Labels
integration: airflow Related to dagster-airflow

Comments

@helloworld
Copy link
Contributor

No description provided.

@mgasner mgasner added refactor integration: airflow Related to dagster-airflow labels Apr 28, 2020
@catherinewu
Copy link
Contributor

catherinewu commented May 15, 2020 •
edited
Loading

After talking to some folks about their airflow workflows, it appears to be a common pattern to re-run a dag from a failed task (which can be downstream of the root task). Thus, we need the ability for arbitrary tasks to be able to create runs, so I think we should keep get_or_create_run() and add use https://dagster.phacility.com/D2719 for the race

catherinewu pushed a commit that referenced this issue May 19, 2020
Add error handling for get_or_create_run race condition
5c6b8e6 
Summary:
Handle the possibility of concurrent RunStorage add_run() by trying to fetch the run again after DagsterRunAlreadyExists

#2412

Test Plan: todo

Reviewers: max, schrockn, sashank

Reviewed By: schrockn

Subscribers: schrockn, sashank, max

Differential Revision: https://dagster.phacility.com/D2719
@catherinewu
Copy link
Contributor

Decided to keep get_or_create_run, with some added robustness, testing, and documentation in https://dagster.phacility.com/D2719

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
integration: airflow Related to dagster-airflow
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mgasner @helloworld @catherinewu