Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prioritize queueing of Dagster assets and pipelines #1272

Open
3 tasks
blarghmatey opened this issue Sep 5, 2024 · 0 comments
Open
3 tasks

Prioritize queueing of Dagster assets and pipelines #1272

blarghmatey opened this issue Sep 5, 2024 · 0 comments
Labels
enhancement New feature or request product:data-platform Issues related to the Data Platform product

Comments

@blarghmatey
Copy link
Member

Description/Context

The current configuration of the Dagster instance is using the QueuedRunCoordinator with a maximum concurrent steps of 10. We can customize this to either just increase/decrease that maximum concurrency, or we can set priority ordering of different tasks to prevent more urgent tasks from getting blocked by lower priority/slower steps.

Plan/Design

  • Determine the relative priorities of the different asset groupings and what type of groupings we want to have for the priority queues (e.g. based on resource constraints such as Airbyte or Open edX, or by task speed/frequency, etc.)
  • Update the Dagster configuration here with the proper configuration based on documentation found here
  • Apply any asset/operation tagging that is necessary to support the queue priorities and concurrency limits.
@blarghmatey blarghmatey added enhancement New feature or request product:data-platform Issues related to the Data Platform product labels Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request product:data-platform Issues related to the Data Platform product
Projects
None yet
Development

No branches or pull requests

1 participant