Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify/improve several DAGs #75

Merged
merged 7 commits into from
Jan 13, 2023
Merged

Simplify/improve several DAGs #75

merged 7 commits into from
Jan 13, 2023

Conversation

hammerhead
Copy link
Member

@hammerhead hammerhead commented Dec 22, 2022 •
edited
Loading

Summary of the changes / Why this is an improvement

Two improvements here:

  1. No need to remember anymore which partitions have already been reallocated. We now can query this information from sys.nodes. To be merged only once CrateDB 5.2 is available.
  2. Since Task mapping against 'params' apache/airflow#24014 is now solved, a few workarounds in the Data Retention DAGs became unnecessary, simplifying the DAGs a little.

Checklist

  • CLA is signed

@hammerhead hammerhead force-pushed the hammerhead/data-retention-5.2 branch 2 times, most recently from 2fde500 to 0d419f8 Compare December 22, 2022 10:07
@hammerhead hammerhead marked this pull request as ready for review December 22, 2022 10:11
@hammerhead hammerhead force-pushed the hammerhead/data-retention-5.2 branch from 0d419f8 to 8e048d6 Compare December 22, 2022 20:38
@hammerhead hammerhead changed the title Simplify reallocation using CrateDB 5.2 node attributes Simplify/improve Data Retention DAGs Dec 23, 2022
@hammerhead
Copy link
Member Author

I pushed another round of changes to the Data Retention DAGs while being at it already (see updated pull request description).

@hammerhead hammerhead force-pushed the hammerhead/data-retention-5.2 branch from b8916b1 to b0dcd0e Compare December 27, 2022 10:48
@hammerhead hammerhead changed the title Simplify/improve Data Retention DAGs Simplify/improve several DAGs Dec 28, 2022
@hammerhead hammerhead merged commit 0bdc52f into main Jan 13, 2023
@hammerhead hammerhead deleted the hammerhead/data-retention-5.2 branch January 13, 2023 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hlcianfagna hlcianfagna hlcianfagna approved these changes

@marijaselakovic marijaselakovic Awaiting requested review from marijaselakovic

@karynzv karynzv Awaiting requested review from karynzv

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hammerhead @hlcianfagna