Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(airflow): upgrade airflow from 2.1.4 to 2.3.3 [DSRE-1039] #1596

Merged
merged 3 commits into from
Dec 7, 2022

Conversation

mikaeld
Copy link
Contributor

@mikaeld mikaeld commented Dec 2, 2022

Description

Upgrade Airflow version from 2.1.4 to 2.3.3

Changes

See Jira tickets for more details

  • Use official constraints file for dependencies and clean up old deprecated dependencies. Exact constraints file used available here and committed as constraints.txt
  • Update Airflow config to reflect changes such as new [database] section that used to be under [core]
  • General housekeeping: update README.md, cleanup imports, address deprecation warnings
  • Migrate database in nonprod and prod

Database Migration

We could not simply use airflow db migrate to run migrations because we use MySQL. We handled DB migration issues using this guide . See Jira tickets for more details on the manual steps required before running a DB migration.

Validation

Todo in staging

Related Jira Ticket

@mikaeld mikaeld force-pushed the feat/airflow-2-3-3-upgrade branch 3 times, most recently from 499ffbf to 0372029 Compare December 7, 2022 16:00
@mikaeld mikaeld changed the title WIP - [DSRE-1039] Upgrade airflow from 2.1.4 to 2.3.3 (cloud composer latest) feat(airflow): upgrade airflow from 2.1.4 to 2.3.3 [DSRE-1039] Dec 7, 2022
@mikaeld mikaeld force-pushed the feat/airflow-2-3-3-upgrade branch from 837464c to 9d409c0 Compare December 7, 2022 18:19
@mikaeld mikaeld self-assigned this Dec 7, 2022
@mikaeld mikaeld merged commit d19cc71 into main Dec 7, 2022
@mikaeld mikaeld deleted the feat/airflow-2-3-3-upgrade branch December 7, 2022 18:30
@haroldwoo haroldwoo restored the feat/airflow-2-3-3-upgrade branch December 8, 2022 17:43
@mikaeld mikaeld mentioned this pull request Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants