Release Airflow 2.0

[mik-laj]

Hello,

I am creating this ticket to help follow this process and coordinate our activities. This is a meta-ticket, so I would ask committers and PMC members to keep it up-to-date so that everyone has easy access to up-to-date information. This ticket should not describe individual changes to the code but summarize information that active community members know, but new community members don't know but want to know. This will make it more transparent and friendly.

More information is available on the wiki:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning

All tasks are collected in "Airflow 2.0rc1" milestone for RC1 and all remaining "administrative" tasks that do not require release are in "Airflow 2.0" milestone

Roadmap

Apart from defining the scope of work, we should also complete these tasks.

High-level features

Feaature	Status	More information
KEDA	✅	-
Improve Webserver performance	-	dropped for 2.0
Enhanced real-time UI	✅	Auto-refresh
Improve Scheduler performance and reliability	✅	Issues
Extend/finish the API	✅	#8107
Production Docker image	✅	Project
Production Helm chart	-	dropped for 2.0
Split Providers	✅	Project

AIPs

Feature	Status	Meta-issue
AIP-8 Split Prociders into Separate Pacjages for Airflow 2.0	✅	-
AIP-9 Automated Dependency Management	✅	-
AIP-10: Multi-layered and multi-stage official Airflow CI image	✅	-
AIP-21: Changes in import paths	✅	-
AIP-16: Use nested commands in CLI	✅	-
AIP-15 Support Multiple-Schedulers for HA & Better Scheduling Performance	✅	#9630
AIP-31 Functional DAGs	✅	AIP-31 Task Flow API for nicer DAG definition
AIP-32: Airflow REST API	✅	#8107
AIP-34 Improvements to SubDags / new concept "TaskGroup"	✅	#10153
AIP-26 Production-ready Airflow Docker Image t	✅	?
AIP-3 Drop support for Python 2	✅	Link

Migration to Airflow 2.0

The 2.0 release of the Airflow is a significant upgrade and includes substantial major changes, For this reason, we must take additional steps to facilitate migration.

We discussed the migration path that a user should do in order to start using Airflow 2.0 easily

1. When the user is using Python 2.7, they need to migrate to Python 3.6+.
2. When the user is using Python 3.6+, they can start to use new operators and hooks from Python 3.6 thanks to backport packages.
3. When the user is on the latest Airflow 1.10 release, they can use the airflow upgrade-check command to see if they can migrate to the new Airflow version. This command checks for common incompatibility problems.
4. In the next step, the user installs the latest version of Airflow and will migrate the database - airflow db upgrade
5. Once the migration is successful the user can happily use Airflow 2.0.

In all steps, the user is assisted by migration guides and runtime warnings.

Backport packages:
We released packages that will allow us to use operators/hooks from Airflow 2.0 in Airflow 1.10.

Link: https://cwiki.apache.org/confluence/display/AIRFLOW/Backported+providers+packages+for+Airflow+1.10.*+series
Status: DONE

airflow upgrade-check
To make it easier for users to upgrade from 1.10 to 2.0, we should create a single upgrade-check command in 1.10 that checks the common incompatibilities.

Link: #8765
Status: WIP - we will continue releasing updates to upgrade check separately from 2.0 release.

Migration guide:
We have a lot of entries in UPDATING.md, but there is no full procedure that will answer the most important question - how to update Airflow to the latest version. Users expect step-by-step instructions from us that will explain what they should do. The guide should contain information about airflow upgrade-check, airflow db upgrade, backport package, etc.

Link: #10089
Status: WIP

Additionally, we should make other changes to the documentation.

Feature	Status	Issue
Prepare migration guide for CLI	✅	#9952
Prepare migration guide for REST API	✅	#8121
Double check UPDATING.md for "2.0" doesn't include things already released	WIP	#9641
Double check UPDATING.md for "2.0" doesn't include breaking changes in import paths	WIP	#10104

Runtime warnings

For each type of change, we should add notifications at the time of use so that the user doesn't have to read the documentation::

Name	Status	Issue	Description
Python API	WIP	#8765	Most of the changes have DeprecationWarning, but we can also detect common issues with upgrade-check.
CLI	✅	#10109	We can detect the use of the old command and propose to use the new counterparts
REST API	✅	#9888	We send the message in HTTP headers.
Database schema	WIP	#8765, #10129	We can also detect known issues with upgrade-check (Airflow 1.10) and db upgrade (Airflow 2..0)
Configuration	✅	#8765	We can also detect known issues with upgrade-check.

How to participate?

It is best for you to find the free ticket in the "Airflow 2.0" milestone and apply for it

You can also request a ticket assignemt on the #airflow-how-to-pr () Slack channel or at regular task-force meetings.

[mik-laj]

Please do not comment on this ticket. To the discussion on the release of the Airflow version, I invite you to:

• Slack channel: #development ()
• task-force meetings.
• developer mailing list: [email protected] (Pony Mail)

[mik-laj]

Task done.