Tasks taking too long time after 2.7.0 Airflow update

[jaetma]

Hi community!
We have been using Airflow quite a long time, and right after updating from version 2.6.3 to 2.7.0, the running time increased extremely high. Tasks that used to take 15 seconds to complete now are taking 10 minutes!

This is problematic because there are more tasks being queued than those that are finished.

We've detected this issue in 3 projects running with Airflow, across 2 instances in Kubernetes and 1 with Docker.
Illustrating image:

[pankajkoti]

HI @jaetma ,
Could you help answer the following questions that could help us check this more?

1. do you have some monitoring in place? Can you check what is the resource consumption for each of the Airflow components/pods?
2. Have you added any new DAGs that could be eating up a lot of resources and hence tasks are contending for resources?
3. What do the task logs say? Are they stuck at some specific step always?
4. How and where is Airflow deployed?
5. Which executor? Celery/Kubernetes/?

[jaetma]

Sure, thanks for your reply. For my instance running with Docker (using Celery):

1. Currently, we only have this DAG running, with a maximum of 4 DAG runs, and those are taking all the machine resources. Tasks are taking over 2 minutes to start, and when running, they are also slow:
   
   

2. No, recently we have not added any DAGs with high resource consumption.

3. Logs are taking too much time to load in the Airflow UI. They are always stuck before starting, in the queued state.

4. Airflow is deployed on a server with docker-compose.

5. We use Celery as the executor.

[jaetma]

Today we did a downgrade to 2.6.3 again, the issue seems to be resolved!

[bparamjeet]

Is this solved in 2.8.3 ?