Is it good to NOT run adopt_or_reset_orphaned_tasks on startup ?

[NickYadance]

The first adopt_or_reset_orphaned_tasks run is slow(~10sec) when restarting scheduler due to lots of ohphaned tasks generated, especially with KubernetesExecutor, which is making serial HTTP calls to adopt those tasks. And as adopt_or_reset_orphaned_tasks can potentially lock all the orphaned task instances, multiple scheduler startup will be stuck for quite some time with database not supporting skip_lock.

Not sure what side effects can have by keeping these ohphaned tasks while the scheduler is running, is it good to avoid it on startup ?

[potiuk]

You are not supposed to run multiple schedulers without SKIP_LOCK. You will have all kinds of weird errors if you do:

https://airflow.apache.org/docs/apache-airflow/stable/concepts/scheduler.html

MariaDB did not implement the SKIP LOCKED or NOWAIT SQL clauses until version 10.6.0. Without these features, running multiple schedulers is not supported and deadlock errors have been reported. MariaDB 10.6.0 and following may work appropriately with multiple schedulers, but this has not been tested.

I suggest you upgrade to a DB that is supported and your problem will be gone.

[NickYadance]

To avoid k8s apiserver backpressure, some parallelism configurations will be needed too, if this goes in a parallel way, as in #26639.

[potiuk]

Yep.

[hterik]

In #28071 there is suggestion of a solution to exclude already completed tasks from adoption, that would save a lot of time in that process.

[NickYadance]

In my deployment, i have configured Airflow to delete pods of completed tasks so it won't work for me.
@hterik

[NickYadance]

As for #26639, i'm trying to utilize the underground api_client of kube_client, which already offers nice support of async request, instead of constructing threadpool manually.