Airflow does not have an inbuilt scalable API for returning the status of a dagRun

[sonalprsd]

Airflow/MWAA does not seem to have any scalable API for returning the status of a dagRun, the APIs states-for-dag-run or list-runs are not scaling well. To fetch the dagRun status, every team seems to have some custom solution using sns_notification or updating the status to some external data store via Airflow callbacks.

My use case is to fetch the Dag status of all the Active runs and update the status tables in the system. There is a poller (with a timeout of 150s configured based on our SLA). The states-for-dag-run API seems to be doing scan operation internally. As the number of DAG runs in system increases, the time to get the status of dagRun increases further. Initially, fetching the status of 100 runs took 2.5 minutes. With increase of dagRuns in the system by 50, the fetch operation to get status for 100 dagRuns is taking more than 5 minutes.

Fetching the status of a workflow is a very common use case and I think airflow should provide a scalable API for doing this.

[potiuk]

Maybe convert it into a feature request. Sounds plausible.

[potiuk]

It's a bit different.

Replacing email notification will not (and should not) cover all the notifications that we are talking about. I cannot emagine sending emails for every state change.

Here we are talking about automatically consumable (and filter-able) stream of events that reflect status of Airflow as a system. Email notification replacement is about human consumable notifications (slack/email). Those two are pretty different beasts.

[sonalprsd]

The use case I have mentioned works for us with other workflow systems, we poll the status of a batch of RunIds. My issue here is that a simple use case to fetch the status of a dagRun should be scalable and not based on a scan operation. The ask is to expose an API which can return dagRun status in most optimized time/by an internal query operation and not a scan.

[sonalprsd]

This can be useful. I know that some companies do it by accessing the backend DB directly and run Sql query to get what they need - which is something we dont encourage. Can you please create a feature request https://github.com/apache/airflow/issues

#27782 <= created a new issue

[hterik]

In 2.5.0 there is now a callback covering all dag run transitions. From #27113
https://airflow.apache.org/docs/apache-airflow/2.5.0/listeners.html
This appears to run within the airflow process, having an api to listen to from outside is of course also very useful.

[potiuk]

Indeed. Very goood point @hterik. And you can easily build your plugin to handle it in the way that it will be a) stored in some DB b) exposed c) streamed.

This is one of the great 2.5 features that follows the "Airflow as a platform" approach.

I think this will be one of my favourite features of Airflow in the coming months and one that will allow to answer many questions :)

[ashb]

We recently (2.4?) added updated_at columns to dag_run and task_instance tables, so we can now add much better filtering/polling: "Give me anything that changed (to this dag/all dags) since time X"

[potiuk]

Unfortunetaly it's not available in the API (as far as I checked) but it's still a bit of a band-aid until more real-time/async interface is available (and has a number of quirks and pattern of the client using it must be very specific - especially about ordering).

I think async is inevitable sooner or later for those kind of interfaces - if not for easiness of use then for efficiency (REST is just bend over backwards to fit those kind of use cases as it has a number of characteristics that makes it awkward and cumbersome to implement and polling is terribly inefficient).

[ashb]

Getting push based notification of changes is a very non-trivial thing to achieve from an architectural stand point, and while it would be nice, adding "global" support for updated_after to the API would be an small change to get to an 80% solution

[bbovenzi]

+1 to adding updated_after a query param for dag run and task instance endpoints. That could even be marked as a "good first issue" for new contributors.

…

On Fri, Nov 18, 2022 at 8:01 AM Ash Berlin-Taylor ***@***.***> wrote: Getting push based notification of changes is a very non-trivial thing to achieve from an architectural stand point, and while it would be nice, adding "global" support for updated_after to the API would be an small change to get to an 80% solution — Reply to this email directly, view it on GitHub <#27765 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDDJBYOKXQGAPN5P4VBJHTWI54Y5ANCNFSM6AAAAAASD66SXE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pierrejeambrun]

+1 on update_after, as for this specific case it might be enough with 'low' effort. An event based/push/webhooks mechanism would be great to solve this problem but would require a significant effort as @ashb mentioned.

[potiuk]

+1 too. update_after seems to be simple enough and easy to implement. There are inherent issues (catching up, duplicates etc) with async solutions and I've implemented enout update_after kind of interfaces to know it's pretty ok.

We just need to actually implement it :)