Links to specific dagruns don't work when older than 25 runs

[hterik]

Apache Airflow version

2.7.1

What happened

1. Browse to AIRFLOW/dagrun/list/?_flt_3_dag_id=my_dag
2. Click any link from the Run Id column, below the most recent 25 runs.
   It should take you to AIRFLOW/dags/mydag/graph?run_id=scheduled__2023-09-21T00%3A00%3A00%2B00%3A00&execution_date=2023-09-21+00%3A00%3A00%2B00%3A00 and it does so correctly
3. The graph view opens but no run is selected.
   When clicking on any item in the graph, the LATEST run is selected in the grid. If the user is not careful to observe this, they are mislead to thinking the results they are browsing in the graph is the ones they selected in step 2.

As workaround, one can append &num_runs=365 to the url, this will take you to the correct run from step 2. But eventually you will exhaust 365 runs as well.

This problem isn't only applicable to the dagrun list. Any links to the /dags/ graph are affected. We have external systems linking to individual dagruns that now get unusable after 25 runs.

What you think should happen instead

When navigating to a run via link, it should always present the run that was provided in run_id querystring.

How to reproduce

See above

Operating System

Debian GNU/Linux 11 (bullseye)

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

No response

Anything else

This was not a problem in Airflow 2.5, regression in 2.7, probably introduced by new embedded graph view (which otherwise is great :))

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[hussein-awala]

I cannot reproduce with main:

Could you provide some screenshots for your problem?

[hterik]

Sure, hope this is enough

The url was given by clicking a run in the List dag runs page.
Something interesting to observe is that after clicking the url, it redirects 2 times, first adding the dag_run_id parameter (not run_id), which then quickly gets removed again.

[hussein-awala]

This date is the max date for the X dag runs, when you choose 2023-01-01 as start date, and 25 as a number of runs, the UI will show you the previous 25 runs before 2023-01-01. To select a dag run, you need to click on the red/green bar in the grid view.

[hterik]

This was not the case in Airflow 2.5, i could click on links referring to any dagrun, no matter how old, and it would correctly select it in the grid-view.
The selection of date and number of runs shouldn't have to be provided manually if run_id is part of the url.

To clarify, i am browsing the dagrun list attached below and clicking an item in the picture 1 here, then it results in the picture 2 (same as above) where no run is selected:

[jscheffl]

I did a test with an example DAG, triggered this manually >25 times. As the GRID view shows 25 items per default all is fine if you click on one of the recent 25 items.

If you click on an item >25 runs, GRID view will not adjust the selection of the 25runs to include in the column section your run. In my setup it shows the grid view, but clicking on the "Details" tab it shows the DAG overview, not details of the (previously) selected DAG runs. So in total it is hard to navigate from a DAG search result to the GRID view t check for details. (Especially because in GRID view the search options are limited compared to the DAG search screen)

[hterik]

I've done some debugging:

Problem 1.
Dag list is linking to the legacy /graph endpoint. This endpoint is only for backwards compatibility purposes and redirects to the new /grid endpoint.
This old function does actually calculate the base_date based on run_id, inside dt_nr_dr_data = get_date_time_num_runs_dag_runs_form_data(request, session, dag)
https://github.com/apache/airflow/blob/0c8e30e43b70e9d033e1686b327eb00aab82479c/airflow/www/views.py#L216-230
But then it forgets to forward the dt_nr_dr_data["base_date"] in the returned kwargs:
https://github.com/apache/airflow/blob/0c8e30e43b70e9d033e1686b327eb00aab82479c/airflow/www/views.py#L2964-2969
SolutionA: Forward base_date
SolutionB: Change dag list to use new url

Problem 2
The new grid endpoint, receives the run_id parameter correctly, after the redirection described above.
However, it then performs a Javascript fetch call to the grid_data API. In this API call, the run_id is NOT included.
The grid_data function that serves this request has not implemented handling of run_id parameter, in the same way that the old get_date_time_num_runs_dag_runs_form_data function did.
https://github.com/apache/airflow/blob/0c8e30e43b70e9d033e1686b327eb00aab82479c/airflow/www/views.py#L3475-3478
Solution C: Implement run_id handling in grid_data and forward the parameter from the frontend.

I think both solution A and C are must haves here. B is optional, more nice to have to get rid of legacy in the future.

[AlexandreGCastor]

Hello,

This was Ok in airflow 2.6.2. This possibility to get logs on the history (more than 25 run ago) is very important for us. It's way too hard to hack the url each time for us. We would probably rollback waiting for this issue.

Alexandre Gué,
Lead DevOps CastorDoc

[hterik]

Sorry for pinging but is there any way we can move this forward? I've already provided a fix in #34887, waiting for review.
IMO not being able to permalink to specific dagruns is quite severe bug.

[QuintenBruynseraede]

Also eagerly waiting for this fix. For the time being, is there a workaround to filter + navigate to a dagrun?

[jermaine151]

I found a workaround to this problem in v2.7.3. If you add &base_date=<same as execution date> to the URL, it will locate runs older than the latest 25. Here's an example link that is working for me:

<base_url>/dags/<dag_id>/grid?execution_date=2023-12-14+19%3A10%3A24.218280%2B00%3A00&base_date=2023-12-14+19%3A10%3A24.218280%2B00%3A00&tab=graph&dag_run_id=<run_id>

[QuintenBruynseraede]

Thanks @jermaine151, that seems to work indeed. It doesn't visually highlight a dagrun in the grid view (see Screenshot), but if you navigate to the Graph tab, then click on a Task it does lead you to the dagrun from the URL.

That last bit is important because in our case multiple runs are triggered on the same second, the gridview doesn't distinguish between those.

[jermaine151]

Thanks @jermaine151, that seems to work indeed. It doesn't visually highlight a dagrun in the grid view (see Screenshot), but if you navigate to the Graph tab, then click on a Task it does lead you to the dagrun from the URL.

That last bit is important because in our case multiple runs are triggered on the same second, the gridview doesn't distinguish between those.

Are you adding the &dag_run_id= to the URL? That should land you on the right page, with the run selected.

[potiuk]

cc: @bbovenzi @pierrejeambrun - would be cool to take a look and fix that one as it seems to be an important feature for those who complain about loosing the old grid view (see #36884) and it seems a bit of an architectural decision on the grid view behaviour.