Task instance log_url is overwrites existing path in base_url

[wolfier]

Apache Airflow version

2.6.3

What happened

A task instance's log_url does not contain the full URL defined in base_url.

What you think should happen instead

The base_url may contain paths that should be acknowledged when build the log_url.

The log_url is built with urljoin. Due to how urljoin builds URLs, any existing paths are ignored leading to a faulty URL.

How to reproduce

This snippet showcases how urljoin ignores existing paths when building the url.

>>> from urllib.parse import urljoin
>>> 
>>> 
>>> urljoin(
...     "https://my.astronomer.run/path",
...     f"log?execution_date=test"
...     f"&task_id=wow"
...     f"&dag_id=super"
...     f"&map_index=-1",
... )
'https://eochgroup.astronomer.run/log?execution_date=test&task_id=wow&dag_id=super&map_index=-1'

Operating System

n/a

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else

This was introduced by #31833.

A way to fix this can be to utilize urlsplit and urlunsplit to account for existing paths.

from urllib.parse import urlsplit, urlunsplit

parts = urlsplit("https://my.astronomer.run/paths")
urlunsplit((
        parts.scheme,
        parts.netloc,
        f"{parts.path}/log",
        f"execution_date=test"
        f"&task_id=wow"
        f"&dag_id=super"
        f"&map_index=-1",
        ""
    )
)

Here is the fix in action.

>>> parts = urlsplit("https://my.astronomer.run/paths")
>>> urlunsplit((
...     parts.scheme,
...     parts.netloc,
...     f"{parts.path}/log",
...     f"execution_date=test"
...     f"&task_id=wow"
...     f"&dag_id=super"
...     f"&map_index=-1",
...     ''))
'https://my.astronomer.run/paths/log?execution_date=test&task_id=wow&dag_id=super&map_index=-1'
>>>
>>> parts = urlsplit("https://my.astronomer.run/paths/test")
>>> urlunsplit((
...     parts.scheme,
...     parts.netloc,
...     f"{parts.path}/log",
...     f"execution_date=test"
...     f"&task_id=wow"
...     f"&dag_id=super"
...     f"&map_index=-1",
...     ''))
'https://my.astronomer.run/paths/test/log?execution_date=test&task_id=wow&dag_id=super&map_index=-1'

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[pankajkoti]

yes, I think how urljoin works is it only takes the base URL until the last /and skips remaining parts(path). If the base URL ended with a trailing /, it would have worked fine.

Additionally the relatively URL should not contain a leading /, otherwise, the output will only be the relative URL.

There are some examples mentioned here: python/cpython#96015

[pankajkoti]

hi @wolfier where exactly are we seeing the effect of this in the UI?

I tried to set a custom path and able to fetch logs for the task in the UI (looks like it is able to get the right path).

[wolfier]

I am seeing this in places that can reference task instances and its attributes. This can be in the following places but not limited to:

• python callables
• callbacks (on_failure_callback)
• jinja templating ( {{ ti.log_url }} )