When the task is in the up_for_retry or rescheduler state, logs cannot be obtained

[stursdsafdaf]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.3

What happened?

When the task is in the up_for_retry or rescheduler state, logs cannot be obtained.
As shown in the figure, only the hostname is displayed

What you think should happen instead?

Tasks in the up_for_retry and rescheduler states have been executed but failed and are in the retry phase. Logs should be displayed normally.

How to reproduce

I deployed 3 nodes and did not enable remote logging.
This happens if the task is running on a non-web node and is in the up_for_retry or rescheduler state.

Operating System

ubuntu 20.04

Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.7.3
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.2
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-ftp==3.10.0
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-mysql==5.6.3
apache-airflow-providers-redis==3.7.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.1

Deployment

Virtualenv installation

Deployment details

I use Python virtual environment deployment, the configuration is as attached
airflow_new.txt

Anything else?

code file :lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py
This code will result in only getting the log of the node where the web is located when remote logging is not used, so only the host name will be output

So, it should be changed to this

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Looks playsible. WDYT @dstandish ?

[AllardQuek]

Hi @stursdsafdaf, I am also facing a similar issue on v2.7.3. May I check if your proposed solution worked for you?

I noticed there is a previously merged PR #39496 that seems to perform a similar fix, though the changes do not seem to be included in the latest version. As I'm new to Airflow, could anyone confirm if the changes from #39496 would solve this issue?

[dstandish]

I think I added a related fix here #41272 which has milestone of 2.10.0.

Did you try 2.10?

[dstandish]

Yeah the reports here are from 2.9.3 and 2.7.3 so I assume that this is resolved by #41272 in 2.10.0 so I will close this.

[AllardQuek]

I think I added a related fix here #41272 which has milestone of 2.10.0.

Did you try 2.10?

We are working with v2.7.3 in production. To implement the fix in an earlier version, which changes do we require from https://github.com/apache/airflow/pull/41272/files? The changes do not seem to mention the up for retry state. Thank you!

[potiuk]

We are working with v2.7.3 in production. To implement the fix in an earlier version, which changes do we require from https://github.com/apache/airflow/pull/41272/files? The changes do not seem to mention the up for retry state. Thank you!

You need to upgrade. We do not have a process to individually guide and explain how to take parts of the code and apply it to your version if you do not want to upgrade. If you want there are a number of 3rd-party commercial companies to assist you with it (and get paid for it), but if you want to rely on volunteer's effort here, the only way to apply fixes is to upgrade.

[AllardQuek]

You need to upgrade. We do not have a process to individually guide and explain how to take parts of the code and apply it to your version if you do not want to upgrade. If you want there are a number of 3rd-party commercial companies to assist you with it (and get paid for it), but if you want to rely on volunteer's effort here, the only way to apply fixes is to upgrade.

Got it, thanks!