Failed to fetch log file from worker. Request URL missing either an 'http://' or 'https://' protocol.

[Vishal2696]

Apache Airflow version

2.2.3

What happened

Airflow Webserver is unable to fetch the logfiles from worker. I'm using celery executor with the airflow components deployed in kubernetes as pods (under a deployment). Below is the error I see from the webserver UI

*** Log file does not exist: /opt/airflow/airflow/logs/mydagname/mytaskname/2022-05-19T04:53:37.167551+00:00/2.log
*** Fetching from: http://:8793/log/mydagname/mytaskname/2022-05-19T04:53:37.167551+00:00/2.log
*** Failed to fetch log file from worker. Request URL missing either an 'http://' or 'https://' protocol.

Below is the error I see in worker pod's STDOUT.

self.protocol = self.protocol if parsed_url.scheme is '' else parsed_url.scheme
[2022-05-19 10:06:22,046: ERROR/ForkPoolWorker-15] Task airflow.executors.celery_executor.execute_command[feae5691-b147-4bea-9b4a-bed01b559539] raised unexpected: AirflowException('Celery command failed on host: 10.22.52.110')
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 90, in execute_command
    _execute_in_fork(command_to_exec, celery_task_id)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 101, in _execute_in_fork
    raise AirflowException('Celery command failed on host: ' + get_hostname())
airflow.exceptions.AirflowException: Celery command failed on host: 10.22.52.110

I tried setting the AIRFLOW__CORE__HOSTNAME_CALLABLE to "airflow.utils.net.get_host_ip_address" in both webserver and worker but nothing changed. Still the same error. The worker is sitting behind a kubernetes service with port 8793 mapped to it.

Metadata DB: Azure Postgres Server Version 11.

Providers Info

apache-airflow-providers-celery | 2.1.0
apache-airflow-providers-ftp | 2.0.1
apache-airflow-providers-http | 2.0.1
apache-airflow-providers-imap | 2.0.1
apache-airflow-providers-microsoft-azure | 3.9.0
apache-airflow-providers-oracle | 2.2.3
apache-airflow-providers-postgres | 2.4.0
apache-airflow-providers-redis | 2.0.1
apache-airflow-providers-sqlite | 2.0.1

Operating System

Airflow docker image built from source code.

Deployment

Other Docker-based deployment

Deployment details

kubernetes

Code of Conduct

• I agree to follow this project's Code of Conduct

[uranusjr]

See #13692; more information is needed.

[Vishal2696]

@uranusjr I'm not sure what all the info you need, but I'm providing as much as I can.

1. The image being run was built from the official apache github repo (Source code). Following the same instructions given in the repo.
2. The task that is problematic is trying to trigger an azure data-factory pipeline. azure.mgmt.datafactory is the library that I'm using.
3. The DAGS are baked into the airflow image as a python artifact.
4. In this particular case I'm running one webserver, one scheduler and one worker. I have observed this in environments with multiple schedulers and multiple workers too.
5. Celery logs or STDOUT logs from the worker have already been attached in the description itself

[potiuk]

@uranusjr I'm not sure what all the info you need, but I'm providing as much as I can.

Just to set expectations: Basically, any information that might help people here to help you. People here help in their free time, and more often than not the problems are connected to people misconfiguring Airlfow, or simply not troubleshooting their deployment as they should. While the community here delivers airflow as software - you are responsible for configuring it properly (and learning how to do it). So if we help here - we do it in our free time, mostly when users had shown that they've done investigations on their side and provided enough info to help others diagnose it.

There shoudl be no expectations that you will for sure get help here. If you provide enough info, you might get it.

I think - if you want to help us diagnose your deployment issue - is to provide even more information. So it's really you who should care about giving as much information as you can, so that we can help you (again - this is free help on free software, if you NEED your problem to be solved, there are companies that provide paid support for troubleshooting airlfow installations).

In this particular case I can advice you to provide more information on circumstances. A number of questions come to my mind immediately - but again - probably you can provide even more information - it's you, who want to have your problem solved, so it's your worry to provide enough information.

• Does it happen always?
• or for specific tasks only?
• or maybe intermittently (sometimes works, sometimes not)?
• what customisations you have (Especially for logging)
• do you have any netwerking/firewall configuration?
• did you check that networking works between the webserver and UI and that dns works properly to see your pods from the URL?
• do you have special configuration for logs ?
• does your pod have persistent storage for logs ?
• is the file that you are trying to access when you see the error /opt/airflow/airflow/logs/mydagname/mytaskname/2022-05-19T04:53:37.167551+00:00/2.log - actually present in the workar that webserver tries to download it from?
• is your callable (the one you configured) actually called and works? Please run it in the worker pod and see what it returns as hostname/Ip address. Maybe your networking is miconfigured and it provides empty value
• do you have the right hostname in task_instance table for the tasks you run ? What is the value for an example failure.

I think going through that list might help you to gather enough information to make it possible to help you.

Converting it into discussion until more info is provided.

[Vishal2696]

@potiuk @uranusjr New anamoly noted today.
Q: do you have any netwerking/firewall configuration?
A: Don't have anything in kubernetes to prevent communication between worker and webserver. However when I try to curl from my webserver to worker I get "403: Forbidden Error". Even after setting the AIRFLOW__WEBSERVER__SECRET_KEY to the same value in webserver and worker, When I see the config file inside the container, they seem to contains different values. Hence the "403: Forbidden" also continues.

[potiuk]

sonce you have "Celery command failed on host" - this means that for some reason, the worker spawned by Celery failed. You need to dig deeper and do some debugging, but my guess is that you might for example not have enough resources (memory for example) and that's why the command might fail without additional log).

You need to look deeper in your K8S logs as apparently - if you see nothing in Airflow logs, the deployment of yours (K8S) might have better logs explaining the reason why things failed. - but this is something you need to investigate yourself.

I recommend using k9s tool for that - it's very useful.

[Vishal2696]

@potiuk I solved this by increasing the memory. (Currently sitting at 1 GB) Since celery worker is treated as multiple processes, and one of the process dying due to lack of memory is not captured by kubernetes. Thus the lack of logs.

[potiuk]

Cool. So the memory was the root cause. BTW this post (about exactly this problem) mentions that there is a way to monitor all OOM issues by K8S, not only those that eventualy kill the container (It's precisely about this case and mention Datadog, but if Datadog can do it, I am pretty sure any monitoring solution can do it): https://engineering.backmarket.com/a-story-of-kubernetes-celery-resource-and-a-rabbit-ec2ef9e37e9f - maybe you should look at surfacing all the OOM events in your monitoring system?

[potiuk]

BTW. Here is a VERY NICE it seems and very detailed article about handling and monitoring OOM in K8S - looks really great and detailed, I added it to my "reading" list: https://mihai-albert.com/2022/02/13/out-of-memory-oom-in-kubernetes-part-4-pod-evictions-oom-scenarios-and-flows-leading-to-them/ I