Kubernetes does not show logs for task instances if remote logging is not configured

[ephraimbuddy]

Without configuring remote logging, logs from Kubernetes for task instances are not complete.

Without remote logging configured, the logging for task instances are outputted as :

logging_level: INFO

BACKEND=postgresql
DB_HOST=airflow-postgresql.airflow.svc.cluster.local
DB_PORT=5432

[2021-04-03 12:35:52,047] {dagbag.py:448} INFO - Filling up the DagBag from /opt/airflow/dags/k8pod.py
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
Running <TaskInstance: k8_pod_operator_xcom.task322 2021-04-03T12:25:49.515523+00:00 [queued]> on host k8podoperatorxcomtask322.7f2ee45d4d6443c5ad26bd8fbefb8292

Apache Airflow version: 2.0.1

Kubernetes version (if you are using kubernetes) (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-21T01:11:42Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 20.04

What happened:

The logs for task instances run are not shown without remote logging configured

What you expected to happen:

I expected to see complete logs for tasks

How to reproduce it:
Start airflow using the helm chart without configuring remote logging.
Run a task and check the logs.
It's necessary to set delete_worker_pods to False so you can view the logs after the task has ended

[ernest-kr]

Adding a chart feature to have a volume claim for logs as well just as how we have for DAGS. We might be able to change using this with extras as well

[Dr-Denzy]

I have been able to reproduce this and after reviewing the issue with @jedcunningham, we are of the opinion that this is not really a bug but more of a 'missing' feature in Airflow Helm chart stemming from how task instance logs are handled.

[jedcunningham]

@ephraimbuddy, we now have more options for logs in the chart:
https://airflow.apache.org/docs/helm-chart/stable/manage-logs.html

The only remaining thing I can think of here would be to add a warning if a user is using KubernetesExecutor but hasn't configured any logging yet. See #16784.

Anything else we can/should do?

[ephraimbuddy]

I think the warning part is cool, I will go ahead and close this. Thanks for the works!!