Airflow version: 2.3.3 RuntimeError: Unable to find any timezone configuration

[psubudhi]

Apache Airflow version

2.10.1

If "Other Airflow 2 version" selected, which one?

2.3.3

What happened?

Started airflow scheduler and webserver and opened the endpoint for 6007 from Jarvis lab. Got below error.

Ooops!
Something bad has happened.

Airflow is used by many users, and it is very likely that others had similar problems and you can easily find
a solution to your problem.

Consider following these steps:

• gather the relevant information (detailed logs with errors, reproduction steps, details of your deployment)

• find similar issues using:

  • GitHub Discussions
  • GitHub Issues
  • Stack Overflow
  • the usual search engine you use on a daily basis

• if you run Airflow on a Managed Service, consider opening an issue using the service support channels

• if you tried and have difficulty with diagnosing and fixing the problem yourself, consider creating a bug report.
  Make sure however, to include all relevant details and results of your investigation so far.

Python version: 3.8.12
Airflow version: 2.3.3
Node: f6d2b5df6971

Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1519, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1517, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1503, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/opt/conda/lib/python3.8/site-packages/airflow/www/auth.py", line 46, in decorated
return func(*args, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/airflow/www/views.py", line 964, in index
return self.render_template(
File "/opt/conda/lib/python3.8/site-packages/airflow/www/views.py", line 709, in render_template
return super().render_template(
File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 322, in render_template
return render_template(
File "/opt/conda/lib/python3.8/site-packages/flask/templating.py", line 154, in render_template
return _render(
File "/opt/conda/lib/python3.8/site-packages/flask/templating.py", line 128, in _render
rv = template.render(context)
File "/opt/conda/lib/python3.8/site-packages/jinja2/environment.py", line 1291, in render
self.environment.handle_exception()
File "/opt/conda/lib/python3.8/site-packages/jinja2/environment.py", line 925, in handle_exception
raise rewrite_traceback_stack(source=source)
File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/dags.html", line 43, in top-level template code
{% elif curr_ordering_direction == 'asc' and request.args.get('sorting_key') == attribute_name %}
File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 21, in top-level template code
{% from 'airflow/_messages.html' import show_message %}
File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code
{% import 'appbuilder/baselib.html' as baselib %}
File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 37, in top-level template code
{% block body %}
File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 16, in block 'body'
{% block messages %}
File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/dags.html", line 104, in block 'messages'
{{ super() }}
File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 62, in block 'messages'
{% call show_message(category='warning', dismissible=false) %}
File "/opt/conda/lib/python3.8/site-packages/jinja2/runtime.py", line 828, in _invoke
rv = self._func(*arguments)
File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/_messages.html", line 25, in template
{{ caller() }}
File "/opt/conda/lib/python3.8/site-packages/jinja2/runtime.py", line 828, in _invoke
rv = self._func(*arguments)
File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 70, in template
>{{ macros.datetime_diff_for_humans(scheduler_job.latest_heartbeat) }}.
File "/opt/conda/lib/python3.8/site-packages/airflow/macros/init.py", line 77, in datetime_diff_for_humans
return pendulum.instance(dt).diff_for_humans(since)
File "/opt/conda/lib/python3.8/site-packages/pendulum/datetime.py", line 824, in diff_for_humans
other = self.now()
File "/opt/conda/lib/python3.8/site-packages/pendulum/datetime.py", line 106, in now
return pendulum.now(tz)
File "/opt/conda/lib/python3.8/site-packages/pendulum/init.py", line 211, in now
dt = _datetime.datetime.now(local_timezone())
File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/init.py", line 60, in local_timezone
return get_local_timezone()
File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py", line 35, in get_local_timezone
tz = _get_system_timezone()
File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py", line 63, in _get_system_timezone
return _get_unix_timezone()
File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py", line 242, in _get_unix_timezone
raise RuntimeError("Unable to find any timezone configuration")
RuntimeError: Unable to find any timezone configuration

What you think should happen instead?

UI should open

How to reproduce

Open the UI

Operating System

Linux f6d2b5df6971 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Versions of Apache Airflow Providers

2.3.3

Deployment

Other

Deployment details

NA

Anything else?

NA

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[koustreak]

env:

• name: AIRFLOW__CORE__DEFAULT_TIMEZONE
  value: "UTC"

Have you tried this one ??

[jscheffl]

Version 2.3.3 of Airflow is pretty pretty old. Have you considered upgrading to a more recent version?

[psubudhi]

We have received the lab with pre-installed airflow, and will check with the team if the airflow can be upgraded to the latest version. But don't we have a solution for the above issue? Thanks, Prem

…

On Tue, 1 Oct 2024 at 02:12, Jens Scheffler ***@***.***> wrote: Version 2.3.3 of Airflow is pretty pretty old. Have you considered upgrading to a more recent version? — Reply to this email directly, view it on GitHub <#42558 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDW4VCHKWMKOYZGF5SICDTZZGZTDAVCNFSM6AAAAABPAT7ZVSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBUGEYDQOBYGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[psubudhi]

I checked the config

default_timezone = utc

[psubudhi]

fyi

[jscheffl]

Re-reading the stack trace there is something seriously wrog with your system anyway. The error is rooted becasue the webser attempts to render an error message that the scheduler in unhealty.

Are you sure that the default_timezone is set consistently across all nodes? Looking at the configuration reference utcis actually the default (see https://airflow.apache.org/docs/apache-airflow/2.3.3/configurations-ref.html)

Note that the upper case version AIRFLOW__CORE__... in your config is the version to over-ride the CFG file from ENV variables. To debug what actually is used, can you open a shell i nthe context of the webserver and run airflow config list on the command line and check which value is active?

[psubudhi]

Thank you Jens for the reply. Here are my inputs to your asks. 1. I am using a single node server. 2. Here is the config list ***@***.***:~# airflow config list|grep timezone default_timezone = utc default_ui_timezone = UTC ***@***.***:~# FYI More log of scheduler and web server attached. Please let me know if you have any details you might need from the host node. Regards, Prem

On Wed, 2 Oct 2024 at 00:37, Jens Scheffler ***@***.***> wrote: Re-reading the stack trace there is something seriously wrog with your system anyway. The error is rooted becasue the webser attempts to render an error message that the scheduler in unhealty. Are you sure that the default_timezone is set consistently across all nodes? Looking at the configuration reference utcis actually the default (see https://airflow.apache.org/docs/apache-airflow/2.3.3/configurations-ref.html ) Note that the upper case version AIRFLOW__CORE__... in your config is the version to over-ride the CFG file from ENV variables. To debug what actually is used, can you open a shell i nthe context of the webserver and run airflow config list on the command line and check which value is active? — Reply to this email directly, view it on GitHub <#42558 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDW4VB7Q75HTOUIBZVPZBDZZLXFHAVCNFSM6AAAAABPAT7ZVSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBWG42TINBRGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

***@***.***:~# airflow config list [core] dags_folder = /home/airflow/dags hostname_callable = socket.getfqdn default_timezone = utc executor = SequentialExecutor parallelism = 32 max_active_tasks_per_dag = 16 dags_are_paused_at_creation = True max_active_runs_per_dag = 16 load_examples = True plugins_folder = /home/airflow/plugins execute_tasks_new_python_interpreter = False fernet_key = donot_pickle = True dagbag_import_timeout = 30.0 dagbag_import_error_tracebacks = True dagbag_import_error_traceback_depth = 2 dag_file_processor_timeout = 50 task_runner = StandardTaskRunner default_impersonation = security = unit_test_mode = False enable_xcom_pickling = False killed_task_cleanup_time = 60 dag_run_conf_overrides_params = True dag_discovery_safe_mode = True dag_ignore_file_syntax = regexp default_task_retries = 0 default_task_weight_rule = downstream default_task_execution_timeout = min_serialized_dag_update_interval = 30 compress_serialized_dags = False min_serialized_dag_fetch_interval = 10 max_num_rendered_ti_fields_per_task = 30 check_slas = True xcom_backend = airflow.models.xcom.BaseXCom lazy_load_plugins = True lazy_discover_providers = True hide_sensitive_var_conn_fields = True sensitive_var_conn_names = default_pool_task_slot_count = 128 max_map_length = 1024 AIRFLOW__CORE__DEFAULT_TIMEZONE = UTC [database] sql_alchemy_conn = sqlite:////home/airflow/airflow.db sql_engine_encoding = utf-8 sql_alchemy_pool_enabled = True sql_alchemy_pool_size = 5 sql_alchemy_max_overflow = 10 sql_alchemy_pool_recycle = 1800 sql_alchemy_pool_pre_ping = True sql_alchemy_schema = load_default_connections = True max_db_retries = 3 [logging] base_log_folder = /home/airflow/logs remote_logging = False remote_log_conn_id = google_key_path = remote_base_log_folder = encrypt_s3_logs = False logging_level = INFO celery_logging_level = fab_logging_level = WARNING logging_config_class = colored_console_log = True colored_log_format = [%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(log_color)s%(message)s%(reset)s colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter log_format = [%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s simple_log_format = %(asctime)s %(levelname)s - %(message)s task_log_prefix_template = log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log log_processor_filename_template = {{ filename }}.log dag_processor_manager_log_location = /home/airflow/logs/dag_processor_manager/dag_processor_manager.log task_log_reader = task extra_logger_names = worker_log_server_port = 8793 [metrics] statsd_on = False statsd_host = localhost statsd_port = 8125 statsd_prefix = airflow statsd_allow_list = stat_name_handler = statsd_datadog_enabled = False statsd_datadog_tags = [secrets] backend = backend_kwargs = [cli] api_client = airflow.api.client.local_client endpoint_url = http://localhost:8080 [debug] fail_fast = False [api] enable_experimental_api = False auth_backends = airflow.api.auth.backend.session maximum_page_limit = 100 fallback_page_limit = 100 google_oauth2_audience = google_key_path = access_control_allow_headers = access_control_allow_methods = access_control_allow_origins = [lineage] backend = [atlas] sasl_enabled = False host = port = 21000 username = password = [operators] default_owner = airflow default_cpus = 1 default_ram = 512 default_disk = 512 default_gpus = 0 default_queue = default allow_illegal_arguments = False [hive] default_hive_mapred_queue = [webserver] base_url = http://localhost:8080 default_ui_timezone = UTC web_server_host = 0.0.0.0 web_server_port = 6007 web_server_ssl_cert = web_server_ssl_key = session_backend = database web_server_master_timeout = 120 web_server_worker_timeout = 120 worker_refresh_batch_size = 1 worker_refresh_interval = 6000 reload_on_plugin_change = False secret_key = CbGpipwchu6Dg00sg5m0ZA== workers = 4 worker_class = sync access_logfile = - error_logfile = - access_logformat = expose_config = False expose_hostname = True expose_stacktrace = True dag_default_view = grid dag_orientation = LR log_fetch_timeout_sec = 5 log_fetch_delay_sec = 2 log_auto_tailing_offset = 30 log_animation_speed = 1000 hide_paused_dags_by_default = False page_size = 100 navbar_color = #fff default_dag_run_display_number = 25 enable_proxy_fix = False proxy_fix_x_for = 1 proxy_fix_x_proto = 1 proxy_fix_x_host = 1 proxy_fix_x_port = 1 proxy_fix_x_prefix = 1 cookie_secure = False cookie_samesite = Lax default_wrap = False x_frame_enabled = True show_recent_stats_for_completed_runs = True update_fab_perms = True session_lifetime_minutes = 43200 instance_name_has_markup = False auto_refresh_interval = 3 warn_deployment_exposure = True audit_view_excluded_events = gantt,landing_times,tries,duration,calendar,graph,grid,tree,tree_data [email] email_backend = airflow.utils.email.send_email_smtp email_conn_id = smtp_default default_email_on_retry = True default_email_on_failure = True [smtp] smtp_host = localhost smtp_starttls = True smtp_ssl = False smtp_port = 25 smtp_mail_from = ***@***.*** smtp_timeout = 30 smtp_retry_limit = 5 [sentry] sentry_on = False sentry_dsn = [local_kubernetes_executor] kubernetes_queue = kubernetes [celery_kubernetes_executor] kubernetes_queue = kubernetes [celery] celery_app_name = airflow.executors.celery_executor worker_concurrency = 16 worker_prefetch_multiplier = 1 worker_enable_remote_control = True worker_umask = 0o077 broker_url = redis://redis:6379/0 result_backend = ***@***.***/airflow flower_host = 0.0.0.0 flower_url_prefix = flower_port = 5555 flower_basic_auth = sync_parallelism = 0 celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG ssl_active = False ssl_key = ssl_cert = ssl_cacert = pool = prefork operation_timeout = 1.0 task_track_started = True task_adoption_timeout = 600 stalled_task_timeout = 0 task_publish_max_retries = 3 worker_precheck = False [celery_broker_transport_options] [dask] cluster_address = 127.0.0.1:8786 tls_ca = tls_cert = tls_key = [scheduler] job_heartbeat_sec = 5 scheduler_heartbeat_sec = 5 num_runs = -1 scheduler_idle_sleep_time = 1 min_file_process_interval = 30 deactivate_stale_dags_interval = 60 dag_dir_list_interval = 300 print_stats_interval = 30 pool_metrics_interval = 5.0 scheduler_health_check_threshold = 30 orphaned_tasks_check_interval = 300.0 child_process_log_directory = /home/airflow/logs/scheduler scheduler_zombie_task_threshold = 300 zombie_detection_interval = 10.0 catchup_by_default = True ignore_first_depends_on_past_by_default = True max_tis_per_query = 512 use_row_level_locking = True max_dagruns_to_create_per_loop = 10 max_dagruns_per_loop_to_schedule = 20 schedule_after_task_execution = True parsing_processes = 2 file_parsing_sort_mode = modified_time standalone_dag_processor = False max_callbacks_per_loop = 20 use_job_schedule = True allow_trigger_in_future = False dependency_detector = airflow.serialization.serialized_objects.DependencyDetector trigger_timeout_check_interval = 15 [triggerer] default_capacity = 1000 [kerberos] ccache = /tmp/airflow_krb5_ccache principal = airflow reinit_frequency = 3600 kinit_path = kinit keytab = airflow.keytab forwardable = True include_ip = True [github_enterprise] api_rev = v3 [elasticsearch] host = log_id_template = {dag_id}-{task_id}-{run_id}-{map_index}-{try_number} end_of_log_mark = end_of_log frontend = write_stdout = False json_format = False json_fields = asctime, filename, lineno, levelname, message host_field = host offset_field = offset [elasticsearch_configs] use_ssl = False verify_certs = True [kubernetes] pod_template_file = worker_container_repository = worker_container_tag = namespace = default delete_worker_pods = True delete_worker_pods_on_failure = False worker_pods_creation_batch_size = 1 multi_namespace_mode = False in_cluster = True kube_client_request_args = delete_option_kwargs = enable_tcp_keepalive = True tcp_keep_idle = 120 tcp_keep_intvl = 30 tcp_keep_cnt = 6 verify_ssl = True worker_pods_pending_timeout = 300 worker_pods_pending_timeout_check_interval = 120 worker_pods_queued_check_interval = 60 worker_pods_pending_timeout_batch_size = 100 [sensors] default_timeout = 604800 [smart_sensor] use_smart_sensor = False shard_code_upper_limit = 10000 shards = 5 sensors_enabled = NamedHivePartitionSensor ***@***.***:~# airflow config list|grep time default_timezone = utc dagbag_import_timeout = 30.0 dag_file_processor_timeout = 50 killed_task_cleanup_time = 60 default_task_execution_timeout = colored_log_format = [%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(log_color)s%(message)s%(reset)s log_format = [%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s simple_log_format = %(asctime)s %(levelname)s - %(message)s default_ui_timezone = UTC web_server_master_timeout = 120 web_server_worker_timeout = 120 log_fetch_timeout_sec = 5 session_lifetime_minutes = 43200 audit_view_excluded_events = gantt,landing_times,tries,duration,calendar,graph,grid,tree,tree_data smtp_timeout = 30 operation_timeout = 1.0 task_adoption_timeout = 600 stalled_task_timeout = 0 scheduler_idle_sleep_time = 1 file_parsing_sort_mode = modified_time trigger_timeout_check_interval = 15 json_fields = asctime, filename, lineno, levelname, message worker_pods_pending_timeout = 300 worker_pods_pending_timeout_check_interval = 120 worker_pods_pending_timeout_batch_size = 100 default_timeout = 604800 ***@***.***:~# airflow config list|grep timezone default_timezone = utc default_ui_timezone = UTC ***@***.***:~# airflow webserver --port 6007 ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ Running the Gunicorn Server with: Workers: 4 sync Host: 0.0.0.0:6007 Timeout: 120 Logfiles: - - Access Logformat: ================================================================= [2024-10-02 00:14:25 +0000] [1730] [INFO] Starting gunicorn 20.1.0 [2024-10-02 00:14:26 +0000] [1730] [INFO] Listening at: http://0.0.0.0:6007 (1730) [2024-10-02 00:14:26 +0000] [1730] [INFO] Using worker: sync [2024-10-02 00:14:26 +0000] [1733] [INFO] Booting worker with pid: 1733 [2024-10-02 00:14:26 +0000] [1734] [INFO] Booting worker with pid: 1734 [2024-10-02 00:14:26 +0000] [1735] [INFO] Booting worker with pid: 1735 [2024-10-02 00:14:26 +0000] [1736] [INFO] Booting worker with pid: 1736 10.232.204.18 - - [02/Oct/2024:00:14:37 +0000] "GET / HTTP/1.1" 302 197 "https://jarvislabs.ai/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" /opt/conda/lib/python3.8/site-packages/airflow/www/views.py:889 SADeprecationWarning: DISTINCT ON is currently supported only by the PostgreSQL dialect. Use of DISTINCT ON for other backends is currently silently ignored, however this usage is deprecated, and will raise CompileError in a future release for all backends that do not support this syntax. [2024-10-02 00:14:37,965] {app.py:1449} ERROR - Exception on /home [GET] Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 2073, in wsgi_app response = self.full_dispatch_request() File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1519, in full_dispatch_request rv = self.handle_user_exception(e) File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1517, in full_dispatch_request rv = self.dispatch_request() File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1503, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) File "/opt/conda/lib/python3.8/site-packages/airflow/www/auth.py", line 46, in decorated return func(*args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/airflow/www/views.py", line 964, in index return self.render_template( File "/opt/conda/lib/python3.8/site-packages/airflow/www/views.py", line 709, in render_template return super().render_template( File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 322, in render_template return render_template( File "/opt/conda/lib/python3.8/site-packages/flask/templating.py", line 154, in render_template return _render( File "/opt/conda/lib/python3.8/site-packages/flask/templating.py", line 128, in _render rv = template.render(context) File "/opt/conda/lib/python3.8/site-packages/jinja2/environment.py", line 1291, in render self.environment.handle_exception() File "/opt/conda/lib/python3.8/site-packages/jinja2/environment.py", line 925, in handle_exception raise rewrite_traceback_stack(source=source) File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/dags.html", line 43, in top-level template code {% elif curr_ordering_direction == 'asc' and request.args.get('sorting_key') == attribute_name %} File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 21, in top-level template code {% from 'airflow/_messages.html' import show_message %} File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 2, in top-level template code {% import 'appbuilder/baselib.html' as baselib %} File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/init.html", line 37, in top-level template code {% block body %} File "/opt/conda/lib/python3.8/site-packages/flask_appbuilder/templates/appbuilder/baselayout.html", line 16, in block 'body' {% block messages %} File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/dags.html", line 104, in block 'messages' {{ super() }} File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 62, in block 'messages' {% call show_message(category='warning', dismissible=false) %} File "/opt/conda/lib/python3.8/site-packages/jinja2/runtime.py", line 828, in _invoke rv = self._func(*arguments) File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/_messages.html", line 25, in template {{ caller() }} File "/opt/conda/lib/python3.8/site-packages/jinja2/runtime.py", line 828, in _invoke rv = self._func(*arguments) File "/opt/conda/lib/python3.8/site-packages/airflow/www/templates/airflow/main.html", line 70, in template {{ macros.datetime_diff_for_humans(scheduler_job.latest_heartbeat) }}</time>. File "/opt/conda/lib/python3.8/site-packages/airflow/macros/__init__.py", line 77, in datetime_diff_for_humans return pendulum.instance(dt).diff_for_humans(since) File "/opt/conda/lib/python3.8/site-packages/pendulum/datetime.py", line 824, in diff_for_humans other = self.now() File "/opt/conda/lib/python3.8/site-packages/pendulum/datetime.py", line 106, in now return pendulum.now(tz) File "/opt/conda/lib/python3.8/site-packages/pendulum/__init__.py", line 211, in now dt = _datetime.datetime.now(local_timezone()) File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/__init__.py", line 60, in local_timezone return get_local_timezone() File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py", line 35, in get_local_timezone tz = _get_system_timezone() File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py", line 63, in _get_system_timezone return _get_unix_timezone() File "/opt/conda/lib/python3.8/site-packages/pendulum/tz/local_timezone.py", line 242, in _get_unix_timezone raise RuntimeError("Unable to find any timezone configuration") RuntimeError: Unable to find any timezone configuration 10.232.204.18 - - [02/Oct/2024:00:14:38 +0000] "GET /home HTTP/1.1" 500 6481 "https://jarvislabs.ai/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" Scheduler log: =============== ***@***.***:~# airflow scheduler ____________ _____________ ____ |__( )_________ __/__ /________ __ ____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / / ___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ / _/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/ [2024-10-02 00:13:55 +0000] [1690] [INFO] Starting gunicorn 20.1.0 [2024-10-02 00:13:55 +0000] [1690] [INFO] Listening at: http://0.0.0.0:8793 (1690) [2024-10-02 00:13:55 +0000] [1690] [INFO] Using worker: sync [2024-10-02 00:13:55 +0000] [1691] [INFO] Booting worker with pid: 1691 [2024-10-02 00:13:55,601] {scheduler_job.py:708} INFO - Starting the scheduler [2024-10-02 00:13:55,601] {scheduler_job.py:713} INFO - Processing each file at most -1 times [2024-10-02 00:13:55,603] {executor_loader.py:105} INFO - Loaded executor: SequentialExecutor [2024-10-02 00:13:55,608] {manager.py:160} INFO - Launched DagFileProcessorManager with pid: 1692 [2024-10-02 00:13:55,610] {scheduler_job.py:1233} INFO - Resetting orphaned tasks for active dag runs [2024-10-02 00:13:55,622] {settings.py:55} INFO - Configured default timezone Timezone('UTC') [2024-10-02 00:13:55,625] {scheduler_job.py:1256} INFO - Marked 1 SchedulerJob instances as failed [2024-10-02 00:13:55,643] {manager.py:406} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2) when using sqlite. So we set parallelism to 1. [2024-10-02 00:13:55 +0000] [1693] [INFO] Booting worker with pid: 1693 [2024-10-02 00:14:00,252] {dag.py:2968} INFO - Setting next_dagrun for Lead_scoring_inference_pipeline to 2024-10-01T23:00:00+00:00, run_after=2024-10-02T00:00:00+00:00 [2024-10-02 00:14:00,302] {dag.py:2968} INFO - Setting next_dagrun for Lead_Scoring_Data_Engineering_Pipeline to 2024-10-02T00:00:00+00:00, run_after=2024-10-03T00:00:00+00:00 [2024-10-02 00:14:00,531] {scheduler_job.py:353} INFO - 2 tasks up for execution: <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.building_db scheduled__2024-10-01T00:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.encoding_categorical_variables scheduled__2024-10-01T07:00:00+00:00 [scheduled]> [2024-10-02 00:14:00,532] {scheduler_job.py:418} INFO - DAG Lead_Scoring_Data_Engineering_Pipeline has 0/16 running and queued tasks [2024-10-02 00:14:00,533] {scheduler_job.py:418} INFO - DAG Lead_scoring_inference_pipeline has 0/16 running and queued tasks [2024-10-02 00:14:00,533] {scheduler_job.py:504} INFO - Setting the following tasks to queued state: <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.building_db scheduled__2024-10-01T00:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.encoding_categorical_variables scheduled__2024-10-01T07:00:00+00:00 [scheduled]> [2024-10-02 00:14:00,592] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_Scoring_Data_Engineering_Pipeline', task_id='building_db', run_id='scheduled__2024-10-01T00:00:00+00:00', try_number=1, map_index=-1) to executor with priority 7 and queue default [2024-10-02 00:14:00,592] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_Scoring_Data_Engineering_Pipeline', 'building_db', 'scheduled__2024-10-01T00:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py'] [2024-10-02 00:14:00,593] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_scoring_inference_pipeline', task_id='encoding_categorical_variables', run_id='scheduled__2024-10-01T07:00:00+00:00', try_number=1, map_index=-1) to executor with priority 4 and queue default [2024-10-02 00:14:00,593] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'encoding_categorical_variables', 'scheduled__2024-10-01T07:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:00,613] {sequential_executor.py:59} INFO - Executing command: ['airflow', 'tasks', 'run', 'Lead_Scoring_Data_Engineering_Pipeline', 'building_db', 'scheduled__2024-10-01T00:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py'] [2024-10-02 00:14:01,312] {dagbag.py:508} INFO - Filling up the DagBag from /home/airflow/dags/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py [2024-10-02 00:14:01,547] {utils.py:145} INFO - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. [2024-10-02 00:14:01,548] {utils.py:157} INFO - NumExpr defaulting to 8 threads. [2024-10-02 00:14:04,527] {lead_scoring_data_pipeline-checkpoint.py:47} INFO - Creating mlflow experiment [2024-10-02 00:14:05,349] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:05,350] {example_local_kubernetes_executor.py:37} WARNING - Could not import DAGs in example_local_kubernetes_executor.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 35, in <module> from kubernetes.client import models as k8s ModuleNotFoundError: No module named 'kubernetes' [2024-10-02 00:14:05,350] {example_local_kubernetes_executor.py:38} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:05,354] {example_python_operator.py:68} WARNING - The virtalenv_python example task requires virtualenv, please install it. [2024-10-02 00:14:05,388] {tutorial_taskflow_api_etl_virtualenv.py:29} WARNING - The tutorial_taskflow_api_etl_virtualenv example DAG requires virtualenv, please install it. [2024-10-02 00:14:05,430] {task_command.py:371} INFO - Running <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.building_db scheduled__2024-10-01T00:00:00+00:00 [queued]> on host b07a3767db24 [2024-10-02 00:14:06,999] {sequential_executor.py:59} INFO - Executing command: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'encoding_categorical_variables', 'scheduled__2024-10-01T07:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:07,807] {dagbag.py:508} INFO - Filling up the DagBag from /home/airflow/dags/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py [2024-10-02 00:14:08,731] {utils.py:145} INFO - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. [2024-10-02 00:14:08,731] {utils.py:157} INFO - NumExpr defaulting to 8 threads. [2024-10-02 00:14:09,066] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:09,067] {example_local_kubernetes_executor.py:37} WARNING - Could not import DAGs in example_local_kubernetes_executor.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 35, in <module> from kubernetes.client import models as k8s ModuleNotFoundError: No module named 'kubernetes' [2024-10-02 00:14:09,068] {example_local_kubernetes_executor.py:38} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:09,071] {example_python_operator.py:68} WARNING - The virtalenv_python example task requires virtualenv, please install it. [2024-10-02 00:14:09,106] {tutorial_taskflow_api_etl_virtualenv.py:29} WARNING - The tutorial_taskflow_api_etl_virtualenv example DAG requires virtualenv, please install it. [2024-10-02 00:14:09,149] {task_command.py:371} INFO - Running <TaskInstance: Lead_scoring_inference_pipeline.encoding_categorical_variables scheduled__2024-10-01T07:00:00+00:00 [queued]> on host b07a3767db24 [2024-10-02 00:14:18,873] {scheduler_job.py:599} INFO - Executor reports execution of Lead_Scoring_Data_Engineering_Pipeline.building_db run_id=scheduled__2024-10-01T00:00:00+00:00 exited with status success for try_number 1 [2024-10-02 00:14:18,873] {scheduler_job.py:599} INFO - Executor reports execution of Lead_scoring_inference_pipeline.encoding_categorical_variables run_id=scheduled__2024-10-01T07:00:00+00:00 exited with status success for try_number 1 [2024-10-02 00:14:18,885] {scheduler_job.py:642} INFO - TaskInstance Finished: dag_id=Lead_scoring_inference_pipeline, task_id=encoding_categorical_variables, run_id=scheduled__2024-10-01T07:00:00+00:00, map_index=-1, run_start_date=2024-10-02 00:14:09.210160+00:00, run_end_date=2024-10-02 00:14:17.928646+00:00, run_duration=8.718486, state=success, executor_state=success, try_number=1, max_tries=1, job_id=698, pool=default_pool, queue=default, priority_weight=4, operator=PythonOperator, queued_dttm=2024-10-02 00:14:00.534756+00:00, queued_by_job_id=696, pid=1716 [2024-10-02 00:14:18,885] {scheduler_job.py:642} INFO - TaskInstance Finished: dag_id=Lead_Scoring_Data_Engineering_Pipeline, task_id=building_db, run_id=scheduled__2024-10-01T00:00:00+00:00, map_index=-1, run_start_date=2024-10-02 00:14:05.489475+00:00, run_end_date=2024-10-02 00:14:05.840342+00:00, run_duration=0.350867, state=success, executor_state=success, try_number=1, max_tries=1, job_id=697, pool=default_pool, queue=default, priority_weight=7, operator=PythonOperator, queued_dttm=2024-10-02 00:14:00.534756+00:00, queued_by_job_id=696, pid=1705 [2024-10-02 00:14:19,041] {dag.py:2968} INFO - Setting next_dagrun for Lead_scoring_inference_pipeline to 2024-10-02T00:00:00+00:00, run_after=2024-10-02T01:00:00+00:00 [2024-10-02 00:14:19,375] {scheduler_job.py:353} INFO - 3 tasks up for execution: <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.rawdata_schema_check scheduled__2024-10-01T00:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.encoding_categorical_variables scheduled__2024-10-01T23:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.checking_input_features scheduled__2024-10-01T07:00:00+00:00 [scheduled]> [2024-10-02 00:14:19,375] {scheduler_job.py:418} INFO - DAG Lead_Scoring_Data_Engineering_Pipeline has 0/16 running and queued tasks [2024-10-02 00:14:19,377] {scheduler_job.py:418} INFO - DAG Lead_scoring_inference_pipeline has 0/16 running and queued tasks [2024-10-02 00:14:19,377] {scheduler_job.py:418} INFO - DAG Lead_scoring_inference_pipeline has 1/16 running and queued tasks [2024-10-02 00:14:19,379] {scheduler_job.py:504} INFO - Setting the following tasks to queued state: <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.rawdata_schema_check scheduled__2024-10-01T00:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.encoding_categorical_variables scheduled__2024-10-01T23:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.checking_input_features scheduled__2024-10-01T07:00:00+00:00 [scheduled]> [2024-10-02 00:14:19,385] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_Scoring_Data_Engineering_Pipeline', task_id='rawdata_schema_check', run_id='scheduled__2024-10-01T00:00:00+00:00', try_number=1, map_index=-1) to executor with priority 6 and queue default [2024-10-02 00:14:19,386] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_Scoring_Data_Engineering_Pipeline', 'rawdata_schema_check', 'scheduled__2024-10-01T00:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py'] [2024-10-02 00:14:19,387] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_scoring_inference_pipeline', task_id='encoding_categorical_variables', run_id='scheduled__2024-10-01T23:00:00+00:00', try_number=1, map_index=-1) to executor with priority 4 and queue default [2024-10-02 00:14:19,387] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'encoding_categorical_variables', 'scheduled__2024-10-01T23:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:19,389] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_scoring_inference_pipeline', task_id='checking_input_features', run_id='scheduled__2024-10-01T07:00:00+00:00', try_number=1, map_index=-1) to executor with priority 3 and queue default [2024-10-02 00:14:19,389] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'checking_input_features', 'scheduled__2024-10-01T07:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:19,411] {sequential_executor.py:59} INFO - Executing command: ['airflow', 'tasks', 'run', 'Lead_Scoring_Data_Engineering_Pipeline', 'rawdata_schema_check', 'scheduled__2024-10-01T00:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py'] [2024-10-02 00:14:20,324] {dagbag.py:508} INFO - Filling up the DagBag from /home/airflow/dags/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py [2024-10-02 00:14:20,569] {utils.py:145} INFO - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. [2024-10-02 00:14:20,570] {utils.py:157} INFO - NumExpr defaulting to 8 threads. [2024-10-02 00:14:25,191] {lead_scoring_data_pipeline-checkpoint.py:47} INFO - Creating mlflow experiment [2024-10-02 00:14:25,518] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:25,520] {example_local_kubernetes_executor.py:37} WARNING - Could not import DAGs in example_local_kubernetes_executor.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 35, in <module> from kubernetes.client import models as k8s ModuleNotFoundError: No module named 'kubernetes' [2024-10-02 00:14:25,520] {example_local_kubernetes_executor.py:38} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:25,524] {example_python_operator.py:68} WARNING - The virtalenv_python example task requires virtualenv, please install it. [2024-10-02 00:14:25,592] {tutorial_taskflow_api_etl_virtualenv.py:29} WARNING - The tutorial_taskflow_api_etl_virtualenv example DAG requires virtualenv, please install it. [2024-10-02 00:14:25,684] {task_command.py:371} INFO - Running <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.rawdata_schema_check scheduled__2024-10-01T00:00:00+00:00 [queued]> on host b07a3767db24 [2024-10-02 00:14:34,723] {sequential_executor.py:59} INFO - Executing command: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'encoding_categorical_variables', 'scheduled__2024-10-01T23:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:37,632] {dagbag.py:508} INFO - Filling up the DagBag from /home/airflow/dags/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py [2024-10-02 00:14:38,717] {utils.py:145} INFO - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. [2024-10-02 00:14:38,717] {utils.py:157} INFO - NumExpr defaulting to 8 threads. [2024-10-02 00:14:39,057] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:39,058] {example_local_kubernetes_executor.py:37} WARNING - Could not import DAGs in example_local_kubernetes_executor.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 35, in <module> from kubernetes.client import models as k8s ModuleNotFoundError: No module named 'kubernetes' [2024-10-02 00:14:39,058] {example_local_kubernetes_executor.py:38} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:39,062] {example_python_operator.py:68} WARNING - The virtalenv_python example task requires virtualenv, please install it. [2024-10-02 00:14:39,097] {tutorial_taskflow_api_etl_virtualenv.py:29} WARNING - The tutorial_taskflow_api_etl_virtualenv example DAG requires virtualenv, please install it. [2024-10-02 00:14:39,149] {task_command.py:371} INFO - Running <TaskInstance: Lead_scoring_inference_pipeline.encoding_categorical_variables scheduled__2024-10-01T23:00:00+00:00 [queued]> on host b07a3767db24 [2024-10-02 00:14:46,811] {sequential_executor.py:59} INFO - Executing command: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'checking_input_features', 'scheduled__2024-10-01T07:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:47,576] {dagbag.py:508} INFO - Filling up the DagBag from /home/airflow/dags/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py [2024-10-02 00:14:48,445] {utils.py:145} INFO - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. [2024-10-02 00:14:48,445] {utils.py:157} INFO - NumExpr defaulting to 8 threads. [2024-10-02 00:14:48,786] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:48,787] {example_local_kubernetes_executor.py:37} WARNING - Could not import DAGs in example_local_kubernetes_executor.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 35, in <module> from kubernetes.client import models as k8s ModuleNotFoundError: No module named 'kubernetes' [2024-10-02 00:14:48,788] {example_local_kubernetes_executor.py:38} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:48,791] {example_python_operator.py:68} WARNING - The virtalenv_python example task requires virtualenv, please install it. [2024-10-02 00:14:48,825] {tutorial_taskflow_api_etl_virtualenv.py:29} WARNING - The tutorial_taskflow_api_etl_virtualenv example DAG requires virtualenv, please install it. [2024-10-02 00:14:48,866] {task_command.py:371} INFO - Running <TaskInstance: Lead_scoring_inference_pipeline.checking_input_features scheduled__2024-10-01T07:00:00+00:00 [queued]> on host b07a3767db24 [2024-10-02 00:14:54,893] {scheduler_job.py:599} INFO - Executor reports execution of Lead_Scoring_Data_Engineering_Pipeline.rawdata_schema_check run_id=scheduled__2024-10-01T00:00:00+00:00 exited with status success for try_number 1 [2024-10-02 00:14:54,893] {scheduler_job.py:599} INFO - Executor reports execution of Lead_scoring_inference_pipeline.encoding_categorical_variables run_id=scheduled__2024-10-01T23:00:00+00:00 exited with status success for try_number 1 [2024-10-02 00:14:54,893] {scheduler_job.py:599} INFO - Executor reports execution of Lead_scoring_inference_pipeline.checking_input_features run_id=scheduled__2024-10-01T07:00:00+00:00 exited with status success for try_number 1 [2024-10-02 00:14:54,899] {scheduler_job.py:642} INFO - TaskInstance Finished: dag_id=Lead_scoring_inference_pipeline, task_id=checking_input_features, run_id=scheduled__2024-10-01T07:00:00+00:00, map_index=-1, run_start_date=2024-10-02 00:14:49.204026+00:00, run_end_date=2024-10-02 00:14:54.036880+00:00, run_duration=4.832854, state=success, executor_state=success, try_number=1, max_tries=1, job_id=701, pool=default_pool, queue=default, priority_weight=3, operator=PythonOperator, queued_dttm=2024-10-02 00:14:19.380465+00:00, queued_by_job_id=696, pid=1758 [2024-10-02 00:14:54,900] {scheduler_job.py:642} INFO - TaskInstance Finished: dag_id=Lead_Scoring_Data_Engineering_Pipeline, task_id=rawdata_schema_check, run_id=scheduled__2024-10-01T00:00:00+00:00, map_index=-1, run_start_date=2024-10-02 00:14:25.895394+00:00, run_end_date=2024-10-02 00:14:30.420493+00:00, run_duration=4.525099, state=success, executor_state=success, try_number=1, max_tries=1, job_id=699, pool=default_pool, queue=default, priority_weight=6, operator=PythonOperator, queued_dttm=2024-10-02 00:14:19.380465+00:00, queued_by_job_id=696, pid=1732 [2024-10-02 00:14:54,900] {scheduler_job.py:642} INFO - TaskInstance Finished: dag_id=Lead_scoring_inference_pipeline, task_id=encoding_categorical_variables, run_id=scheduled__2024-10-01T23:00:00+00:00, map_index=-1, run_start_date=2024-10-02 00:14:39.228254+00:00, run_end_date=2024-10-02 00:14:46.043804+00:00, run_duration=6.81555, state=success, executor_state=success, try_number=1, max_tries=1, job_id=700, pool=default_pool, queue=default, priority_weight=4, operator=PythonOperator, queued_dttm=2024-10-02 00:14:19.380465+00:00, queued_by_job_id=696, pid=1747 [2024-10-02 00:14:55,185] {scheduler_job.py:353} INFO - 3 tasks up for execution: <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.loading_data scheduled__2024-10-01T00:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.checking_input_features scheduled__2024-10-01T23:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.generating_models_prediction scheduled__2024-10-01T07:00:00+00:00 [scheduled]> [2024-10-02 00:14:55,186] {scheduler_job.py:418} INFO - DAG Lead_Scoring_Data_Engineering_Pipeline has 0/16 running and queued tasks [2024-10-02 00:14:55,186] {scheduler_job.py:418} INFO - DAG Lead_scoring_inference_pipeline has 0/16 running and queued tasks [2024-10-02 00:14:55,186] {scheduler_job.py:418} INFO - DAG Lead_scoring_inference_pipeline has 1/16 running and queued tasks [2024-10-02 00:14:55,187] {scheduler_job.py:504} INFO - Setting the following tasks to queued state: <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.loading_data scheduled__2024-10-01T00:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.checking_input_features scheduled__2024-10-01T23:00:00+00:00 [scheduled]> <TaskInstance: Lead_scoring_inference_pipeline.generating_models_prediction scheduled__2024-10-01T07:00:00+00:00 [scheduled]> [2024-10-02 00:14:55,192] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_Scoring_Data_Engineering_Pipeline', task_id='loading_data', run_id='scheduled__2024-10-01T00:00:00+00:00', try_number=1, map_index=-1) to executor with priority 5 and queue default [2024-10-02 00:14:55,193] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_Scoring_Data_Engineering_Pipeline', 'loading_data', 'scheduled__2024-10-01T00:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py'] [2024-10-02 00:14:55,194] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_scoring_inference_pipeline', task_id='checking_input_features', run_id='scheduled__2024-10-01T23:00:00+00:00', try_number=1, map_index=-1) to executor with priority 3 and queue default [2024-10-02 00:14:55,194] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'checking_input_features', 'scheduled__2024-10-01T23:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:55,195] {scheduler_job.py:546} INFO - Sending TaskInstanceKey(dag_id='Lead_scoring_inference_pipeline', task_id='generating_models_prediction', run_id='scheduled__2024-10-01T07:00:00+00:00', try_number=1, map_index=-1) to executor with priority 2 and queue default [2024-10-02 00:14:55,196] {base_executor.py:91} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'generating_models_prediction', 'scheduled__2024-10-01T07:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:14:55,250] {sequential_executor.py:59} INFO - Executing command: ['airflow', 'tasks', 'run', 'Lead_Scoring_Data_Engineering_Pipeline', 'loading_data', 'scheduled__2024-10-01T00:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py'] [2024-10-02 00:14:55,912] {dagbag.py:508} INFO - Filling up the DagBag from /home/airflow/dags/Lead_scoring_data_pipeline/.ipynb_checkpoints/lead_scoring_data_pipeline-checkpoint.py [2024-10-02 00:14:56,141] {utils.py:145} INFO - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. [2024-10-02 00:14:56,141] {utils.py:157} INFO - NumExpr defaulting to 8 threads. [2024-10-02 00:14:59,093] {lead_scoring_data_pipeline-checkpoint.py:47} INFO - Creating mlflow experiment [2024-10-02 00:14:59,253] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:59,254] {example_local_kubernetes_executor.py:37} WARNING - Could not import DAGs in example_local_kubernetes_executor.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 35, in <module> from kubernetes.client import models as k8s ModuleNotFoundError: No module named 'kubernetes' [2024-10-02 00:14:59,254] {example_local_kubernetes_executor.py:38} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:14:59,258] {example_python_operator.py:68} WARNING - The virtalenv_python example task requires virtualenv, please install it. [2024-10-02 00:14:59,291] {tutorial_taskflow_api_etl_virtualenv.py:29} WARNING - The tutorial_taskflow_api_etl_virtualenv example DAG requires virtualenv, please install it. [2024-10-02 00:14:59,548] {task_command.py:371} INFO - Running <TaskInstance: Lead_Scoring_Data_Engineering_Pipeline.loading_data scheduled__2024-10-01T00:00:00+00:00 [queued]> on host b07a3767db24 [2024-10-02 00:15:04,403] {sequential_executor.py:59} INFO - Executing command: ['airflow', 'tasks', 'run', 'Lead_scoring_inference_pipeline', 'checking_input_features', 'scheduled__2024-10-01T23:00:00+00:00', '--local', '--subdir', 'DAGS_FOLDER/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py'] [2024-10-02 00:15:05,080] {dagbag.py:508} INFO - Filling up the DagBag from /home/airflow/dags/Lead_scoring_inference_pipeline/.ipynb_checkpoints/lead_scoring_inference_pipeline-checkpoint.py [2024-10-02 00:15:05,983] {utils.py:145} INFO - Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8. [2024-10-02 00:15:05,983] {utils.py:157} INFO - NumExpr defaulting to 8 threads. [2024-10-02 00:15:06,324] {example_kubernetes_executor.py:39} WARNING - The example_kubernetes_executor example DAG requires the kubernetes provider. Please install it with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:15:06,325] {example_local_kubernetes_executor.py:37} WARNING - Could not import DAGs in example_local_kubernetes_executor.py Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 35, in <module> from kubernetes.client import models as k8s ModuleNotFoundError: No module named 'kubernetes' [2024-10-02 00:15:06,326] {example_local_kubernetes_executor.py:38} WARNING - Install Kubernetes dependencies with: pip install apache-airflow[cncf.kubernetes] [2024-10-02 00:15:06,329] {example_python_operator.py:68} WARNING - The virtalenv_python example task requires virtualenv, please install it. [2024-10-02 00:15:06,367] {tutorial_taskflow_api_etl_virtualenv.py:29} WARNING - The tutorial_taskflow_api_etl_virtualenv example DAG requires virtualenv, please install it. [2024-10-02 00:15:06,412] {task_command.py:371} INFO - Running <TaskInstance: Lead_scoring_inference_pipeline.checking_input_features scheduled__2024-10-01T23:00:00+00:00 [queued]> on host b07a3767db24 MLFLOW : Log: ================== ***@***.***:~/airflow/dags/Lead_scoring_data_pipeline# mlflow server --backend-store-uri='sqlite:///database/lead_scoring_data_cleaning.db' --default-artifact-root="mlruns/" --port=6006 --host=0.0.0.0 [2024-10-02 00:13:43 +0000] [1650] [INFO] Starting gunicorn 20.1.0 [2024-10-02 00:13:43 +0000] [1650] [INFO] Listening at: http://0.0.0.0:6006 (1650) [2024-10-02 00:13:43 +0000] [1650] [INFO] Using worker: sync [2024-10-02 00:13:43 +0000] [1652] [INFO] Booting worker with pid: 1652 [2024-10-02 00:13:44 +0000] [1653] [INFO] Booting worker with pid: 1653 [2024-10-02 00:13:44 +0000] [1654] [INFO] Booting worker with pid: 1654 [2024-10-02 00:13:44 +0000] [1655] [INFO] Booting worker with pid: 1655

[jscheffl]

Conflig looks good but still I feel like the exception is not usual.

Hypothesis: Something is broken in your install. One option would be a clean re-install (don't forget to use the right constraints) and even much better I'd recommend to use a recent version. 2.3.3 for sure will not receive fixes. And the error is so basic that I doubt it is a SW problem but rather something in your setup or environment. Config at least looks "proper".