Dag Dependency page not showing anything

[SamiRehmanNostra]

Apache Airflow version: 2.1.

Environment: Ubuntu 20.04

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): UBUNTU 20.04 LTS
• Kernel (e.g. uname -a): Linux 20.04.1-Ubuntu SMP Tue Jun 1 09:54:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: python and pip

What happened: After performing the upgrade from 2.0.2 to 2.10 using the guide available in the documentation, Airflow upgraded successfully, Dag dependency page isn't working as expected.
The DAG dependency page doesn't show the dependency graph.

What you expected to happen: I expected the dag dependency page to show the dags and their dependency in a Graph view

How to reproduce it: Its reproduced by opening these pages every time.

How often does this problem occur? Once? Every time etc?
Every time

Any relevant logs to include? Put them here in side a detail tag:

Upgrade Check Log

/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/configuration.py:34
6 DeprecationWarning: The hide_sensitive_variable_fields option in [admin] has been moved to the hide_sensitive_var_conn_fields option in [core] - the old setting has been used, but please update your config.
/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/configuration.py:34
6 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.
/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/plugins_manager.py:
239 DeprecationWarning: This decorator is deprecated.In previous versions, all subclasses of BaseOperator must use apply_default decorator for thedefault_args feature to work properly.
In current version, it is optional. The decorator is applied automatically using the metaclass.
/home/ubuntu/env_airflow/lib/python3.8/site-packages/airflow/configuration.py:34
6 DeprecationWarning: The default_queue option in [celery] has been moved to the default_queue option in [operators] - the old setting has been used, but please update your config.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[bbovenzi]

@SamiRehmanNostra Could you open the browser console and screenshot any error messages in the logs?

Edit:
I was able to reproduce on a fresh build. Please check that your scheduler is running. It seems that the scheduler is required to load the dag dependency data. We should see if we can get the data with just the webserver.

[SamiRehmanNostra]

Hi @bbovenzi ,

Please see the screenshot attached below:

d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,20)scale(0)". i @ d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". i @ d3.min.js:1 d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". i @ d3.min.js:1 d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". i @ d3.min.js:1 d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". i @ d3.min.js:1 d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". i @ d3.min.js:1 d3.min.js:1 Error: <g> attribute transform: Expected number, "translate(NaN,NaN)scale(0)". i @ d3.min.js:1
Let me know if you need any more information.

Thank you,
Sami

[ashb]

@SamiRehmanNostra Do you by chance have no dags at all?

[SamiRehmanNostra]

@ashb I have about 13 active dags, all running on different schedules.
Weirdly enough I was installing a bunch of ODBC drivers today and restarted the web-server about 3 times. The issue seems to have resolved itself.

Not sure what has changed. Let me know if I need to share any logs.

[ashb]

Hmmm tricky. Without having a reliable way of reproducing this it's going to be tricky to fix.

Did you clear the DB when restarting the webserver? (Not likely, just checking)

[bbovenzi]

@ashb I have been able to reliably replicate it. If I only run the webserver the returned nodes and edges will be empty arrays. As soon as I run the scheduler both will populate and the graph will render.

[ashb]

@bbovenzi Cool -- lets protect against that case as it is a cause of this problem.

[bbovenzi]

Yeah, I'll add some sort of "Dependency data could not be loaded right now" error message.

[SamiRehmanNostra]

@ashb I didn't reset the DB. Seems like @bbovenzi has narrowed it down. Thank you

[SamiRehmanNostra]

@bbovenzi Its back people!

[ManiBharataraju]

@bbovenzi @ashb - It appears like for a fresh build this piece of code is setting value as None.

airflow/airflow/serialization/serialized_objects.py

Line 682 in 9a1fc19

serialize_dag["dag_dependencies"] = [

We just upgraded to 2.1.1 and see the dag dependencies empty. Tried restarting the scheduler multiple times but doesn't help.

Could it be possible that task_type in the below piece of code is BaseOperator instead of the actual operator? IDK, haven't been able to reproduce this issue in my local.

airflow/airflow/serialization/serialized_objects.py

Line 341 in 9a1fc19

if task.task_type == "TriggerDagRunOperator":

I see this error message on the UI. Not sure if it is an underlying UI code issue or issue with the way things are being set.

[xiaoxugao]

Hey, is this issue solved? I still have the same problem as @SamiRehmanNostra .

[SamiRehmanNostra]

Hi @xiaoxugao ,

I might need to open another issue, I upgraded to 2.2.0 I initially had no issues, but over the last couple of weeks been having some issues again.

Not sure what the error is apart from this JS error.

Thank you,
Sami

[bbovenzi]

We can just reopen this issue. You're right that it should've been fixed in 2.2.0. Let me replicate and see if something different is going on.

[TerryYin1777]

any updates on this? I have the same issue with @SamiRehmanNostra. I'm on airflow 2.2.2

[bbovenzi]

any updates on this? I have the same issue with @SamiRehmanNostra. I'm on airflow 2.2.2

Yes, I finally got back around to debugging this and I think I found a fix.

[rvatsastro]

working as expected now

[sthakkar-rdc]

@rvatsastro thanks for reporting. did you have to do anything to resolve the issue ? We are on MWAA 2.2.2 and still seeing the same issue

[leozitor]

Hey guys is anyone facing issues regarding this, I notices in my code that if i have a TriggerDagRunOperator inside task_group like this, it shows the dag dependency on viewer

@task_group(group_id='concat_video')
    def concatenate_video(task_id: int):
        print("triggering job")
        submit_job = TriggerDagRunOperator(
            task_id='submit_job',
            trigger_dag_id='video_concatenation_job',
            wait_for_completion=True,
            deferrable=True,

            poke_interval=10,
            conf={
                    'task_id': task_id,
                },
            )

        submit_job

but if I use without a task_group ( for this example doesn't make sense to use a group for a unique task)
it doesn't show in dag dependency viewer

@task
def concatenate_video(task_id: int):
         print("triggering job")
         submit_job = TriggerDagRunOperator(
          task_id='submit_job',
          trigger_dag_id='video_concatenation_job',
          wait_for_completion=True,
          deferrable=True,
          # max_active_tis_per_dag=2, # it may be used if not deferrable
          poke_interval=10,
          conf={
                  'task_id': task_id,
              },
          )

      submit_job

Does this make a sense?