No datasets emitted in DbtLocalBaseOperator after Cosmos upgrade to 1.3.0

[jakob-hvitnov-telia]

After updating Cosmos from 1.1.3 to 1.3.0 we have run into an issue where we are no longer seeing emission of datasets when using theDbtRunLocal operator.
At the same update we went from Airflow 2.6.3 to 2.7.2.

Dag log output (from dbt_run task) before the update:

[2024-01-04 12:49:06] {{local.py:238}} INFO - Inlets: [Dataset(uri='correct_uri_to_inlet01_anonymized', extra=None), Dataset(uri='correct_uri_to_inlet02_anonymized', extra=None), Dataset(uri='correct_uri_to_inlet03_anonymized', extra=None), Dataset(correct_uri_to_inlet04_anonymized', extra=None)]
[2024-01-04, 23:10:00] CET {{local.py:239}} INFO - Outlets: [Dataset(uri='correct_uri_to_outlet01_anonymized', extra=None), Dataset(uri='correct_uri_to_outlet01_anonymized', extra=None), Dataset(uri='correct_uri_to_outlet01_anonymized', extra=None), Dataset(uri='correct_uri_to_outlet01_anonymized', extra=None)]

Dag log output after the update:

[2024-01-05 10:26:17] {{local.py:238}} INFO - Inlets: []
[2024-01-05 10:26:17] {{local.py:239}} INFO - Outlets: []

We are running on AWS MWAA.

I suspect the change in has something to do with this commit as this introduces the get_datasets method, which in its docstring clarifies that the method relies on:

• profiles
• {project_dir}/target/manifest.json
• {project_dir}/target/run_results.json

It is not quite clear to me if these are generated by our call to DbtRunLocal at the moment.
There might be configuration options I am overlooking:

dbt_run = DbtRunLocalOperator(
        profile_config=profile_config,
        project_dir= os.path.join(PurePath(__file__).parent.parent.parent.parent, "dbt"),
        task_id="dbt_run",
        select="path:path/to/our/dbt/models",
        install_deps=False,
        emit_datasets=True
        )

Docstring typo

Also in DbtLocalBaseOperator there seems to be an error in the docstring (lines 97-98) where it says:

:param install_deps: If true, install dependencies before running the command
:param install_deps: If true, the operator will set inlets and outlets

I think the phrasing should probably be:

:param install_deps: If true, install dependencies before running the command
:param emit_datasets: If true, the operator will set inlets and outlets

A fix for this is in https://github.com/astronomer/astronomer-cosmos/pull/797

[dosubot]

🤖

That sounds like a great proposal! It's clear that you've put thought into the issue and have a solid plan for addressing it. Encouraging user contributions is a fantastic way to improve Cosmos for everyone, and your proposed solution seems like a step in the right direction. I encourage you to go ahead and create a pull request with your proposed changes. Your contribution will be valuable to the Cosmos community. Keep up the great work!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[tatiana]

@jakob-hvitnov-telia do things work as expected if you install apache-airflow-providers-openlineage or if you use install astronomer-cosmos[openlineage]?

[jakob-hvitnov-telia]

@tatiana installing apache-airflow-providers-openlineage did the trick!

So in the end it seems the Airflow update was more the culprit than the Cosmos update.

Thank you very much for your help!