[Bug] [1.4.0a] Fix #869 - Cosmos does not set dbt project dir to the tmp dir

[dwreeves]

Fix #869.

Version of dbt I was running is 1.6.4. It's unclear to me if this is specific to that version, but I couldn't get the Dbt docs operator to work without this change. With it, it works, and I do not see any other issues.

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	64f2d4e
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/65e6a3b6cdc32000084617df

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.39%. Comparing base (fbd61ec) to head (64f2d4e).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #873   +/-   ##
=======================================
  Coverage   95.39%   95.39%           
=======================================
  Files          57       57           
  Lines        2759     2759           
=======================================
  Hits         2632     2632           
  Misses        127      127           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dwreeves]

May want to hold off on merging this. I'm running into another issue that suggests this doesn't fully fix the full problem. TLDR: I hit this error when running the "fixed" version of the DbtDocsS3Operator:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpsw4m1r00/target/graph.gpickle'

Note this happens after it successfully uploads manifest.json.

So what's going on? This suggests to me that what is happening is: it's actually just pulling what already exists in my dags/dbt/target directory and isn't actually updating anything with the dbt docs generate command. Since for my deployment, I run dbt compile && dbt parse, which crucially does produce a manifest.json but does not produce a graph.gpickle. So that's how that error can occur: I feel like the command dbt docs generate is not writing to /tmp/tmpsw4m1r00/target; if it sees any files there it's because they were copied from dags/dbt/target prior to executing the commands.

Frankly though, it's still a little confusing because I see this in the logs, too:

[2024-03-06, 00:19:30 UTC] {log.py:232} WARNING - [�[34m2024-03-06T00:19:30.622+0000�[0m] {�[34mlocal.py:�[0m247} INFO�[0m - �[35m(astronomer-cosmos)�[0m - 00:19:30  Catalog written to /tmp/tmpsw4m1r00/target/catalog.json

So why is catalog.json ostensibly being written according to these logs, but not graph.gpickle? Beats me. I will have to experiment this weekend.

Also, to be clear, this is an improvement over the original code. So what exists in this PR is a step in the right direction, just not the full picture.

This is possibly related to #880. (Which does NOT use a "fixed" version of the operator.) It seems like something about 1.4.0a is really messing with how dbt (1.6) uses the file system.

[tatiana]

Thanks for fixing this, @dwreeves !