[CT-3320] [Regression] DBT deps no longer working on git repository with subdirectory

[DanMawdsleyBA]

Is this a regression in a recent version of dbt-core?

• I believe this is a regression in dbt-core functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

Install dbt deps with a github repository using a subdirectory no longer works. I was using 1.5.5 previously but since upgrading to 1.7 deps is no longer working.

Expected/Previous Behavior

To be able to run deps from a GitHub repository like previously.

Steps To Reproduce

Run the dbt deps command with a packages file having a dependency on github repository with a subdirectory

Relevant log output

�[0m12:55:54.901755 [debug] [MainThread]: Checking out revision BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes.
�[0m12:55:54.903756 [debug] [MainThread]: Executing "git remote set-branches origin BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes"
�[0m12:55:55.090351 [debug] [MainThread]: STDOUT: "b''"
�[0m12:55:55.090351 [debug] [MainThread]: STDERR: "b''"
�[0m12:55:55.092917 [debug] [MainThread]: Executing "git fetch origin --depth 1 --tags BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes"
�[0m12:56:04.606659 [debug] [MainThread]: STDOUT: "b''"
�[0m12:56:04.607668 [debug] [MainThread]: STDERR: "b'From https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning\n * branch            BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes -> FETCH_HEAD\n'"
�[0m12:56:04.607668 [debug] [MainThread]: Executing "git tag --list"
�[0m12:56:04.766131 [debug] [MainThread]: STDOUT: "b''"
�[0m12:56:04.766131 [debug] [MainThread]: STDERR: "b''"
�[0m12:56:04.766131 [debug] [MainThread]: Executing "git reset --hard origin/BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes"
�[0m12:56:05.199503 [debug] [MainThread]: STDOUT: "b'HEAD is now at b47a4a3 Trivial change\n'"
�[0m12:56:05.200510 [debug] [MainThread]: STDERR: "b''"
�[0m12:56:05.200510 [debug] [MainThread]: Executing "git rev-parse HEAD"
�[0m12:56:05.350800 [debug] [MainThread]: STDOUT: "b'b47a4a39ce265e40c302b737ca74c2e545ac39a4\n'"
�[0m12:56:05.351802 [debug] [MainThread]: STDERR: "b''"
�[0m12:56:05.351802 [debug] [MainThread]: Already at b47a4a3, nothing to do.
�[0m12:56:05.618757 [info ] [MainThread]: Updating lock file in file path: C:\Users\u236618\Documents\Github\olympus-utils\olympus_dbt\olympus_dbt_master/package-lock.yml
�[0m12:56:05.638316 [debug] [MainThread]: Set downloads directory='C:\Users\u236618\AppData\Local\Temp\dbt-downloads-uc2bak5s'
�[0m12:56:05.641309 [info ] [MainThread]: Installing https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning.git/src/sql/snowflake/airflow/dbt
�[0m12:56:05.644772 [debug] [MainThread]: Executing "git clone --depth 1 https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning.git/src/sql/snowflake/airflow/dbt 9882dd5f21e29810ec7cbc2c59917d99"
�[0m12:56:06.530265 [debug] [MainThread]: STDOUT: "b''"
�[0m12:56:06.531274 [debug] [MainThread]: STDERR: "b"Cloning into '9882dd5f21e29810ec7cbc2c59917d99'...\nremote: Not Found\nfatal: repository 'https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning.git/src/sql/snowflake/airflow/dbt/' not found\n""
�[0m12:56:06.533262 [debug] [MainThread]: command return code=128
�[0m12:56:06.535642 [error] [MainThread]: Encountered an error:
Internal Error
  Error checking out spec='None' for repo https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning.git/src/sql/snowflake/airflow/dbt
  Cloning into '9882dd5f21e29810ec7cbc2c59917d99'...
  remote: Not Found
  fatal: repository 'https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning.git/src/sql/snowflake/airflow/dbt/' not found
�[0m12:56:06.545933 [debug] [MainThread]: Command `dbt deps` failed at 12:56:06.544931 after 58.57 seconds

Environment

- OS:Windows 10
- Python: 3.11
- dbt (working version): 1.5.5
- dbt (regression version): 1.7

Which database adapter are you using with dbt?

snowflake

Additional Context

Below is the log output from 1.5.5

�[0m11:09:50.697690 [debug] [MainThread]: Checking out revision BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes.
�[0m11:09:50.698687 [debug] [MainThread]: Executing "git remote set-branches origin BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes"
�[0m11:09:50.797694 [debug] [MainThread]: STDOUT: "b''"
�[0m11:09:50.798703 [debug] [MainThread]: STDERR: "b''"
�[0m11:09:50.799693 [debug] [MainThread]: Executing "git fetch origin --depth 1 --tags BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes"
�[0m11:09:56.508206 [debug] [MainThread]: STDOUT: "b''"
�[0m11:09:56.508206 [debug] [MainThread]: STDERR: "b'From https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning\n * branch            BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes -> FETCH_HEAD\n'"
�[0m11:09:56.508206 [debug] [MainThread]: Executing "git tag --list"
�[0m11:09:56.592886 [debug] [MainThread]: STDOUT: "b''"
�[0m11:09:56.592886 [debug] [MainThread]: STDERR: "b''"
�[0m11:09:56.592886 [debug] [MainThread]: Executing "git reset --hard origin/BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes"
�[0m11:09:56.693193 [debug] [MainThread]: STDOUT: "b'HEAD is now at 3f5a85e improved meta fields\n'"
�[0m11:09:56.693193 [debug] [MainThread]: STDERR: "b''"
�[0m11:09:56.693193 [debug] [MainThread]: Executing "git rev-parse HEAD"
�[0m11:09:56.777840 [debug] [MainThread]: STDOUT: "b'3f5a85e8e1571b545fdc52f4ca4701c39173dbe9\n'"
�[0m11:09:56.777840 [debug] [MainThread]: STDERR: "b''"
�[0m11:09:56.777840 [debug] [MainThread]: Already at 3f5a85e, nothing to do.
�[0m11:09:56.777840 [info ] [MainThread]: Installed from revision BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes

[jaypeedevlin]

Seems possibly related to what's described in this comment: #6643 (comment)

[graciegoheen]

Thank you for opening @DanMawdsleyBA would you be able to provide the code for your packages.yml file? Are you using the subdirectory: config?

[DanMawdsleyBA]

Yeah I am using the subdirectory see below:

packages:
  - git: "https://github.com/BritishAirways-Ent/olympus-oprn-crw-ops-planning.git"
    revision: "BADICREW-1-inc-0143754-easa-cabin-flight-crew-dashboard-data-hermes"
    subdirectory: "src/sql/snowflake/airflow/dbt"

[leo-schick]

@DanMawdsleyBA The way you do this looks wrong. Can you read https://docs.getdbt.com/docs/build/packages#private-packages and change your package file accordingly?

[DanMawdsleyBA]

@leo-schick This was previously working package on 1.5.5 so I would expect it to work on 1.7 . As mentioned by @jaypeedevlin I think it involves an issue with the package lock file incorrectly using sub directories (which didn't exist in 1.5.5)

[katieclaiborne]

I can confirm that we've experienced a similar issue when upgrading to version 1.7.

[graciegoheen]

Reproducible example

Create a packages.yml file with a subdirectory:

packages:
  - git: "https://github.com/dbt-labs/dbt-labs-experimental-features.git"
    revision: "main"
    subdirectory: "business-hours"

Execute dbt deps

In 1.6, package is installed successfully:

16:02:51  Running with dbt=1.6.1
16:02:54  WARNING: The git package "https://github.com/dbt-labs/dbt-labs-experimental-features.git" 
        is None.
        This can introduce breaking changes into your project without warning!

See https://docs.getdbt.com/docs/package-management#section-specifying-package-versions
16:02:54  WARNING: The git package "https://github.com/dbt-labs/dbt-labs-experimental-features.git" 
        is None.
        This can introduce breaking changes into your project without warning!

See https://docs.getdbt.com/docs/package-management#section-specifying-package-versions
16:02:54  Installing https://github.com/dbt-labs/dbt-labs-experimental-features.git
16:02:55  Installed from revision main
16:02:55  and subdirectory business-hours
16:02:55  Installing dbt-labs/dbt_utils
16:02:56  Installed from version 0.8.0
16:02:56  Updated version available: 1.1.1
16:02:56  
16:02:56  Updates available for packages: ['dbt-labs/dbt_utils']                 
Update your versions in packages.yml, then run dbt deps

In 1.7, get the following error:

16:01:20  Running with dbt=1.7.0
16:01:23  WARNING: The git package "https://github.com/dbt-labs/dbt-labs-experimental-features.git" 
        is None.
        This can introduce breaking changes into your project without warning!

See https://docs.getdbt.com/docs/package-management#section-specifying-package-versions
16:01:26  WARNING: The git package "https://github.com/dbt-labs/dbt-labs-experimental-features.git" 
        is None.
        This can introduce breaking changes into your project without warning!

See https://docs.getdbt.com/docs/package-management#section-specifying-package-versions
16:01:26  Updating lock file in file path: /Users/gracegoheen/dev/coalesce-ci-demo-2023/package-lock.yml
16:01:26  Installing https://github.com/dbt-labs/dbt-labs-experimental-features.git/business-hours
16:01:26  Encountered an error:
Internal Error
  Error checking out spec='None' for repo https://github.com/dbt-labs/dbt-labs-experimental-features.git/business-hours
  Cloning into '1eae545a2334eb5be7c8098e9c10fd77'...
  remote: Not Found
  fatal: repository 'https://github.com/dbt-labs/dbt-labs-experimental-features.git/business-hours/' not found