[CT-3011] [Regression] dbt 1.6 implicitly overwrites the project variables of imported projects

[gunnava]

Is this a regression in a recent version of dbt-core?

• I believe this is a regression in dbt-core functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

dbt 1.6 implicitly overwrites the project variables of imported projects when the source.yml file lies within a different project that is imported via the packages.yml.

There is a production_db project variable in both projects and when the la_lake project is imported, the production_db variable for the la_lake sources changes to the value set in the la_production_data project.

vars are configured as below:
In the current project (la_production_data):

production_db: 'la_production_data'

Imported project (la_lake):

production_db: 'la_lake'

The below compiles to select * from la_production_data_dev_anja_reuter.raw_production_data_tdb.tdb_device_header in dbt 1.6. However in dbt 1.5 it compiles correctly to select * from la_lake.raw_production_data_tdb.tdb_device_header and refers to the imported project's variable.

When I explicitly overwrite the variable in the project.yml with the original value like below, the sources compile like they did in previous versions.

Expected/Previous Behavior

When a source from an imported package is referenced, it should use the var from the src project as it does in dbt 1.5 and should not implicitly overwrite it.

Steps To Reproduce

1. Define a source in an imported project
2. Define the same var in both the imported package and the parent package.
3. Reference the source from the imported package in the current project
4. Compile it in dbt 1.5 and dbt 1.6 to see the difference.

Relevant log output

In dbt 1.6 logs, the model is compiled to:
 
source as (
select * from laproductiondatadevanjareuter.rawproductiondatatdb.tdbdeviceheader
  select * from la_production_data_dev_anja_reuter.raw_production_data_tdb.tdb_device_header
 
Whereas in the lower version it is compiled to 
select * from la_lake.raw_production_data_tdb.tdb_device_header

Environment

- OS:
- Python:
- dbt (working version): 1.5
- dbt (regression version): 1.6

Which database adapter are you using with dbt?

No response

Additional Context

No response

[jtcohen6]

Hey @gunnava, thanks for opening! This was an intentional change in v1.6, to resolve a longstanding inconsistency in the precedence rules for vars inheritance:

• [CT-1888] [Bug] Package project yml configs overriding root project configs when used in other yml files #6705

With the changes in v1.6, you now have full control from the root-level dbt_project.yml — similar to how you can (re)configure any resources installed from a package (docs). You can then:

• Set that variable's value everywhere
• Or, set that variable's value for specific projects / installed packages

See the verification case below for details. I'm going to close this as not a bug / something that we won't fix. While it will be a behavior change for some folks upgrading to v1.6, I believe we are providing a better experience with greater optionality for users. We have already called this out in the migration guide (https://docs.getdbt.com/guides/migration/versions/upgrading-to-v1.6#multi-project-collaboration), although we could mention it higher up in the section on "Behavior Changes."

Verification case

# localpkg/dbt_project.yml
name: "my_package"
vars:
  my_variable: 1234

-- models/my_model.sql
select {{ var("my_variable") }} as my_variable

# packages.yml
packages:
  - local: localpkg

# dbt_project.yml ("root")
name: "my_dbt_project"
profile: <profile_name>

# empty to start
vars: {}

$ dbt compile -s my_model
...
select 123 as my_variable

Then try modifying the vars in the "root" dbt_project.yml:

# set 'my_variable' everywhere
vars:
  my_variable: 456
  
# set 'my_variable' only for the root project -- leave the package's value in place
vars:
  my_dbt_project:
    my_variable: 789

# set 'my_variable' to a new value, only for the package
vars:
  my_package:
    my_variable: 012