[Bug] Custom materialization not respected when passed via dbt_project.yaml or model.yaml

[rbs392]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

models/dummy/test.py

def model(dbt, session):
    dbt.config(materialized="test_mat")
    return None

dbt_project.yml

...
models:
  dummy:
    test:
      +materialized: test_mat

When the dbt.config(materialized="test_mat") is removed from models/dummy/test.py the materialization code is not called

Expected Behavior

materialized config is respected when specified in dbt_project.yml and model.yml not only in the actual model

Steps To Reproduce

https://gist.github.com/rbs392/5e9d4a29aed63e385839c4f354ce0b56

Relevant log output

when run with materialized config from dbt_project.yml

12:43:33  Running with dbt=1.7.9
12:43:34  Registered adapter: databricks=1.7.8
12:43:34  Unable to do partial parsing because a project config has changed
12:43:35  Found 1 model, 0 sources, 0 exposures, 0 metrics, 536 macros, 0 groups, 0 semantic models
12:43:35
12:43:36  Concurrency: 1 threads (target='dev')
12:43:36
12:43:36  1 of 1 START python table model dummy.test ...................................... [RUN]
12:43:36  Databricks adapter: Job submission response={'id': '1079e76b1aa64c14b84d407a13fbfe4f'}
12:43:47  1 of 1 ERROR creating python table model dummy.test ............................. [ERROR in 10.99s]
12:43:47
12:43:47  Finished running 1 table model in 0 hours 0 minutes and 12.51 seconds (12.51s).
12:43:47
12:43:47  Completed with 1 error and 0 warnings:

when run with materialized set in model file

12:44:03  Running with dbt=1.7.9
12:44:05  Registered adapter: databricks=1.7.8
12:44:05  Found 1 model, 0 sources, 0 exposures, 0 metrics, 536 macros, 0 groups, 0 semantic models
12:44:05
12:44:05  Concurrency: 1 threads (target='dev')
12:44:05
12:44:05  1 of 1 START python test_mat model dummy.test ................................... [RUN]
12:44:05  1 of 1 ERROR creating python test_mat model dummy.test .......................... [ERROR in 0.04s]
12:44:06
12:44:06  Finished running 1 test_mat model in 0 hours 0 minutes and 1.40 seconds (1.40s).
12:44:06
12:44:06  Completed with 1 error and 0 warnings:

Environment

- OS: Mac os
- Python: 3.9.5
- dbt: 1.7.9
- dbt-databricks: 1.7.8

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

dbt-databricks

[dbeatty10]

Thanks for reporting this @rbs392 and providing a nice example @rbs392 👍

I was able to reproduce what you described using dbt-duckdb, and I've included a reproduction case for each below:

1. materialized config in dbt_project.yml works for SQL models, but not for python models.
2. materialized config in a .yml works for SQL models, but not for python models.

Both of these are stated as being supported options in the documentation here.

Reprex

Case 1: dbt_project.yml

models:
  my_project:
    dummy:
      test:
        +materialized: test_mat

Case 2: models/_models.yml

models:
  - name: test
    config:
      materialized: test_mat

Executing dbt run -s test should give the following error message:

Compilation Error in model test (models/dummy/test.sql)
  Not implemented

Custom materialization macro

macros/mat.sql

{% materialization test_mat, supported_languages=['python', 'sql'] %}
    {% do exceptions.raise_compiler_error("Not implemented") %}
{% endmaterialization %}

SQL model vs. Python model

Both of the two cases above behave as expected when there is a SQL named models/dummy/test.sql like this:

select 1 as id

But neither behaves as expected when there is a Python named models/dummy/test.py like this:

def model(dbt, session):
    return None

[rbs392]

Hi @dbeatty10 thanks for the update. Any timeline on when the issue is expected to be fixed?

[dbeatty10]

@rbs392 We don't have any timeline for this. In the meantime, we'd suggest using the workaround you identified.

Workaround

Add the materialized config into the model-level configuration with the .py file like this:

models/dummy/test.py

def model(dbt, session):
    dbt.config(materialized="test_mat")
    return None

[dbeatty10]

This might have the same root cause as #8520, and it might be resolved by #8538

[devmessias]

Hi @rbs392, at my company we faced the same issue and we also work with Databricks. My fix in #8538 work for us, but the PR needs a rebase because we are not using version 1.7 of dbt-core.

[dbeatty10]

Thanks for your PR in #8538 @devmessias ! We've got it queued up for an initial code review.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[devmessias]

due github-actions: The issue has been solved but the PR must be merged