Allow models to execute on different warehouses #488
Conversation
rcypher-databricks
requested review from
andrefurlan-db,
susodapop and
benc-db
as code owners
rcypher-databricks
force-pushed
the
main
branch
from
135d4e6
to
ef89e18
Compare
|
Out of curiosity, how did you validate? Do you have tests that are uncommitted, or did you look at the logs of some existing tests? |
|
I verified by modifying one of the integration tests and stepping through the code as well as adding some temporary print statements. I validated the open session parameters for a model that specifies a different warehouse than the one in the profile. |
|
Stylistically looks good, and to the extent I can follow, the logic looks right. Assuming we go this way, please add unit tests for the new logic, and at least 1 functional test that uses the compute definitions from the existing profiles. |
Per discussion with dbt, we're going to try to finish migrating to their functional tests, so no new integration tests. |
|
Definitely going to add tests if this is the implementation we decide to use. That's why its a draft PR at this point |
Signed-off-by: Raymond Cypher <[email protected]>
Raise an exception if a model specifies a compute resource that is not defined in the profile. Signed-off-by: Raymond Cypher <[email protected]>
Signed-off-by: Raymond Cypher <[email protected]>
rcypher-databricks
force-pushed
the
main
branch
from
fbe3f25
to
e1f89c0
Compare
| return {"model_names": ["target3"]} | ||
|
|
||
|
|
||
| class TestSpecifyingForProjectModelsInFolder(BaseSpecifyingCompute): |
There was a problem hiding this comment.
Do we know if we can specify a compute to use with models of a particular tag? This came up in a customer call where they would want to tag certain models as heavy_compute for example.
There was a problem hiding this comment.
After reading the dbt docs on tags, I don't think that would work, which is probably fine. I think having the named compute approach gets us 95% of the way to what it would be if they could target compute to tags.
Resolves PECO-1182
Description
Allows models in a dbt project to run on different SQL warehouses.
The available warehouses are specified as named values in the dbt profile.
The configuration for a model can specify by name which warehouse it should run on.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.