Ensure optimize is run with liquid_clustered_by #463
Conversation
… tests at least validate that a zorder statement was issued
benc-db
requested review from
andrefurlan-db,
susodapop and
rcypher-databricks
as code owners
tests/integration/base.py
Outdated
| @@ -605,6 +605,12 @@ def _assertTablesEqualSql(self, relation_a, relation_b, columns=None): | |||
|
|
|||
| return sql | |||
|
|
|||
| def assert_in_log(self, message) -> None: | |||
There was a problem hiding this comment.
This is eloquent. Will it make sense to use it elsewhere in the test suite retroactively?
There was a problem hiding this comment.
The only other place this behavior exists today checks for several different messages. For the most part you shouldn't look at the log, since you can just as easily look at stdout with run_and_capture; here I specifically need to look at debug messages, which I'm only doing because I don't know another good way to observe optimize. So short answer is, I may expand this in the future to take multiple messages, at which point that other test can fold in, but I also don't want to encourage log scanning lol.
| def assert_in_log(self, message) -> None: | ||
| log_file = os.path.join(self._logs_dir, "dbt.log") | ||
| with open(log_file, "r") as f: | ||
| log = f.read() |
There was a problem hiding this comment.
One recommendation, run either .upper() or .lower() on both the input and the log message in case of capitalisation differences.
Resolves #448
Description
Discovered that we weren't running optimize with liquid_cluster_by because the optimize clause was only set up to respect z-ordering.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.