Run optimize after Liquid Clustering

[dbarrundia-tiger]

Disclaimer: this might be a question and not really a feature request.

Describe the feature

In the liquid clustering docs: https://docs.databricks.com/en/delta/clustering.html#how-to-trigger-clustering it says we should trigger OPTIMIZE table_name; to trigger clustering. So I am thinking if a model has liquid_clustered_by then we should run OPTIMIZE table_name; also

Describe alternatives you've considered

Somehow using this? https://github.com/databricks/dbt-databricks/blob/main/dbt/include/databricks/macros/adapters.sql#L368

Additional context

Please include any other relevant context here.

Who will this benefit?

Everyone using liquid_clustered_by

Are you interested in contributing this feature?

Let us know if you want to write some code, and how we can help.

[dbarrundia-tiger]

CC: @benc-db to see if you have any thoughts?

[benc-db]

I need to give this some thought, as from the docs this is something that needs to be done on on-going basis. We could use optimize on every run, but is that too often?

[benc-db]

Also, I need to dig a little more to understand the behavior today, because just inspecting the code, I would expect this

dbt-databricks/dbt/include/databricks/macros/materializations/table.sql

Line 34 in a6ddca9

{% do optimize(target_relation) %}

or
this

dbt-databricks/dbt/include/databricks/macros/materializations/incremental/incremental.sql

Line 89 in a6ddca9

{% do optimize(target_relation) %}

would already be running optimize

[dbarrundia-tiger]

I need to give this some thought, as from the docs this is something that needs to be done on on-going basis. We could use optimize on every run, but is that too often?

That's what i thought also, but it's not running the optimize, meaning that macro is somehow not running as expected?

[benc-db]

Will investigate; changing the tag to 'bug' because I believe that the correct operation of this feature includes running optimize.