Don't change the state of Fivetran asset if nothing changes in the data.

[marcilj]

Would it be possible to avoid updating the status of an asset handled by Dagster-Fivetran if there's nothing that changes in the data.

That could be done by adding a column in the metadata block like we are doing for partition_expr in snowflake.

That could also be done automatically since we know all table will have a column called _FIVETRAN_SYNCED. So adding the logic to compare _FIVETRAN_SYNCED with the last one store could allow us to validate if the data changed or not.

One of the problem I have with it being updated, it that all my downstream asset are getting tagged as Upstream data changed and that's triggering materialization of assets since I've configure auto materialization policies on them.

We should have a way to identify if the data changed or not or at least be able to overwrite the asset function to validate that ourselves.

Thank you very much.

[smackesey]

cc @OwenKephart

[OwenKephart]

Hi @marcilj -- this seems possible, although probably not something that would be built in by default into the integration, because it would involve creating some additional scheme to store / compare previous _FIVETRAN_SYNCED values.

There don't currently exist great hooks for modifying the behavior here, but the specific way I'd go about implementing this would be to modify the behavior of the generate_materializations function here: https://sourcegraph.com/github.com/dagster-io/dagster/-/blob/python_modules/libraries/dagster-fivetran/dagster_fivetran/asset_defs.py?L96:32-96:57

In particular, you'd just want it to not yield a materialization for any table where the _FIVETRAN_SYNCED value did not change.