You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
AutomateDV assumes that the data being pointed at the macros is always deltas. In our architecture, we have a data lake where we create an event base history for each source table. We'd like to point macros at this (and can for the most part) but apply_source_filter is not across all of the macros.
Describe the solution you'd like
I'd like to have the apply_source_filter config available on all macros. This would allow idempotent builds over event base history tables from source where we can simply restart jobs, or full-refresh the data vault should issues arise. I would expect this to be available on all macros excluding possibly hubs and links.
Describe alternatives you've considered
The config is already there for the main sat macro, but I don't see any harm adding it to more as an option so there is the flexibility to limit the data considered by the macro rather than having to create a delta layer.
Is your feature request related to a problem? Please describe.
AutomateDV assumes that the data being pointed at the macros is always deltas. In our architecture, we have a data lake where we create an event base history for each source table. We'd like to point macros at this (and can for the most part) but apply_source_filter is not across all of the macros.
Describe the solution you'd like
I'd like to have the apply_source_filter config available on all macros. This would allow idempotent builds over event base history tables from source where we can simply restart jobs, or full-refresh the data vault should issues arise. I would expect this to be available on all macros excluding possibly hubs and links.
Describe alternatives you've considered
The config is already there for the main sat macro, but I don't see any harm adding it to more as an option so there is the flexibility to limit the data considered by the macro rather than having to create a delta layer.
AB#5352
The text was updated successfully, but these errors were encountered: