Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] apply_source_filter config added to ALL appropriate macros #222

Open
philmaddocks opened this issue Jan 8, 2024 · 0 comments
Open
Assignees
Labels
feature This is is requesting a new feature

Comments

@philmaddocks
Copy link

philmaddocks commented Jan 8, 2024

Is your feature request related to a problem? Please describe.
AutomateDV assumes that the data being pointed at the macros is always deltas. In our architecture, we have a data lake where we create an event base history for each source table. We'd like to point macros at this (and can for the most part) but apply_source_filter is not across all of the macros.

Describe the solution you'd like
I'd like to have the apply_source_filter config available on all macros. This would allow idempotent builds over event base history tables from source where we can simply restart jobs, or full-refresh the data vault should issues arise. I would expect this to be available on all macros excluding possibly hubs and links.

Describe alternatives you've considered
The config is already there for the main sat macro, but I don't see any harm adding it to more as an option so there is the flexibility to limit the data considered by the macro rather than having to create a delta layer.

AB#5352

@philmaddocks philmaddocks added the feature This is is requesting a new feature label Jan 8, 2024
@DVAlexHiggs DVAlexHiggs added feature This is is requesting a new feature and removed feature This is is requesting a new feature labels May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature This is is requesting a new feature
Projects
None yet
Development

No branches or pull requests

2 participants