Support freshness policies and freshness policy sensors on dynamically parititoned assets #18106
danielknode
started this conversation in
Ideas
Replies: 1 comment
-
Yeah is there a timeline for building support for dynamically partitioned freshness polices? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We are building a multi-tenant system where each asset is partitioned by tenant. We use dynamic partitions to achieve this where each tenant has its own partition (tenants ID). We want to enforce freshness policies on a per-tenant basis, or in other words, each partition should be able to have a freshness policy applied. Then when a lazy AMP is applied to the asset, it will auto materialize tenant partitions as they become out of date. And of course we want observability into when tenants data is out of date, so we would like the freshness policy sensor to support dynamic partitions as well, it should indicate which partition (tenant) was late and by how much.
Beta Was this translation helpful? Give feedback.
All reactions