You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For analytics use case we need to incrementally copy and export lineage_events to DWH. For this use case there is no good way to identify incrementally created events in database. The current event_time in lineage_events table is client generated and can be back dated.
Proposing to alter lineage_events table to add a new created_at timestamp column. Also index this new column.
More discussion to be had regarding backfill and default values of this column.
The text was updated successfully, but these errors were encountered:
For analytics use case we need to incrementally copy and export lineage_events to DWH. For this use case there is no good way to identify incrementally created events in database. The current event_time in lineage_events table is client generated and can be back dated.
Proposing to alter lineage_events table to add a new created_at timestamp column. Also index this new column.
More discussion to be had regarding backfill and default values of this column.
The text was updated successfully, but these errors were encountered: