Create "run history" page for batched updates #4461
Labels
💻 aspect: code
Concerns the software code in the repository
🌟 goal: addition
Addition of new feature
🟩 priority: low
Low priority and doesn't need to be rushed
🧱 stack: catalog
Related to the catalog and Airflow DAGs
🔧 tech: airflow
Involves Apache Airflow
Milestone
Description
As we centralize on the batched update DAG as a standard update mechanism for the project, it may be beneficial to have a log of the batched update runs that have been performed. This can serve as both a changelog for the data as well as a rubric for how to perform updates on more complex fields (like the tags).
It might be hard for this to live as a page on our docs site, because batched update runs can (and should) happen outside the context of PRs. It may, however, be possible to create a new Airflow page which would generate content dynamically using a custom plugin. This page could query all the batched update runs and surface the following information, in reverse chronological order (potentially with hard links for each run so they could be referenced externally):
query_id
table_name
select_query
update_query
batch_size
The text was updated successfully, but these errors were encountered: