Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create "run history" page for batched updates #4461

Open
AetherUnbound opened this issue Jun 6, 2024 · 0 comments
Open

Create "run history" page for batched updates #4461

AetherUnbound opened this issue Jun 6, 2024 · 0 comments
Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔧 tech: airflow Involves Apache Airflow

Comments

@AetherUnbound
Copy link
Collaborator

Description

As we centralize on the batched update DAG as a standard update mechanism for the project, it may be beneficial to have a log of the batched update runs that have been performed. This can serve as both a changelog for the data as well as a rubric for how to perform updates on more complex fields (like the tags).

It might be hard for this to live as a page on our docs site, because batched update runs can (and should) happen outside the context of PRs. It may, however, be possible to create a new Airflow page which would generate content dynamically using a custom plugin. This page could query all the batched update runs and surface the following information, in reverse chronological order (potentially with hard links for each run so they could be referenced externally):

  • query_id
  • table_name
  • select_query
  • update_query
  • batch_size
  • link to the DAG run
  • link to the GitHub issue
  • duration for select (if available)
  • duration for update (if available)
@AetherUnbound AetherUnbound added 🌟 goal: addition Addition of new feature 💻 aspect: code Concerns the software code in the repository 🔧 tech: airflow Involves Apache Airflow 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Jun 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔧 tech: airflow Involves Apache Airflow
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant