Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set up staging ingestion server connection in Airflow #3492

Closed
Tracked by #392
stacimc opened this issue Dec 7, 2023 · 1 comment
Closed
Tracked by #392

Set up staging ingestion server connection in Airflow #3492

stacimc opened this issue Dec 7, 2023 · 1 comment
Assignees
@AetherUnbound
Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs
Milestone
Rapid iteration o...

Comments

@stacimc
Copy link
Collaborator

stacimc commented Dec 7, 2023

Problem

#3232 created a DAG for creating staging ES indices, which works by connecting to the staging ingestion server. In order to use the DAG we need to set up an Airflow connection to the staging ingestion server.

Description

First we should add an official DNS name to the staging data refresh server. Then we'll add the connection to Airflow using an Airflow variable AIRFLOW_CONN_STAGING_DATA_REFRESH.
@stacimc stacimc added 🟨 priority: medium Not blocking but should be addressed soon 🌟 goal: addition Addition of new feature 💻 aspect: code Concerns the software code in the repository 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Dec 7, 2023
@openverse-bot openverse-bot moved this to 📋 Backlog in Openverse Backlog Dec 7, 2023
@stacimc stacimc mentioned this issue Dec 7, 2023
Closed
16 tasks
@AetherUnbound AetherUnbound moved this from 📋 Backlog to 📅 To Do in Openverse Backlog Feb 12, 2024
@AetherUnbound AetherUnbound self-assigned this Feb 12, 2024
@AetherUnbound
Copy link
Collaborator

This connection has been added!
image

@openverse-bot openverse-bot moved this from 📅 To Do to ✅ Done in Openverse Backlog Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AetherUnbound AetherUnbound

Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs
Projects
Openverse Backlog
Archived in project
Milestone
Rapid iteration on Elasticsearch indices
Development

No branches or pull requests
2 participants
@AetherUnbound @stacimc