Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Airflow Connections for Elasticsearch clusters #2371

Closed
AetherUnbound opened this issue Jun 9, 2023 · 2 comments
Closed

Add Airflow Connections for Elasticsearch clusters #2371

AetherUnbound opened this issue Jun 9, 2023 · 2 comments
Labels
🕹 aspect: interface Concerns end-users' experience with the software 🌟 goal: addition Addition of new feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔒 staff only Restricted to staff members

Comments

@AetherUnbound
Copy link
Collaborator

Description

Add the connection information for the staging & production Elasticsearch connections in the Airflow UI, as described by the implementation plan: https://docs.openverse.org/projects/proposals/search_relevancy_sandbox/20230518-implementation_plan_staging_index_rapid_iteration.html#airflow-connections

Additional context

This is blocked by #2370 and is a part of #392

@AetherUnbound AetherUnbound added ⛔ status: blocked Blocked & therefore, not ready for work 🌟 goal: addition Addition of new feature 🔒 staff only Restricted to staff members 🕹 aspect: interface Concerns end-users' experience with the software 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Jun 9, 2023
@github-project-automation github-project-automation bot moved this to 📋 Backlog in Openverse Backlog Jun 9, 2023
This was referenced Jun 9, 2023
@AetherUnbound AetherUnbound self-assigned this Jun 30, 2023
@AetherUnbound AetherUnbound moved this from 📋 Backlog to 📅 To do in Openverse Backlog Jun 30, 2023
@AetherUnbound AetherUnbound moved this from 📅 To do to 📋 Backlog in Openverse Backlog Jul 18, 2023
@AetherUnbound AetherUnbound removed their assignment Oct 4, 2023
@AetherUnbound AetherUnbound moved this from 📋 Backlog to ⛔ Blocked in Openverse Backlog Oct 4, 2023
@stacimc
Copy link
Collaborator

stacimc commented Nov 21, 2023

@AetherUnbound I've forgotten why this is needed separate from the existing connections, and how it's meant to be used. The IP describes and links to the ElasticsearchPythonHook, which doesn't seem to accept a conn_id. I can get it working locally thogh with something like:

# Using the existing value from AIRFLOW_CONN_ELASTICSEARCH_HTTP_STAGING
elastic_hook = ElasticsearchPythonHook(hosts="http://es:9200")

conn = elastic_hook.get_conn()
old_index = conn.indices.get(index="audio-init")

# Omitted for space, but I was also able to create a new index 

Do we mean to use the ElasticsearchHook instead?

@AetherUnbound
Copy link
Collaborator Author

Yea I was just thinking about this one 😅 I made it based on the assumption of how other providers work, it seems it's unnecessary for Elasticsearch! We can close this.

@github-project-automation github-project-automation bot moved this from ⛔ Blocked to ✅ Done in Openverse Backlog Nov 21, 2023
@obulat obulat removed the ⛔ status: blocked Blocked & therefore, not ready for work label Nov 28, 2023
@openverse-bot openverse-bot moved this from ✅ Done to 📋 Backlog in Openverse Backlog Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🕹 aspect: interface Concerns end-users' experience with the software 🌟 goal: addition Addition of new feature 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: catalog Related to the catalog and Airflow DAGs 🔒 staff only Restricted to staff members
Projects
Archived in project
Development

No branches or pull requests

3 participants