Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Allow storing dbt ls cache into Object Store #1072

Closed
1 task
Tracked by #1103
tatiana opened this issue Jun 27, 2024 · 2 comments · Fixed by #1147
Closed
1 task
Tracked by #1103

[Feature] Allow storing dbt ls cache into Object Store #1072

tatiana opened this issue Jun 27, 2024 · 2 comments · Fixed by #1147
Assignees
@pankajkoti
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration dbt:list Primarily related to dbt list command or functionality enhancement New feature or request parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing triage-needed Items need to be reviewed / assigned to milestone
Milestone
Cosmos 1.6.0

Comments

@tatiana
Copy link
Collaborator

tatiana commented Jun 27, 2024 •
edited
Loading

Description

Following up on #1014 , we'd like to allow users to store the dbt ls cache into a remote object store, as an alternative to store it in Airflow Variables. This task relates to #927.

Use case/motivation

Even though the dbt ls is being compressed without significantly affecting the Airflow Metadata database, some users prefer not to have cache information mixed up with their Airflow variables.

Related issues

Are you willing to submit a PR?

  • Yes, I am willing to submit a PR!
@tatiana tatiana added enhancement New feature or request triage-needed Items need to be reviewed / assigned to milestone labels Jun 27, 2024
@tatiana tatiana changed the title [Feature] Allow storing dbt ls cache into Airflow Variable [Feature] Allow storing dbt ls cache into Object Store Jun 27, 2024
@tatiana tatiana added this to the Cosmos 1.6.0 milestone Jun 27, 2024
@dosubot dosubot bot added area:config Related to configuration, like YAML files, environment variables, or executer configuration dbt:list Primarily related to dbt list command or functionality parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing labels Jun 27, 2024
@tatiana tatiana mentioned this issue Jul 10, 2024
Merged
@tatiana tatiana mentioned this issue Jul 31, 2024
Closed
18 tasks
@pankajkoti
Copy link
Contributor

Aiming to create a PR for this tonight. We discussed the approach and clarified doubts in our call today

@pankajkoti pankajkoti mentioned this issue Aug 9, 2024
Merged
2 tasks
@pankajkoti
Copy link
Contributor

PR created and tested with Astro locally. Reviewed the approach with Pankaj Singh and also reached out to users for seeking feedback by releasing an alpha.

Aiming to add tests and docs today and hoping to merge it by tomorrow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pankajkoti pankajkoti

Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration dbt:list Primarily related to dbt list command or functionality enhancement New feature or request parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
Milestone
Cosmos 1.6.0
Development

Successfully merging a pull request may close this issue.

Add support to store and fetch dbt ls cache in remote stores astronomer/astronomer-cosmos
2 participants
@tatiana @pankajkoti