Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimise the load dataset for snowflake when table and schema exist #492

Closed
9 tasks
sunank200 opened this issue Jun 28, 2022 · 0 comments
Closed
9 tasks

Optimise the load dataset for snowflake when table and schema exist #492

sunank200 opened this issue Jun 28, 2022 · 0 comments
Assignees
@tatiana @sunank200
Labels
feature New feature or request

Comments

@sunank200
Copy link
Contributor

Is your feature request related to a problem? Please describe.
For the optimisation of load data, in case the schema and table exist, use the COPY INTO command by creating the stage.

Describe the solution you'd like
Check if the table and schema exist. Create a snowflake stage based on the snowflake integration provided by the user. Generate the query to COPY INTO and run.

Acceptance Criteria

  • All checks and tests in the CI should pass
  • Unit tests (90% code coverage or more, once available)
  • Integration tests (if the feature relates to a new database or external service)
  • Example DAG
  • Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • Exception handling in case of errors
  • Logging (are we exposing useful information to the user? e.g. source and destination)
  • Improve the documentation (README, Sphinx, and any other relevant)
  • How to use Guide for the feature (example)
@sunank200 sunank200 added the feature New feature or request label Jun 28, 2022
@sunank200 sunank200 mentioned this issue Jun 28, 2022
Closed
4 tasks
@sunank200 sunank200 self-assigned this Jun 28, 2022
@sunank200 sunank200 mentioned this issue Jun 28, 2022
Closed
2 tasks
tatiana added a commit that referenced this issue Jul 11, 2022
@tatiana
Add Snowflake stage methods
e638c6c 
- Create
- Check existance
- Drop

Relates to: #492
Co-authored-by: Ankit Chaurasia <[email protected]>
@tatiana tatiana mentioned this issue Jul 11, 2022
Merged
tatiana added a commit that referenced this issue Jul 11, 2022
@tatiana
Add Snowflake stage methods
b0bc1de 
- Create
- Check existance
- Drop

Relates to: #492
Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana added a commit that referenced this issue Jul 12, 2022
@tatiana
Add Snowflake stage methods (#523)
727fc4b 
- Create
- Check existance
- Drop

Relates to: #492
Co-authored-by: Ankit Chaurasia <[email protected]>
utkarsharma2 pushed a commit that referenced this issue Jul 12, 2022
@tatiana @utkarsharma2
Add Snowflake stage methods (#523)
695eff2 
- Create
- Check existance
- Drop

Relates to: #492
Co-authored-by: Ankit Chaurasia <[email protected]>
@tatiana tatiana self-assigned this Jul 14, 2022
@tatiana tatiana closed this as completed Jul 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tatiana tatiana

@sunank200 sunank200

Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Optimize loading datasets for snowflake [WIP] astronomer/astro-sdk
2 participants
@tatiana @sunank200