-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support loading datasets of 10GB in Snowflake in less than 5 min #430
Comments
tatiana
changed the title
Support loading datasets of 10GB in Snowflake in less than 2 min
Support loading datasets of 10GB in Snowflake in less than 5 min
Jun 7, 2022
2 tasks
tatiana
added a commit
that referenced
this issue
Jul 14, 2022
Refactor how tables are created in BaseDatabase.load_file_to_table We should prioritise creating the table using the `table.columns` if they are specified by the user and have the dataframe autodetection as a fallback. Most of the complexity of #487 was the creation of tables, and this step aims to simplify the Snowflake `load_file` optimization. Relates to: #430, #481, #493, #494
tatiana
added a commit
that referenced
this issue
Jul 15, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana
added a commit
that referenced
this issue
Jul 18, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana
added a commit
that referenced
this issue
Jul 19, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana
added a commit
that referenced
this issue
Jul 19, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
kaxil
pushed a commit
that referenced
this issue
Jul 20, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana
added a commit
that referenced
this issue
Jul 20, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
I believe this was closed by mistake, we haven't merged the changes into master yet..! |
sunank200
pushed a commit
that referenced
this issue
Jul 22, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
utkarsharma2
pushed a commit
that referenced
this issue
Jul 25, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana
added a commit
that referenced
this issue
Jul 25, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana
added a commit
that referenced
this issue
Jul 25, 2022
Fix: #430 Co-authored-by: Ankit Chaurasia <[email protected]>
tatiana
added a commit
that referenced
this issue
Jul 26, 2022
Fix: #430 Reduce the time to load to Snowflake by 20% for 5GB datasets (from 24.46 min to 5.49 min). Further details are in the PR results file. Co-authored-by: Ankit Chaurasia <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Dependencies
Acceptance criteria
The text was updated successfully, but these errors were encountered: