You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are not using Table's columns attribute when we create a table. Currently, we rely on load_file_to_table which internally uses pandas to auto-detect the schema while creating a table. There is no way where users can supply schema of their own.
Describe the solution you'd like
Table objects have field columns which is supposed to have column data if the user has passed in columns using this field we should create a table by using the database.create_table() method and instruct pandas/libs(Bigquery, Snowflake) to append to the table rather than creating a new one.
@utkarsharma2 as we discussed, the BaseDatabase has a create_table method which already uses the table.columns, which are in SQLAlchemy format :)
So I believe it is just a matter of using that. What I think is missing in the BQ implementation of load_file is to use/support this feature - potentially both on native and on non-native implementations. I'm handling this for Snowflake ATM, and I asked Daniel to take care of the Postgres one.
utkarsharma2
changed the title
While ceating tables allow user to pass custom table schema
Allow user to customize table schema while creating
Jul 13, 2022
Please describe the feature you'd like to see
We are not using Table's columns attribute when we create a table. Currently, we rely on
load_file_to_table
which internally uses pandas to auto-detect the schema while creating a table. There is no way where users can supply schema of their own.Describe the solution you'd like
Table objects have field
columns
which is supposed to have column data if the user has passed in columns using this field we should create a table by using thedatabase.create_table()
method and instruct pandas/libs(Bigquery, Snowflake) to append to the table rather than creating a new one.Acceptance Criteria
The text was updated successfully, but these errors were encountered: