-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize column altering checks #29
Comments
Theres another optimization approach that attempts to skip the alter queries completely. I think right now we're running alter queries without checking if theyre needed. When a table exists we could instead request the table metadata then for each column we diff the existing column metadata to what we'd create if it didnt exist, effectively skipping all queries for columns that dont need changes. |
I noticed that target-postgres avoids this issue by not allowing column altering. This could be a short term fix but ideally we'd make changes to optimize the process and still allow altering. |
Comment in #57 (comment) about how performance issues were improved but not fixed. |
Currently theres a lot of wasted time during startup where the target is running lots of queries to check the existing schema against the schema message it received in case it needs to make alterations. One reason this is slow is because it iterates every column and either:
In doing this its requesting ddl to be generated then executes it in a serial loop.
An optimization to speed this up would be to bring the execution up to the prepare_table method after the for loop. The
prepare_column
method would instead return the ddl instead of executing it, so at the prepare_table level it can send one large script.@edgarrmondragon what do you think about this?
The text was updated successfully, but these errors were encountered: