-
Notifications
You must be signed in to change notification settings - Fork 403
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slow SELECT query on large table #741
Comments
The |
@elprans thanks. your solution does not solve the issue we had with |
Well, my best guess would be that Postgres is grossly mis-planning the query somehow if a prepared statement is used. Try |
This does not use a prepared statement. Tried |
TL;DR our query is fine in PSQL client, but somehow slow when executed via |
Did you try to reproduce that issue with the newer versions of Postgres (13.3) and Python (3.9.5)? |
I just had very similar symptoms, with a query that was fast in psql taking 6 seconds via asyncpg. After a lot of testing I narrowed it down to having ENUM columns in the query results. My query returning 16 columns took 15ms... adding an enum column made that into 1.5 seconds, and adding a second enum column made that into 6 seconds! The fix discussed at #530 (comment) seemed to work, and reduced the time back down to 20ms, so I guess this is related to the postgres JIT rather than something in asyncpg. Perhaps that might be the cause in your query too? |
Thanks @Tomcat-Engineering - the JIT issue sounds like a good rabbit hole to follow 😃 Unfortunately, I no longer have means to test the above using asyncpq .... since I switched to Sqlite as our huge database is generated once a month and it's read-only 100% of time so the little dB engine is more suited for now. |
the issue with a local PostgreSQL install?: No, and I was able to reproduce it locally.
uvloop?: We use Starlette for serving API requests.
We have a large (~30 million rows) database table. I have noticed the SQL would run terribly slow when executed via
asyncpg
but very fast viapsql
.Schema for table:
Query to execute.
This query never results in more than 100 rows (out of mentioned 30 million).
Time to execute via
psql
: less than 1 msSame query via
asyncpg
: more than 6 seconds (!)I have not looked at source code for
asyncpg
so not sure what's going on here.Can someone tell me why
asyncpg
runs this query so slow? Thanks!FYI I have fixed this "temporarily" by some quick data normalisation where I added
zipcode_x
that contains no space so I could got rid of the SQL'sREPLACE
function.and now
asyncpg
is very fast (as it should be).The text was updated successfully, but these errors were encountered: