Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
In order to alleviate the serialisation caused by the single-threaded REST API frontend server we switch the respective database functions to asyncio. This allows the server to at least accept more requests while database operations are in progress. The works very well with MariaDB. Postgres shows a slow response on initial operations up to about 15 seconds caused by an autovacuum operation triggered by an introspection feature of asyncpg (MagicStack/asyncpg#530). The workaround is to disable the JIT of newer postgresql versions server-side for the time being. sqlite flat-out runs into "database is locked" errors. More research is required here. We rewrite the database URL to specific asyncio dialects for now. The plan is to switch to asyncio completely so we can leave them alone in the end. The SQLAlchemy dependency is raised to 1.4.24 because this is the first version to support asyncmy, the new asyncio mysql driver. Move sleeping for retries out of the critical section protected by locking to allow for even more parallelism. Remove the lock on analysis journal retrieval since reads should never conflict with each other. We need to keep the locking in analysis_add() even with async because multiple async calls of the routine may be in progress at various stages of processing and conflict with and possibly deadlock each other, particularly when using sqlite which will throw 'database is locked' errors after a timeout. Having a threading and asyncio Lock protect adding and updating of analyses from threading and asyncio context is likely to not work as required. The only hope is to switch analysis update to asyncio as well. Start to move to the 2.0 SQLAlchemy API using session.execute() and select(), delete() and update() statements. For the asyncio API this is requied since session.query() is not supported there. We switch some non-asyncio users as well while we're at it. This also allows for reusing of statements across retries. Start using session context handlers to get rid of explicit session closing. The testsuite is updated to match.
- Loading branch information