Skip to content

Commit

Permalink
db: Add initial asyncio support
Browse files Browse the repository at this point in the history
In order to alleviate the serialisation caused by the single-threaded
REST API frontend server we switch the respective database functions to
asyncio. This allows the server to at least accept more requests while
database operations are in progress.

The works very well with MariaDB.

Postgres shows a slow response on initial operations up to about 15
seconds caused by an autovacuum operation triggered by an introspection
feature of asyncpg (MagicStack/asyncpg#530).
The workaround is to disable the JIT of newer postgresql versions
server-side for the time being.

sqlite flat-out runs into "database is locked" errors. More research is
required here.

We rewrite the database URL to specific asyncio dialects for now. The
plan is to switch to asyncio completely so we can leave them alone in
the end.

The SQLAlchemy dependency is raised to 1.4.24 because this is the first
version to support asyncmy, the new asyncio mysql driver.

Move sleeping for retries out of the critical section protected by
locking to allow for even more parallelism. Remove the lock on analysis
journal retrieval since reads should never conflict with each other. We
need to keep the locking in analysis_add() even with async because
multiple async calls of the routine may be in progress at various stages
of processing and conflict with and possibly deadlock each other,
particularly when using sqlite which will throw 'database is locked'
errors after a timeout. Having a threading and asyncio Lock protect
adding and updating of analyses from threading and asyncio context is
likely to not work as required. The only hope is to switch analysis
update to asyncio as well.

Start to move to the 2.0 SQLAlchemy API using session.execute() and
select(), delete() and update() statements. For the asyncio API this is
requied since session.query() is not supported there. We switch some
non-asyncio users as well while we're at it. This also allows for
reusing of statements across retries.

Start using session context handlers to get rid of explicit session
closing.

The testsuite is updated to match.
  • Loading branch information
michaelweiser committed Jan 5, 2022
1 parent f3c0a74 commit ba7a3d4
Show file tree
Hide file tree
Showing 4 changed files with 295 additions and 228 deletions.
Loading

0 comments on commit ba7a3d4

Please sign in to comment.