Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SQLite URI mode and arguments #196

Open
brendanlong opened this issue Apr 26, 2020 · 1 comment
Open

Support SQLite URI mode and arguments #196

brendanlong opened this issue Apr 26, 2020 · 1 comment

Comments

@brendanlong
Copy link

brendanlong commented Apr 26, 2020

Background

In sqlite3, if you set uri=True, you can pass query param arguments to the underlying database, including options like cache=shared and mode=memory, which are useful for unit testing (you can create an in-memory DB which stays alive between multiple connections).

For example, without the query params, :memory: databases are completely separate:

import sqlite3

db1 = sqlite3.connect(":memory:")
db1.execute("CREATE TABLE Example(id INT)")
db1.execute("SELECT * FROM Example")

db2 = sqlite3.connect(":memory:")
db2.execute("SELECT * FROM Example") # sqlite3.OperationalError: no such table: Example

But if we pass cache=shared, they're shared:

import sqlite3

db1 = sqlite3.connect("file::memory:?cache=shared", uri=True)
db1.execute("CREATE TABLE Example(id INT)")
db1.execute("SELECT * FROM Example")

db2 = sqlite3.connect("file::memory:?cache=shared", uri=True)
db2.execute("SELECT * FROM Example") # no error

mode=memory lets you create named memory DB's so you can share them by name instead of having a single shared namespace.

The problem

The problem is that databases handles query params itself and doesn't pass them to the underlying connection:

import asyncio
import sqlite3

from databases import Database

async def example():
    _ = sqlite3.connect("file::memory:?cache=shared", uri=True) # Hold a reference open since SQLite deletes the shared DB when ths last reference closes
    db1 = Database("sqlite:///file::memory:?cache=shared", uri=True)
    await db1.execute("CREATE TABLE Example(id INT)")
    await db1.execute("SELECT * FROM Example")
    db2 = Database("sqlite:///file::memory:?cache=shared", uri=True)
    await db2.execute("SELECT * FROM Example") # sqlite3.OperationalError: no such table: Example

asyncio.run(example())

Proposed solutions

I messed around with this locally and come up with this solution:

diff --git a/databases/backends/sqlite.py b/databases/backends/sqlite.py
index 6a733ac..7f46c76 100644
--- a/databases/backends/sqlite.py
+++ b/databases/backends/sqlite.py
@@ -1,5 +1,6 @@
 import logging
 import typing
+from urllib.parse import urlencode
 import uuid
 
 import aiosqlite
@@ -51,12 +52,14 @@ class SQLiteBackend(DatabaseBackend):
 
 class SQLitePool:
     def __init__(self, url: DatabaseURL, **options: typing.Any) -> None:
-        self._url = url
+        self._database = url.database
+        if url.options:
+            self._database += "?" + urlencode(url.options)
         self._options = options
 
     async def acquire(self) -> aiosqlite.Connection:
         connection = aiosqlite.connect(
-            database=self._url.database, isolation_level=None, **self._options
+            database=self._database, isolation_level=None, **self._options
         )
         await connection.__aenter__()
         return connection

... which basically just passes options through, but it doesn't seem to match how other backends work (with kwargs and URL query params being treated identically).

If you wanted to make options and kwargs interchangeable, you could do something like:

function_kwarg_names = {"timeout", "detect_types", "isolation_level", "factory", "check_same_thread", "cached_statements", "uri"}
self._function_options = {key: value for (key, value) in options.items() if key in function_kwargs}
self._uri_options = {key: value for (key, value) in options.items() if key not in function_kwargs}
....
self._database = url.database + "?" + urlencode(self._uri_options)

I can make a PR if that would be helpful but I'm not really sure how to approach this.

This would provide a workaround to fix #75

@yeralin
Copy link

yeralin commented Jul 17, 2020

Any update on this?

nvjoshi2 pushed a commit to irulast/databases that referenced this issue Jan 23, 2022
nvjoshi2 pushed a commit to irulast/databases that referenced this issue Jan 24, 2022
zanieb pushed a commit that referenced this issue Jul 12, 2023
zanieb pushed a commit that referenced this issue Jul 12, 2023
zanieb pushed a commit that referenced this issue Jul 26, 2023
zanieb pushed a commit that referenced this issue Jul 26, 2023
zanieb added a commit that referenced this issue Aug 2, 2023
* add support for sqlite connection string query parameters, cached memory databases

* add additional comments #196

* tweaked comments #196

* Lint

---------

Co-authored-by: Nathan Joshi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants