sqlite3.DatabaseError: database disk image is malformed

[simon-liebehenschel]

My question is not about the error but about the package design.

Should a user verify a database file before any usage of aiohttp-client-cache or we can add a broken database handling to the aiohttp-client-cache itself?

Maybe add a flag recreate_database_if_broken: bool? Or this is my problem to verify the file and keep it not broken?

Here is a full traceback if you want to look at it:

/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp/client.py:1167: in __aenter__
    self._resp = await self._coro
/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp_client_cache/session.py:46: in _request
    response, actions = await self.cache.request(
/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp_client_cache/backends/base.py:134: in request
    response = None if actions.skip_read else await self.get_response(actions.key)
/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp_client_cache/backends/base.py:142: in get_response
    response = await self.responses.read(key) or await self._get_redirect_response(str(key))
/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp_client_cache/backends/sqlite.py:193: in read
    return self.deserialize(await super().read(key))
/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp_client_cache/backends/sqlite.py:164: in read
    async with self.get_connection() as db:
/usr/local/lib/python3.11/contextlib.py:204: in __aenter__
    return await anext(self.gen)
/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp_client_cache/backends/sqlite.py:92: in get_connection
    await self._init_db()
/opt/pysetup/.venv/lib/python3.11/site-packages/aiohttp_client_cache/backends/sqlite.py:101: in _init_db
    await self._connection.execute(
/opt/pysetup/.venv/lib/python3.11/site-packages/aiosqlite/core.py:190: in execute
    cursor = await self._execute(self._conn.execute, sql, parameters)
/opt/pysetup/.venv/lib/python3.11/site-packages/aiosqlite/core.py:133: in _execute
    return await future
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <Connection(Thread-8, stopped 140703640315584)>
    def run(self) -> None:
        """
        Execute function calls on a separate thread.
    
        :meta private:
        """
        while True:
            # Continues running until all queue items are processed,
            # even after connection is closed (so we can finalize all
            # futures)
            try:
                future, function = self._tx.get(timeout=0.1)
            except Empty:
                if self._running:
                    continue
                break
            try:
                LOG.debug("executing %s", function)
>               result = function()
E               sqlite3.DatabaseError: database disk image is malformed
/opt/pysetup/.venv/lib/python3.11/site-packages/aiosqlite/core.py:106: DatabaseError

Package version: 0.9.1

[JWCook]

That's a good question. This isn't a case I've run into myself. Do you happen to know what can cause a malformed database? System crash during a write? Underlying filesystem/storage hardware issue? Cosmic rays?

If this is something you expect might happen again, then I think it would be reasonable to add handling for this. A couple threads on StackOverflow indicate that a malformed db can sometimes be recovered using the sqlite3 CLI, so it might be worthwhile to keep it around and let the user deal with it (either delete or try to recover) instead of automatically deleting it.

What would you think of something like this?

• Rename the old db
• Log a warning (Traceback + "Malformed database moved to {original_name}.bak")
• Create a new db in the original location

[simon-liebehenschel]

Thanks. After 2 weeks since I created this discussion, I think that it is a user problem to handle a database state. aiohttp-client-cache maintainer(s) must not care about that.

In my case most likely something happened with Kubernetes GitLab runner cache. This happens too rarely so I did not spend my time on downloading and analyzing a database file.

[JWCook]

If this does come up more frequently in the future, let me know and we can revisit this. I would also be open to a PR for this.

I generally try to respond to issues and PRs within a week or so, but I do sometimes get quite busy with other things (job, family, medical, other open source projects, etc.). I'm currently the sole maintainer, but I am open to adding more. You have helped out quite a bit, so I would be willing to give you collaborator permissions on this repo if you're interested.

[simon-liebehenschel]

I'm currently the sole maintainer, but I am open to adding more. You have helped out quite a bit, so I would be willing to give you collaborator permissions on this repo if you're interested.

Sorry, I have not found a way to reply outside of an unrelated thread (e.g. via email), so I will reply here (actually, my profile does not mention a way to contact me either 😄). I am interested in some help with maintenance and keeping an eye on open issues. If I can be more useful with "collaborator" permissions, I would appreciate it. If this topic is still relevant, you can add me.
Either way, I will try to contribute when I have time.

[JWCook]

Great! You now have write permissions to the repo, and permissions to manage issues and PRs. Any help with triage for incoming issues would be appreciated. You can even publish a release if you need to, for example if there's a bugfix you need ASAP and I'm not available for some reason. There are some more details about maintenance in the contributing guide.