[7.6.1] RuntimeError: TorrentState cannot be stored in the database

[sentry-for-tribler]

Sentry Issue: TRIBLER-5M

RuntimeError: ('Object TorrentState[new:22] cannot be stored in the database. DatabaseError: database disk image is malformed
--LONG TEXT--
Traceback (most recent call last):
  File "tribler_core/modules/popularity/popularity_community.py", line 121, in process_torrents_health
  File "pony/orm/core.py", line 476, in __exit__
  File "pony/orm/core.py", line 490, in _commit_or_rollback
  File "pony/orm/core.py", line 381, in commit
  File "pony/orm/core.py", line 370, in rollback_and_reraise
  File...

[drew2a]

This type of error can be reproduced by replacing normal metadata.db with corrupted metadata.db.
Corrupted metadata.db can be made just by adding some noise text (or binary) to it.

[drew2a]

The root cause of the corrupted file still remains unknown.

One possible workaround in case of this error:

1. Create a backup for metadata.db by renaming it to metadata_backup.db
2. Create a new metadata.db

[drew2a]

@kozlovsky what do you think, what is the best approach here?

[kozlovsky]

The root cause is indeed unclear. According to some reports, corruption on OS X can be caused by using memory-mapped I/O in SQLite, and in that case, we can recompile SQLite to turn off using memory-mapped I/O.
couchbase/couchbase-lite-ios#1482 (comment)
http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-fullfsync-on-macOS-td95366.html#a95401
But the problem is not limited to the OS X system only, sot it requires further investigation.

One possible workaround in case of this error:

1. Create a backup for metadata.db by renaming it to metadata_backup.db
2. Create a new metadata.db

Looks like a correct approach. In principle, it is possible to save at least partial data from the corrupted database by performing backup and then restore. But this way we cannot be sure that the resulted database contains all data. So it is probably not worth it, and we just need to create afresh database so the user can start Tribler without experiencing the same error again.

It may be not easy to rename the database file correctly, as another thread may hold a second connection to the database. Because of this, renaming should be performed at the next startup. As a flag that the renaming is necessary, we can create a file, like metadata.db.corrupted-detected, and check the presence of it on startup.

Beside metadata.db file itself it is also necessary to rename other metadata.db-* files (when present), like metadata.db-journal, metadata.db-wal and metadata.db-shm.

[drew2a]

@kozlovsky yes... Why I offered "rename" — because it should work even there is no free disk space. But this is just a rough idea.

[ichorid]

Indeed, we don't have to try to save the Channels DB data, because all the data is duplicated in the /channels folder in the form of serialized channel torrents. This includes the user's personal channel. Also, the entries for the user's personal channel torrents are serialized to disk in the same folder, named accordingly.

The solution

In case of DB crash, we can show the user a dialog window offering them to "drop the database and restore personal channel from disk". After that, we create the DB anew and read the user's channel from disk.

[ichorid]

Actually, duplicate of #5252

[sentry-for-tribler]

Sentry issue: TRIBLER-17

A corrupted database seems like the most frequently reported issue now.

[ichorid]

the Sentry issue is not there. Closing this one