Calibre-web freezing or locking up entirely when attempting to fetch metadata

[jdude104]

Describe the bug/problem
When editing metadata of a book, clicking Fetch Metadata does not return results, and calibre-web becomes entirely unresponsive until container is restarted.

To Reproduce
Steps to reproduce the behavior:

1. Go to a book in library
2. Click on "Edit Metadata"
3. Click on "Fetch Metadata"

Logfile
This log is one example seen after restarting

[2024-01-01 18:16:11,628]  INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-01 18:16:11,631]  INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-01 18:16:14,008]  INFO {cps.server:207} Starting Gevent server on [::]:8083
[2024-01-01 18:16:19,560] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:19,566] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:19,571] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:19,576] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:23,792] DEBUG {cps.updater:329} Stable version: {'version': '0.6.21'}
[2024-01-01 18:16:36,444] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:36,448] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:36,453] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:36,457] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 18:16:46,906] DEBUG {cps.metadata_provider.douban:77} start searching Accel World, Vol. 04: Flight Toward a Blue Sky on douban
[2024-01-01 18:17:08,259]  INFO {cps.server:284} webserver stop (restart=False)
[2024-01-01 18:17:08,261]  INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2024-01-01 18:17:36,686]  WARN {cps.config_sql:352} Log path  not valid, falling back to default
[2024-01-01 18:17:36,743]  INFO {cps:168} *** "pytz" version does not meet the requirements. Should: available, Found: Not available, please consider installing required version ***
[2024-01-01 18:17:36,744]  INFO {cps:177} Starting Calibre Web...
[2024-01-01 18:17:37,259]  WARN {py.warnings:109} /lsiopy/lib/python3.10/site-packages/flask_limiter/extension.py:308: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(
[2024-01-01 18:17:37,576]  INFO {apscheduler.scheduler:181} Scheduler started
[2024-01-01 18:17:37,579]  INFO {apscheduler.scheduler:895} Added job "end scheduled task" to job store "default"
[2024-01-01 18:17:40,105]  INFO {cps.server:207} Starting Gevent server on [::]:8083
[2024-01-01 18:17:49,942] DEBUG {cps.updater:329} Stable version: {'version': '0.6.21'}

sometimes they instead look like this:

[2024-01-01 17:36:56,247]  INFO {cps.web:1207} Serving book: Delicious in Dungeon v01 (2017) (Digital) - Unknown
[2024-01-01 17:37:37,440]  INFO {cps.helper:962} Downloading file: /books/Unknown/Delicious in Dungeon v01 (2017) (Digital) (LuCaZ) (810)/Delicious in Dungeon v01 (2017) (Digital) - Unknown.epub
[2024-01-01 17:40:49,221]  WARN {cps.config_sql:352} Log path  not valid, falling back to default
[2024-01-01 17:42:51,420] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 17:42:51,426] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 17:42:51,431] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 17:42:51,437] ERROR {cps.db:867} Author Tadano, Nobuaki not found to display name in right order
[2024-01-01 17:42:56,692] ERROR {cps.db:867} Author Niels Ferguson, Bruce Schneier not found to display name in right order
[2024-01-01 17:42:59,660] ERROR {cps.db:867} Author Chestein, Alex Zandra Van not found to display name in right order
[2024-01-01 17:42:59,725] ERROR {cps.db:867} Author  not found to display name in right order
[2024-01-01 17:42:59,730] ERROR {cps.db:867} Author  not found to display name in right order
[2024-01-01 17:45:20,221] ERROR {cps.metadata_provider.amazon:129} 503 Server Error: Service Unavailable for url: https://www.amazon.com/s?k=Delicious+in+Dungeon+&i=digital-text&sprefix=Delicious+in+Dungeon+%2Cdigital-text&ref=nb_sb_noss
[2024-01-01 17:48:27,426]  INFO {cps.server:284} webserver stop (restart=False)
[2024-01-01 17:48:27,428]  INFO {apscheduler.scheduler:212} Scheduler has been shut down
[2024-01-01 17:48:57,173]  WARN {cps.config_sql:352} Log path  not valid, falling back to default
[2024-01-01 17:48:57,234]  INFO {cps:168} *** "pytz" version does not meet the requirements. Should: available, Found: Not available, please consider installing required version ***
[2024-01-01 17:48:57,235]  INFO {cps:177} Starting Calibre Web...
[2024-01-01 17:48:57,754]  WARN {py.warnings:109} /lsiopy/lib/python3.10/site-packages/flask_limiter/extension.py:308: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
  warnings.warn(

Expected behavior
Expected search to return some result or at least an error, and not cause container or calibre-web as a whole to lock up

Screenshots

Environment (please complete the following information):

• OS: Linux 6.1.64-Unraid x86_64
• Python version: unsure
• Calibre-Web version: 0.6.21
• Docker container: Linuxserver via Unraid
• Special Hardware: Dell Poweredge R510
• Browser: Firefox 121.0

Additional context
It did work just fine roughly a year ago, issue has only started occurring the past 4-6 months

[ashermiddleton]

Relevant issue: #2719

Seems to be due to amazon blocking the calibre-web server's IP from retrieving the cover. I'm also having this issue when fetching metadata for a few files quickly, it then sometimes results in an incorrect book name at the file-level because of being unable to fetch the metadata, which then has to be manually resolved.

[dwilliamhouston]

I mentioned this in the Discord site a week or so ago also. I thought it had to do with missing imagemagik or the optional dependencies in the docker container but was told that they were there. I haven't checked but I have also had this issues for a long while.

[clairiosa]

I'm having the same issue, fresh install as of today.

[2024-01-12 11:30:18,726]  INFO {cps.server:218} Starting Gevent server on [::]:8083
[2024-01-12 11:30:24,551] DEBUG {cps.uploader:261} Temporary file: /tmp/calibre_web/e78dad6e39941056cf299a761e52fda5
[2024-01-12 11:30:25,195] DEBUG {cps.helper:543} Moving title: /tmp/calibre_web/e78dad6e39941056cf299a761e52fda5 to /opt/calibre/Brandon Sanderson/Yumi and the Nightmare Painter (4)/Yumi and the Nightmare Painter - Brandon Sanderson
[2024-01-12 11:30:25,203] DEBUG {cps.services.worker:91} Add Task for user: admin - Upload Yumi and the Nightmare Painter
[2024-01-12 11:30:29,454]  INFO {scholarly:96} Getting https://scholar.google.com/scholar?hl=en&q=Yumi%20and%20the%20Nightmare%20Painter&as_vis=0&as_sdt=0,33
[2024-01-12 11:30:29,926] DEBUG {cps.metadata_provider.douban:77} start searching Yumi and the Nightmare Painter on douban
[2024-01-12 11:30:30,605]  INFO {httpx:1027} HTTP Request: GET https://scholar.google.com/scholar?hl=en&q=Yumi%20and%20the%20Nightmare%20Painter&as_vis=0&as_sdt=0,33 "HTTP/1.1 200 OK"

Here are the final logs before it crashes. I've tried multiple books and they all go the same way.

OS: Ubuntu 23.04
Python version:Python 3.11.4
Calibre-Web version: 1/6/24, 10:08 AM
Docker container: n/a
Special Hardware: proxmox lxc
Browser: Firefox 121.0.1

[snppls]

I am also having this issue. Fetching metadata crashes my whole docker image.

[wireframed]

I have the same issue, just updated to latest docker version (0.6.23). It would be nice, if the issue is Amazon blocking the request, to do two things:

• Handle server non-response gracefully with a 5-10 second timeout
• Allow configuring the sources to request data from globally. Only two of the sources seem relevant at all (Google Scholar and Comicvine) seem rather specialized so if you mainly have books, odds are those aren't going to give good matches anyway).

As a temporary fix, I've renamed the /cps/metadata_provider files I don't want/need to .bck, so that they won't be used. It works fine using just Google (if what you want is on their service). Remember, if you're using docker you have to restart the container, because metadata providers are loaded during start-up. This of course will be undone, if you update or re-pull the image. :)

[34292885]

I ran into the same problem as you and it stopped the whole VPS from working when getting the metadata

[DrFr33ze]

I can confirm, that renaming the file /app/calibre-web/cps/metadata_provider/amazon.py (and restarting the container) worked.
I would be happy if a fix could be included in the next release. Thanks!