[Feature] Support additional scrapers (ScreenScraper, TGDB, SteamGridDB)

[XargonWan]

Please add ScreenScraper as additional scraper as it seems to be more complete of IGDB on retro gaming.

Describe the solution you'd like
There could be a list in the options (similarly to Plex agents) to select which scarapers the user wants, and the config (for an ulimited/ish scraping ScreenScraper needs an user account).
Romm could scrape with the first scraper in the list and then fallback on the other ones if no match (or match but no media) is found.

Screenscraper is providing more than just cover, but wheel, descriptions, logo and more.

Describe alternatives you've considered
I don't know any other scraper services, but maybe someone does.

[zurdi15]

Yeah seems like a good idea to integrate more than just IGDB and let the user decide. I need to work a little bit more in the code base to allow more database integrations easily. When I reach a better quality code ScreenScraper will be integrated.

[victor10100111001]

I think this might be a good choice. Especially since it seems like Zurdi can request an api key that everyone running this project can use. From their website (translated):

Who can use the API?
The ScreenScraper API can only be integrated into fully free and distributed applications, or, if not, with the prior authorisation and conditions dictated by the ScreenScraper team. Failure to comply with this rule may be the subject of a breakdown of account, see any legal proceedings.
If you are a developer and want to integrate our API, contact us via the forum to present your software and obtain your usernames and passwords to provide to the API to validate your rights to use it 

This would remove some barrier of entry

[binarygeek119]

theres also https://thegamesdb.net/ and https://gamesdb.launchbox-app.com/ (althought in not sure is launchbox lets you use their db openly)

[gantoine]

Alright I've gone through the more prominent DBs/API out there and collected some notes. The ones I think are worth exploring are marked with a ⭐ . Please let me know if I missed any and I'll add them to the list.

IGBD: ~250K games, ~190 platforms ⭐

Pros

• Free (owned and run by Twitch)
• Extensive list of API endpoints that can get pretty specific
• Great rate limits (4 requests/second limit with no global max)
• Protobuf support

Cons

• Greater focus on modern games
• Low number of art assets

TheGamesDB: ~76K games, ~150 platforms ⭐

It seems the preferred way to interact with it is to mass-download the database and use the API async to keep your local copy up-to-date.

Pros

• Focused on retro games
• JSON dump can be downloaded from CDN

Cons

• Local copy of DB must be synced
• Sparse metadata and related assets

MobyGames: ~330K games, ~320 platforms ⭐

Pros

• Free for personal use
• Sizeable collection of games and platforms
• Decent collection of crowdsourced assets

Cons

• Low API request limits (1 requests/second, max 360 requests/hour)
• API only supports game and platform queries (incl. art assets)

As per the API TOS: "MobyGames must be credited wherever data from the API is used. Typically, a link to the MobyGames page for the game will suffice. "

RAWG: ~350K games, ~50 platforms

Pros

• Free for personal use
• Rich metadata on par with IGDB
• Support for DLCs, achievements, trailers, and more

Cons

• 20K API calls/month
• Some endpoints locked behind paid plans
• Poor collection of art assets

As per the TOS: "Free for personal use as long as you attribute RAWG as the source of the data and/or images and add an active hyperlink from every page where the data of RAWG is used."

GiantBomb: ~80K games, ~190 platforms

Pros

• Decent rate limits (450 requests/15 minutes)
• Extensive API endpoints
• Wiki-style entries for most popular games

Cons

• Paid only (~5$/month)
• Mediocre collection of art assets

ScreenScraper.fr: ~300K games, ~105 platforms ⭐

Pros

• Extensive collection of art assets
• Free for personal use and open-source projects
• Decent API footprint

Cons

• Low API rates (20K requests/24 hours for registered users, 3K otherwise)

As per the TOS: "L'API ScreenScraper ne peut être intégré que dans les applications entièrement gratuites et distribuées, ou, dans le cas contraire, avec l'autorisation préalable et les conditions dictées par l'équipe de ScreenScraper."

SteamGridDB: ? games, ? platforms

Seems to be mostly focused on art assets based around their concept of "grids", collections of assets uploaded by users.

Pros

• Prebuilt python library
• Asset collection split by logos, hero images and icons
• ~290K Grids, ~75K heroes, ~75K logos, ~35K icons

Cons

• Unclear API limits
• No game metadata provided
• More focused on modern games

GameTDB: ? games, 7 platforms

Only the following platforms are currently supported: Wii, WiiU, DS, 3DS, Switch, PS3, Gamecube

Pros

• Solid collection of box art assets (for supported platforms)

Cons

• Limited platforms support
• No API/undocumented API

[Casuallynoted]

Possibly the ability to import Launchbox's metadata xml files? Not sure how practical that is though

https://forums.launchbox-app.com/topic/52709-launchbox-db-api/

They definitely seem to have their stuff together when it comes to resources, I can say that. The submission process with IGDB has been a time haha

[parasiteoflife]

Yes, please, implement other scrapers, giving my phone number to twitch is something I wont ever do and unfortunately for us is the only way to enable 2FA on their site.

[Emporea]

@zurdi15 Ok thanks for pointing out this existing issue. I searched for IGDB / alternative, but this issue didn't pop up.
Can you estimate how long it will take to implement other sources? What needs to be achieved first in your opinion so this could happen?

Thanks for all your work on this project.

[maxexcloo]

Screenscraper is also nice as most frontends (such as ES-DE) use it also so it would be easier to potentially match/share data.

[LeeTeng2001]

bump

[wgoff194]

I too will bump this, ScreenScraper is absolute gold!

[adamantike]

Please only "bump" issues by reacting to the issue description with 👍, which is useful for developers to sort by most requested features. Any other kind of "+1" comments only generate noise for people subscribed to these issues, looking for actual progress.