RSS search feed categories not handled for YggTorrent

[gotson]

I create a new issue to track this. Not sure if it should be in https://github.com/Prowlarr/Indexers instead, feel free to move it if needed.

When searching via RSS, the categories for the YGGTorrent indexer is not cascaded to the actual search.

YGGTorrent is in v1. However it seems to be working for some other v1 indexers like JPopSuki.

YggTorrent search

http://{server}/2/api?t=search&apikey=8c8b5c3b3af441a8a712806e44242524&cat=7000

[Info] NzbSearchService: Searching indexer(s): [YGGtorrent] for Term: [], Offset: 0, Limit: 0, Categories: [7000] 
[Info] Cardigann: Adding request: https://www3.yggtorrent.nz/engine/search?category=all&name=&description=&file=&uploader=&sub_category=&do=search&order=desc&sort=publish_date 
[Info] Cardigann: Adding request: https://www3.yggtorrent.nz/engine/search?category=all&name=&page=50&description=&file=&uploader=&sub_category=&do=search&order=desc&sort=publish_date

JPopSuki search 1

http://{server}/8/api?t=search&apikey=8c8b5c3b3af441a8a712806e44242524&cat=3000

[Info] NzbSearchService: Searching indexer(s): [JPopsuki] for Term: [], Offset: 0, Limit: 0, Categories: [3000] 
[Info] Cardigann: Adding request: https://jpopsuki.eu/ajax.php?filter_cat[1]=1&filter_cat[2]=1&searchtags=japanese&tags_type=0&searchstr=&order_by=s3&order_way=desc&disablegrouping=1&section=torrents

JPopSuki search 2

http://{server}/8/api?t=search&apikey=8c8b5c3b3af441a8a712806e44242524&cat=5000

[Info] NzbSearchService: Searching indexer(s): [JPopsuki] for Term: [], Offset: 0, Limit: 0, Categories: [5000] 
[Info] Cardigann: Adding request: https://jpopsuki.eu/ajax.php?filter_cat[5]=1&filter_cat[6]=1&filter_cat[7]=1&searchtags=japanese&tags_type=0&searchstr=&order_by=s3&order_way=desc&disablegrouping=1&section=torrents

---

sigh...

cat, not cats

also added: https://wiki.servarr.com/en/prowlarr/search#api-endpoints

I tried that before, and it works only for some indexers. For example on FileList it's working, but for YggTorrent it's not propagating to the actual indexer search:

[Info] NzbSearchService: Searching indexers: [YGGtorrent] for {Term: , Offset: 0, Limit: 0, Categories: [7000]} 
[Info] Cardigann: Adding request: https://www3.yggtorrent.nz/engine/search?category=all&name=&description=&file=&uploader=&sub_category=&do=search&order=desc&sort=publish_date 
[Info] Cardigann: Adding request: https://www3.yggtorrent.nz/engine/search?category=all&name=&page=50&description=&file=&uploader=&sub_category=&do=search&order=desc&sort=publish_date 

Originally posted by @gotson in Prowlarr/Prowlarr#182 (comment)

AB#1788

AB#1793

[bakerboy448]

Seems the category is currently set in the YML config not passed as a query parameter

Indexers/definitions/v1/yggtorrent.yml

Line 113 in 6abd82d

- name: category

Indexers/definitions/v1/yggtorrent.yml

Line 228 in 6abd82d

- path: "engine/search?category={{ .Config.category }}&name={{ .Keywords }}&description=&file=&uploader=&sub_category=&do=search&order={{ .Config.type }}&sort={{ .Config.sort }}"

[gotson]

I think the problem is here:

Indexers/definitions/v1/yggtorrent.yml

Line 228 in c7ce0a4

- path: "engine/search?category={{ .Config.category }}&name={{ .Keywords }}&description=&file=&uploader=&sub_category=&do=search&order={{ .Config.type }}&sort={{ .Config.sort }}"

If I understand correctly, the category is retrieved from the configuration of the indexer, instead of being retrieved dynamically. Is there any reason to do that ?

Is it possible to change the indexer definition to allow overriding the category from RSS parameters?

[gotson]

By creating a new instance of the indexer with the Books category set, i can narrow down the results. However, i cannot select a subcategory (like ebooks, or comics, or mags).

What would be a proper way to handle this ?

The way i see it, we could keep the existing hardcoded category configuration, and add a new subcategory configuration, which would be set in the search url for the sub_category parameter.

Or we could allow dynamic configuration via parameters, which should enable RSS search via cat=7010.

I also realize that this is not only impacting RSS, but all type of search. When searching from the Search screen in the Prowlarr UI, the same behaviour happens.

[bakerboy448]

@garfield69 sorry for the ping, but can you give some background as to why ygg has hardcoded (via config) categories?

[garfield69]

can you give some background as to why ygg has hardcoded (via config) categories?

Goes back to Nov 2017 with Jackett/Jackett#2066
Admittedly since the time that this was implemented, no one has done a review to see if this is still useful or needed, as presumably the web site may have upgraded their search engine since then.

[bakerboy448]

[meta - xfered to indexers repo]
@gotson seems like you have the passion and know how to do some testing with the definition and see hardcoding is still needed / if changes can be made to use dynamic cats like every other indexer and not a hardcoded config option?

[gotson]

[meta - xfered to indexers repo] @gotson seems like you have the passion and know how to do some testing with the definition and see hardcoding is still needed / if changes can be made to use dynamic cats like every other indexer and not a hardcoded config option?

Thanks for finding the historical reasons and some context, that helps knowing that the issue is in the indexer definition and not somewhere else. I will try to fix this, and if I get satisfactory results i'll submit a PR.

[bakerboy448]

@gotson if you're not already aware, you can test it out as a custom yml. swap the filename and id in the yml so it doesn't conflict

https://wiki.servarr.com/prowlarr/indexers#adding-a-custom-yml-definition

[gotson]

@gotson if you're not already aware, you can test it out as a custom yml. swap the filename and id in the yml so it doesn't conflict

https://wiki.servarr.com/prowlarr/indexers#adding-a-custom-yml-definition

yes, i figured that much from the doc already, thanks!

[gotson]

@bakerboy448 I would need some advice if possible.

YGGTorrent only accepts zero or one main category, and zero or one sub-category. The main and sub category parameters are different. Example:

• searching for TV, all types: ?category=5000 (using the prowlarr id, not the mapped id)
• searching for TV/HD: ?category=5000&sub_category=5040. ?sub_category=5040 also seems to work

If the same parameter is provided multiple times, only the last one is taken into account.

I would need to be able to use the right parameter, depending on the categories provided. I see that .Query.Categories provides the prowlarr IDs, and .Categories provides the mapped IDs.

I was thinking of checking the prowlarr ID to see whether it ends with 000, in which case it would be a main category, else it would be a sub-category. But the if conditions don't seem to be applicable to loops, and even if it did, I wouldn't have the mapped ID there.

[bakerboy448]

I'll defer to @Qstick on his thoughts for that

[gotson]

I'm also puzzled as how Jackett can handle this, given it has the same Cardigann definition 🤔

[bakerboy448]

I'm also puzzled as how Jackett can handle this, given it has the same Cardigann definition 🤔

Pretty sure jackett would have the same issue....

I think it may just be easiest to flip the category - using the same categories available in the drop down to the user - based on the parent category sent in the request from *arr if the keywords are empty

that'd be the simplest route

[ilike2burnthing]

Jackett/Jackett#14248

[ilike2burnthing]

Jackett/Jackett#14250

Removes old category setting and the new sub-cat setting from Jackett/Jackett#14248 earlier.

Closest thing to addressing this issue other than performing a separate search for every selected sub-category.

See info note:

To avoid unnecessary additional requests, it's recommended to only use indexer-specific categories (>=100000) when configuring this indexer in Sonarr, Radarr, and Lidarr, or when manually searching in TV, Movies, and Audio categories.

[ilike2burnthing]

resolved in 0bbd604

[bakerboy448]

Any fix would be related to / would likely need to fix Jackett/Jackett#13697

[ilike2burnthing]

Jackett/Jackett#14250 actually takes advantage of Jackett/Jackett#13697 not being fixed, as we only want to use category=all when no categories are selected.

Short of performing individual searches for every selected subcategory, as far as I can see this issue is resolved. Or rather it was, #333 temporarily reverted the fix due to Prowlarr/Prowlarr#1628.

[bakerboy448]

@mynameisbogdan Prowlarr/Prowlarr@ef1ad59 towards this?

[mynameisbogdan]

Seem like it since you mentioned Jackett/Jackett#13697