possible bug: Trying to understand how to download from some sites.

[Butterfly-Dragon]

So.
I am trying to understand if i am doing things wrong or if there is an error somewhere.

My base config starts with this:

{
    "extractor":
    {
        "base-directory": "D:/Downloads/Downloader/Manga/",
		"directory": ["Files"],
	    "filename": "{filename}.{extension}",
        "archive": "D:/Downloads/Downloader//!Downloader/SQL/gallery-dl-archive.sqlite3",
	    "cache.file": "D:/Downloads/Downloader//!Downloader/SQL/tmp/cache.sqlite3",
        "skip": "abort:3",
		"retries": -1,
        "sleep": 0,
        "restrict-filenames": "auto",
		"timeout": null,
        "postprocessors": null,
		"cookies": "D:/Downloads/Downloader//!Downloader/cookies.txt",
		"category-transfer": true,
		"parent-directory": true,
		"adjust-extensions": true,
		"refresh-token": "cache",
		"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.9 Safari/537.36",
		
        "oauth":
        {
            "browser": true
        },

For FurAffinity i have this set up

        "furaffinity":
        {
			"include": ["gallery", "scraps", "favorite"],
            "filename": "{filename} - {title}.{extension}"
		},

if insert a random user name and i get results like this:

[gallery-dl][debug] Starting DownloadJob for 'https://www.furaffinity.net/user/muetank'
[furaffinity][debug] Using FuraffinityUserExtractor for 'https://www.furaffinity.net/user/muetank'
[furaffinity][debug] Using FuraffinityGalleryExtractor for 'https://www.furaffinity.net/gallery/muetank/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.furaffinity.net:443
[urllib3.connectionpool][debug] https://www.furaffinity.net:443 "GET /user/muetank/1/ HTTP/1.1" 404 None
[furaffinity][error] HttpError: '404 Not Found' for 'https://www.furaffinity.net/user/muetank/1/'
[furaffinity][debug] Using FuraffinityScrapsExtractor for 'https://www.furaffinity.net/scraps/muetank/'
[urllib3.connectionpool][debug] https://www.furaffinity.net:443 "GET /user/muetank/1/ HTTP/1.1" 404 None
[furaffinity][error] HttpError: '404 Not Found' for 'https://www.furaffinity.net/user/muetank/1/'
[furaffinity][debug] Using FuraffinityFavoriteExtractor for 'https://www.furaffinity.net/favorites/muetank/'
[urllib3.connectionpool][debug] https://www.furaffinity.net:443 "GET /favorites/muetank/ HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.furaffinity.net:443 "GET /view/40290138/ HTTP/1.1" 200 None
[furaffinity][debug] Using download archive 'D:/Downloads/Downloader//!Downloader/SQL/gallery-dl-archive.sqlite3'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): d.facdn.net:443
[urllib3.connectionpool][debug] https://d.facdn.net:443 "GET /art/arthon/1611544343/1611544328.arthon_good_time_with_friends_1280x720.png HTTP/1.1" 200 8308989
[urllib3.connectionpool][debug] https://www.furaffinity.net:443 "GET /view/40298405/ HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.furaffinity.net:443 "GET /view/40294745/ HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.furaffinity.net:443 "GET /view/40269095/ HTTP/1.1" 200 None

Which means it's downloading the "favorite" gallery but is missing the "gallery" and "scraps" galleries (they should go instead of "user" in the URL)

For newgrounds i tried putting down this as the relative config

		"newgrounds":
		{
			"include": ["movies", "art"],
            "filename": "{user} - {title} - {filename}.{extension}"
		},

but i get a similar result:

[gallery-dl][debug] Starting DownloadJob for 'https://cartesiancoordinator.newgrounds.com/'
[newgrounds][debug] Using NewgroundsUserExtractor for 'https://cartesiancoordinator.newgrounds.com/'
[newgrounds][debug] Using NewgroundsMoviesExtractor for 'https://cartesiancoordinator.newgrounds.com/movies'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): cartesiancoordinator.newgrounds.com:443
[urllib3.connectionpool][debug] https://cartesiancoordinator.newgrounds.com:443 "GET /user/page/1 HTTP/1.1" 403 57
[newgrounds][error] The page you requested could not be found.
[newgrounds][debug] Using NewgroundsArtExtractor for 'https://cartesiancoordinator.newgrounds.com/art'
[urllib3.connectionpool][debug] https://cartesiancoordinator.newgrounds.com:443 "GET /user/page/1 HTTP/1.1" 403 57
[newgrounds][error] The page you requested could not be found.

basically it's looking at the wrong url once again (i think)

but if i go to hentai-foundry:

        "hentaifoundry":
        {
			"include": ["pictures", "scraps"],
			"cookies": null,
            "filename": "{user} - {title} - {filename}.{extension}"
        },

everything seems to go alright

[gallery-dl][debug] Starting DownloadJob for 'https://www.hentai-foundry.com/user/tarakanovich/profile'
[hentaifoundry][debug] Using HentaifoundryUserExtractor for 'https://www.hentai-foundry.com/user/tarakanovich/profile'
[hentaifoundry][debug] Using HentaifoundryPicturesExtractor for 'https://www.hentai-foundry.com/pictures/user/tarakanovich'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.hentai-foundry.com:443
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "HEAD /?enterAgree=1 HTTP/1.1" 200 0
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "POST /site/filters HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/tarakanovich/page/1 HTTP/1.1" 301 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/page/1 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/848154/Wild-ride HTTP/1.1" 200 None
[hentaifoundry][debug] Using download archive 'D:/Downloads/Downloader//!Downloader/SQL/gallery-dl-archive.sqlite3'
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/848153/Jenny HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/848152/CG---Kallen HTTP/1.1" 200 None
[hentaifoundry][debug] Using HentaifoundryScrapsExtractor for 'https://www.hentai-foundry.com/pictures/user/tarakanovich/scraps'
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "HEAD /?enterAgree=1 HTTP/1.1" 200 0
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "POST /site/filters HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/tarakanovich/scraps/page/1 HTTP/1.1" 301 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/scraps/page/1 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/402163/Stream-Stream-Cream HTTP/1.1" 200 None
[hentaifoundry][debug] Using download archive 'D:/Downloads/Downloader//!Downloader/SQL/gallery-dl-archive.sqlite3'
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/381649/Stream-anounce HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "GET /pictures/user/Tarakanovich/373612/Doodle-coloring HTTP/1.1" 200 None

Sooo... it could be a bug, or maybe i'm doing things wrong, and i'm asking where is the problem. 😅

[mikf]

Your issues are caused by enabling category-transfer in general.
Remove the "category-transfer": true, line and this will work.

[mikf]

Also, there is no restrict-filenames option. You probably meant path-restrict.

And "cache.file": "PATH" needs to be "cache": {"file": "PATH"}
to have an effect. Your current cache file should be at %APPDATA%\gallery-dl\cache.sqlite3

[Butterfly-Dragon]

i believe the "restrict filename" used to be there, because i would not be able to come up with it without reading the docs. Might have been removed at some point. 😅 Also i'll just remove that since it's already defaulting to "auto"

so i will remove that and the category-transfer

the cache.file is still in the docs https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#cachefile

But i basically took (long ago) the example config file provided and started adding to it by reading the doc.

[mikf]

i believe the "restrict filename" used to be there, because i would not be able to come up with it without reading the docs

You're right. It was called restrict-filenames when it got added, and was then renamed to path-restrict in 1.10.2 when path-remove was added.

cache.file is still in the docs

Yeah, but, like with all config options in there, this means a file property inside a cache object, just like extractor.pixiv.ugoira is an ugoira property inside pixiv inside extractor:

{
    "extractor": {"pixiv": {"ugoira": true}},
    "cache": {"file": "PATH"}
}

The cache thing is a remnant from the early days, when I thought there would eventually be more cache options alongside file, but that never happened.

[Butterfly-Dragon]

yes, that ... is the kind of stuff i write at 3 in the morning 😂👍💖

thanks for clarifying.

And yes, once i did what you asked everything works now.