[NSFW] [Aryion] missing folders in certain galleries

[ghost]

I'm backing up a few galleries on this site, and I've come across a couple where gallery-dl misses content in folders (unless I specifically run the command for each folder url).

Affected galleries:
https://aryion.com/g4/gallery/acidbath
https://aryion.com/g4/gallery/Anchors

$ gallery-dl -v https://aryion.com/g4/gallery/Anchors
https://pastebin.com/wucpFGp9

Expected: It downloads all items in the gallery.
Result: It only downloads items outside of folders.

Let me know if there's any other information I should provide, I've never posted an issue on github before. Thanks!

[kattjevfel]

Did you provide gallery-dl with a cookies file? Otherwise it will skip submissions hidden from the public.

[ghost]

Did you provide gallery-dl with a cookies file? Otherwise it will skip submissions hidden from the public.

Yes, and for other artists it’s getting everything (public or not, folders included), just not these two. Most of Anchor’s gallery is public anyway.

[mikf]

gallery-dl is using the "Latest Updates" list like https://aryion.com/g4/latest.php?id=431815 to get "all" items of a specific user, but that doesn't include everything in these (and probably more) cases.

You can use the folder links for those two users and gallery-dl will recursively grab everything:

• https://aryion.com/g4/view/348883
• https://aryion.com/g4/view/431815

Maybe it should internally be using those instead of https://aryion.com/g4/latest.php?id=..., but then there wouldn't be a predictable "from newest to oldest" order

[ghost]

Those links work perfectly for me, thanks. [EDIT: Correction, the Anchor one works, but the acidbath one (https://aryion.com/g4/view/348883) doesn't. I still have to run gallery-dl for every individual folder there.]

The file names all start with the image ID(?), which seems to be in chronological order anyway. I don't see why that wouldn't work, but maybe I'm missing something obvious.

Edit 2: Oddly enough, the folder link doesn't work for this artist, but the profile link does.

$ gallery-dl https://aryion.com/g4/view/86647 doesn't find anything

$ gallery-dl https://aryion.com/g4/user/HipHugger works perfectly

Edit 3: I've accidentally closed and reopened the issue, I'm gonna stop pressing buttons now. Sorry.

[mikf]

I've changed the default behavior to recursively go through a user's gallery page instead of using the "Latest Updates" listing, because the latter might not contain all posts. (There is a recursive option to restore the old behavior, by the way)

I've also added a warning message for when posts/folders can't be opened. For example https://aryion.com/g4/view/86647 now shows a 401 or 403 error depending on cookie usage instead of showing no output at all:
(but https://aryion.com/g4/user/HipHugger works, as does https://aryion.com/g4/gallery/acidbath)

$ gallery-dl https://aryion.com/g4/view/86647
[aryion][warning] Unable to fetch post 86647 ('401 LOGIN REQUIRED')
$ gallery-dl --cookies cookies-aryion-com.txt https://aryion.com/g4/view/86647
[aryion][warning] Unable to fetch post 86647 ('403 FORBIDDEN')