[Q] Do I need session cookies if I use OAuth ?

[Galewin]

I setup OAuth for DA, but I still get a warning message telling me it can't scratch mature content because of no session ID. Isn't it the point to use OAuth, to not use cookies everytime ?

I use the suggested firefox extension to get a .txt file with cookies (btw, can't find the difference between the button for "deviantart.com" and "www.deviantart.com"), but it now gets invalid in like 24 hours wich is really annoying.

[biznizz]

You need your browser cookies to download mature scraps since DA's system requires it. Ideally, this is what you'd need to get everything from a DA scrape:

1. Register your gallery-dl as a DA app using this guide

2. Refresh your Oauth as the above suggests after registering the app. Each token lasts about 3 months, so you'll need to run Oauth at least once per quarter.

3. Export your DA cookies using the Firefox extension. My suggestion is to copy the DA cookies into a separate txt file (both .deviantart.com and www.deviantart.com) and have your config link to that. It's fairly common that when the cookies update, they'll get corrupted whenever you try getting a lot of scraps, so you can just export your cookies again, overwrite the old exported txt and copy the new cookies over. I strongly suggest getting Notepad++ to keep track of your txt files for easy editing (especially your config files!)

Your extractor should look similar to this (relevant info is will have ** surrounding the line **)

"deviantart":
		{
			"include": "gallery,scraps",
			"refresh-token": "cache",
            **"client-id": "SECRET",**
			**"client-secret": "SECRET",**
			"flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": true,
            "original": true,
		    "quality": 100,
		    "extra": true,
            "wait-min": 0,
			**"cookies": "C:\\Users\\USER\\cookies.txt",**
			**"cookies-update": true**
        },

[mikf]

DeviantArt's OAuth API has no endpoint to fetch scraps, so gallery-dl uses the website itself for that and you need to be logged in (with cookies) to be able to access mature-rated content that way. This is far from optimal or user friendly, but there doesn't appear to be another way.

The exported cookies become invalid rather fast when you keep browsing dA with the same login session. You could try opening a private browser window, registering a new account, logging in with "Keep me logged in" enabled, enabling "Display Mature Content" in the dA settings, exporting the cookies from that, and then never touching that account again.

[Galewin]

Thanks for your answer Biznizz, but I already knew that and everything was correctly setup, sorry if I explained poorly.

DeviantArt's OAuth API has no endpoint to fetch scraps, so gallery-dl uses the website itself for that and you need to be logged in (with cookies) to be able to access mature-rated content that way. This is far from optimal or user friendly, but there doesn't appear to be another way.

The exported cookies become invalid rather fast when you keep browsing dA with the same login session. You could try opening a private browser window, registering a new account, logging in with "Keep me logged in" enabled, enabling "Display Mature Content" in the dA settings, exporting the cookies from that, and then never touching that account again.

Ho, I didn't know cookies could also expire based on the number of request you do, that explain why it expires so fast now I scrap like 10 artists.

I have a question then, what is the purpose of the "cookies-update" set to true in my config file for DA ?

[mikf]

It's not number of requests, but using the same session in multiple locations at once (browser + gallery-dl), combined with how dA handles its session cookies, that causes the session to become invalid for gallery-dl ... I think.

cookies-update is there to tell gallery-dl to update the cookies in your cookies.txt file after it is done. With this option set to false, cookies,txt files are treated as read-only, which doesn't work for dA. (cookies-update is true by default, btw)

[Galewin]

It's not number of requests, but using the same session in multiple locations at once (browser + gallery-dl), combined with how dA handles its session cookies, that causes the session to become invalid for gallery-dl ... I think.

cookies-update is there to tell gallery-dl to update the cookies in your cookies.txt file after it is done. With this option set to false, cookies,txt files are treated as read-only, but that doesn't work for dA. (cookies-update is true by default, btw)

Thank you for the clarification.

Just an idea like that, would it be feasible in theory to indicate the internet browser folder you use, and extract the most recent session cookie ? Sorry if it's a stupid question, just being curious.

[rautamiekka]

Just an idea like that, would it be feasible in theory to indicate the internet browser folder you use, and extract the most recent session cookie ? Sorry if it's a stupid question, just being curious.

If either condition is met: the browser

1. has committed the changes to the db while still running.
2. is shut down.

The 1st one has the problem that with SQLite3 the app's likely using a sorta a cache for uncommitted data, which if contains the up-to-date cookies will insta-invalidate the existing ones committed into the db.

^ I'm not 101-sure but it seems the cache's likely to be used even when shutting down the app, so it's still not 100% certain you'll get the up-to-date cookies, unless gallery-dl runs the cache file separately.

[mikf]

Just an idea like that, would it be feasible in theory to indicate the internet browser folder you use, and extract the most recent session cookie ? Sorry if it's a stupid question, just being curious.

There is --cookies-from-browser since 6742f3b (v1.22.0).

There was a time when mature scraps could be downloaded without cookies (#1777), but that is again no longer possible (#3129).