[Request] Site Support: baraag.net

[biznizz]

It looks like there's a twitter alternative called baraag.net for NSFW artists. Was wondering if it could be considered up for support consideration.

Here's a SFW post: https://baraag.net/@ADullPerson/103909177325116150

[mikf]

It's a Mastodon instance.

There is already a way for you to manually add support for any instance you want (see #144),
but that's nowhere documented at the moment ...

2587296 adds "global" support for baraag.net (as well as mastodon.social), so you don't need to do anything extra in that regard.

[biznizz]

Cool beans. Updated to the latest dev build and it worked!

Just a few followup questions before I go.

I've noticed that there's a Mastodon structure listed for the example-config in the docs bin; since baraag is part of Mastodon, is that structure necessary to add to my config (like to get full/"orig" pictures) or is that unnecessary to get the most out of the baraag extractor?

And if it is necessary, how would a baraag configuration be structured, as an example?

[mikf]

You can configure baraag like any other extractor, deviantart for example, by putting an baraag entry inside the extractor block and specifying a filename format etc in it. (*)

The Mastodon structure is there to let gallery-dl dynamically create extractors for Mastodon instances with the given OAuth credentials for them. The important part here is the access-token, which brings us to rate limits.

Back in the day, pawoo.net (and probably most Mastodon instances) had a rate limit of something like 100,000 requests every 5 minutes, and having a global access-token for all users was possible. That's not the case anymore, as I found out yesterday. Now you only get 300 requests every 5 minutes for each access token, so I'd advise you to get your own with gallery-dl oauth:mastodon:baraag.net. gallery-dl still has its own builtin token, but that won't be enough when more people start using it.

(I've also added some code that waits if the rate limit gets reached (220c06b), but that should never happen if you're the only one who uses your specific access token)

(*) config example

{
    "extractor":
    {
        "baraag":
        {
            "filename": "{filename}.{extension}"
        },
        "mastodon":
        {
            "baraag.net": {
                "access-token": "..."
            },
            "pawoo.net" : {
                "access-token": "..."
            }
        }
    }
}

[biznizz]

Alright, got an Oauth token, though the example claims that there can be a client-id and client-secret can also be created.

Is that similar to Deviantart where you register an app, because it looks very similar, but there isn't a step-by-step in the docs bin.

Also, I tried to run gallery-dl oauth:mastodon:pawoo.net, but I keep getting an The redirect uri included is not valid. error page, this is after I tried to run it, but it wanted me to login/register (which I did). I'm not getting the typical Ouath token page for pawoo. The configuration.rst page for getting an access-token for Pawoo is also just blank as well.

[mikf]

Is that similar to Deviantart where you register an app, because it looks very similar, but there isn't a step-by-step in the docs bin.

They are both using OAuth2, so yes, this is very similar.
Registering an app for an unknown Mastodon instance happens automatically, you don't need to do anything extra for that to happen.

For pawoo.net I've manually, more or less by hand, registered an "app" several years ago and used something as a redirect URL, which isn't compatible anymore it seems. I've set up a new "app" and updated the OAuth credentials with 88fca0a. Getting an access token for pawoo.net should work now.

The configuration.rst page for getting an access-token for Pawoo is also just blank as well.

Ah yes, this is also from "several years ago", way before I added support for basically all mastodon instances. I should go and update that ...

[Hrxn]

{
    "extractor":
    {
        "baraag":
        {
            "filename": "{filename}.{extension}"
        },
        "mastodon":
        {
            "baraag.net": {
                "access-token": "..."
            },
            "pawoo.net" : {
                "access-token": "..."
            }
        }
    }
}

Okay, so in these cases (a Mastodon instance like Pawoo) the "access-token" has to be set in extractor.mastodon."pawoo.net" and not in extractor.pawoo, where options like "filename" etc. would belong, right?

[biznizz]

Alright, updated to the latest dev build and managed to get an access-token for pawoo.

Only thing I'd like to know is how to get client-id and client-secret for Pawoo and baraag since I only know how to do that for Deviantart.

Thank you for all your hard work!

[mikf]

@Hrxn You can use both, extractor.mastodon."pawoo.net" as well as extractor.pawoo, but for unknown Mastodon instances you need to have an entry with the instance's domain inside extractor.mastodon, so you might as well put the access-token in there.

@biznizz you don't need your own client-id and client-secret like you do for DeviantArt. On Mastodon you have a separate rate limit for each access-token, so just having your own token is more than enough here.

edit: if you really want to get your own client-id/-secret, you can do that by opening an interactive session in your Python interpreter and calling _register() from an OAuthMastodon instance:

>>> import gallery_dl.extractor as e
>>> extr = e.find("oauth:mastodon:baraag.net")
>>> extr.instance
'baraag.net'
>>> extr._register(extr.instance)
{'id': '...', 'name': '...', 'website': None, 'redirect_uri': 'http://localhost:6414/', 'vapid_key': '...', 'client-id': '...', 'client-secret': '...'}

[biznizz]

Wow, thank you. I understand that getting my own client-ids/secrets is unnecessary, but I like to be prepared.

Have used your method to put in client-ids/secrets for both pawoo and baraag, thank you very much!