[Twitter] "Following" extractor breaks when you are subscribed to the platform-banned user

[oxi7589]

TwitterFollowingExtractor returns an empty list of users for accounts whose following list contains someone who is banned from platform. In most cases the offending account just disappears from the list, but it looks like in some cases it lingers on for longer. My assumption is that this happens when the "almost banned" account is appealing the ban decision.

(venv) PS> python .\gallery_dl\ -j --limit 1 --cookies cookies-twitter-com.txt https://twitter.com/[MyScreenName]/following
[]

GraphQL response contains the following error, but otherwise is complete and perfectly usable, at least for my purposes:

(HTTP 200 OK)

{
  "errors": [
    {
      "message": "_Missing: User not found.",
      "locations": [
        {
          "line": 707,
          "column": 3
        }
      ],
      "path": [
        "user",
        "following_timeline",
        "timeline",
        "instructions",
        2,
        "entries",
        88,
        "content",
        "itemContent",
        "user",
        "legacy"
      ],
      "extensions": {
        "name": "GenericError",
        "source": "Server",
        "code": 50,
        "kind": "NonFatal",
        "tracing": {
          "trace_id": "8d90565bf6d18343"
        }
      },
      "code": 50,
      "kind": "NonFatal",
      "name": "GenericError",
      "source": "Server",
      "tracing": {
        "trace_id": "8d90565bf6d18343"
      }
    }
  ],
  "data": {
    (normal response continues here)
  }

The data[] element for the "almost banned" user looks like this:

[
  6,
  "https://twitter.com/i/user/1288019650152824832",
  {
    "id": "VXNlcjoxMjg4MDE5NjUwMTUyODI0ODMy",
    "rest_id": "1288019650152824832"
  }
],

i.e. it has no "legacy" section. Note that when this user did have their own screen name, it was not "_Missing".

The problem with that is that TwitterAPI()._call() raises exception when it sees non-empty errors list and a status code of 200. I was able to successfully extract my following list by commenting out these two lines:

            # if msg and response.status_code < 400:
            #     raise exception.StopExtraction(msg)

but this probably affects other parts of the Twitter-related code in a negative way since this is not only used for the "following" thing.

I am thus seeking help from the gallery-dl maintainers to come up with a solution that fixes the "following" extractor without breaking anything else in the process.

[mikf]

Should be fixed with 4377f1c by not treating these errors as fatal and only printing a warning instead. I wasn't able to test this on a live example myself, so please let me know if it actually does what it is supposed to.

[oxi7589]

Sadly the twitter that was causing the problem has since been banned for good as is no longer returned by the "following" API. Reproducing this problem artificially is too tricky, as if the account is removed voluntarily by its owner it doesn't happen. I did, however, manage to do some testing by hacking gallery-dl to use my saved .json responses from when it was happening, and that has worked as expected.