Not downloading from twitter private accounts anymore

[ghost]

i tried to download images from a private twitter account but it raised this error: [twitter][error] HttpError: '404 Not Found' for 'https://twitter.com/i/profiles/show/nin_1014/media_timeline'
I checked that my twitter loggin was properly set in gallery-dl.conf and i checked that i could still access the posts from my account
The verbose returns this:

[gallery-dl][debug] Python 3.6.7 - Windows-10-10.0.18362-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.3
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/mcmcim/media'
[twitter][debug] Using TwitterMediaExtractor for 'https://twitter.com/mcmcim/media'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/mcmcim/media_timeline?include_available_features=1&include_entities=1&reset_error_state=false&lang=en HTTP/1.1" 404 65
[twitter][error] HttpError: '404 Not Found' for 'https://twitter.com/i/profiles/show/mcmcim/media_timeline'```

[mikf]

I'd guess your cached session cookies are outdated/invalid. Try forcing a re-login by either clearing your entire cache with gallery-dl --clear-cache or by deleting the twitter entry from your cache database with for example sqlitebrowser.

[ghost]

well i did that and when login in it raised a new error

[gallery-dl][debug] Python 3.6.7 - Windows-10-10.0.18362-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24.3
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/mcmcim/media'
[twitter][debug] Using TwitterMediaExtractor for 'https://twitter.com/mcmcim/media'
[twitter][info] Logging in as *********@gmail.com
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /login HTTP/1.1" 200 None
[twitter][error] An unexpected error occurred: ValueError - substring not found. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "c:\users\mattl\appdata\local\programs\python\python36\lib\site-packages\gallery_dl\job.py", line 49, in run
    for msg in self.extractor:
  File "c:\users\mattl\appdata\local\programs\python\python36\lib\site-packages\gallery_dl\extractor\twitter.py", line 42, in items
    self.login()
  File "c:\users\mattl\appdata\local\programs\python\python36\lib\site-packages\gallery_dl\extractor\twitter.py", line 113, in login
    self._update_cookies(self._login_impl(username, password))
  File "c:\users\mattl\appdata\local\programs\python\python36\lib\site-packages\gallery_dl\cache.py", line 115, in __call__
    value = self.func(*args, **kwargs)
  File "c:\users\mattl\appdata\local\programs\python\python36\lib\site-packages\gallery_dl\extractor\twitter.py", line 121, in _login_impl
    pos = page.index('name="authenticity_token"')
ValueError: substring not found```

I'm using double authentification so this might be the cause of the issue?

[mikf]

I'm using double authentification so this might be the cause of the issue?

This would also be problem, but that's not why it's failing here.

At least on my Twitter login page, there's an authenticity_token value that needs to be sent alongside your username&password when logging in, but that's not the case for you and it "crashes" because it can't find it. (Would be nice to know what the HTML source of your login page looks like)

Since the "regular" login doesn't work, you could export your browser cookies to a cookies.txt file, load them with --cookies, and see if that works.

Edit: I did remember that right; we've had a very similar "discussion" before: #214

[ghost]

If you want the HTML, here it is:
Login on Twitter.zip
(I took it from my browser so i don't know if it's the same one as when gallery-dl tried to log in)

I did remember that right; we've had a very similar "discussion" before

I completly forgot about that x)
Though this time it wasn't the same issue as the previous fix didn't worked

[ghost]

also i'm going to try the cookie method to see if it work

[mikf]

See #598 (comment)

Twitter seems to have updated the login page, but you can force the old version by using "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" as user-agent