Twitter Extractor Issue

[LottieVixen]

PowerShell command run:

gallery-dl --verbose -c C:\Users\foxsa\gallery-dl.conf --write-unsupported videotweets -o videos=true --cookies C:\Users\foxsa\twitter.txt "twitter.com/Foxsan48"

Output:

[gallery-dl][debug] Version 1.5.3
[gallery-dl][debug] Python 3.4.4 - Windows-10-10.0.17134
[gallery-dl][debug] requests 2.19.1 - urllib3 1.23
[gallery-dl][debug] Starting DownloadJob for 'twitter.com/Foxsan48'
[twitter][debug] Using TwitterTimelineExtractor for 'twitter.com/Foxsan48'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/Foxsan48/timeline/tweets?include_available_features=1&include_entities=1&lang=en&reset_error_state=false HTTP/1.1" 200 17807
[twitter][error] An unexpected error occurred: KeyError - 'items_html'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug] Traceback
Traceback (most recent call last):
  File "E:\gallery-dl\gallery_dl\job.py", line 64, in run
  File "E:\gallery-dl\gallery_dl\extractor\twitter.py", line 32, in items
  File "E:\gallery-dl\gallery_dl\extractor\twitter.py", line 86, in _tweets_from_api
KeyError: 'items_html'

I think the expected response has changed, not sure as I've not committed to this repo before.

System Windows 10, Powershell, config standard, youtube-dl installed

hopefully that's more than enough info

Amendment 1: forgot to put the command used to launch this at the start

[LottieVixen]

This still exists on the master branch also obtained via

pip3.6.exe install --upgrade https://github.com/mikf/gallery-dl/archive/master.zip

PowerShell command run:

gallery-dl --verbose -c C:\Users\foxsa\gallery-dl.conf --write-unsupported videotweets -o videos=true --cookies C:\Users\foxsa\twitter.txt "twitter.com/Foxsan48"

output:

[gallery-dl][debug] Version 1.6.0-dev
[gallery-dl][debug] Python 3.6.5 - Windows-10-10.0.17134-SP0
[gallery-dl][debug] requests 2.20.1 - urllib3 1.24.1
[gallery-dl][debug] Starting DownloadJob for 'twitter.com/Foxsan48'
[twitter][debug] Using TwitterTimelineExtractor for 'twitter.com/Foxsan48'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/Foxsan48/timeline/tweets?include_available_features=1&include_entities=1&reset_error_state=false&lang=en HTTP/1.1" 200 17799
[twitter][error] An unexpected error occurred: KeyError - 'items_html'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug] Traceback
Traceback (most recent call last):
  File "c:\program files (x86)\python36-32\lib\site-packages\gallery_dl\job.py", line 51, in run
    for msg in self.extractor:
  File "c:\program files (x86)\python36-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 33, in items
    for tweet in self.tweets():
  File "c:\program files (x86)\python36-32\lib\site-packages\gallery_dl\extractor\twitter.py", line 89, in _tweets_from_api
    data["items_html"], '<div class="tweet ', '\n</li>'):
KeyError: 'items_html'

[mikf]

Twitter returns a more detailed API response when you are logged in:

{
  "inner": {
    "min_position": "1048907902361075713",
    "max_position": "1056382060560969728",
    "has_more_items": true,
    "items_html": "...",
    "new_latent_count": 20
  },
  "note": {
    "d": {
      "status": "ok",
      "response": null
    },
    "b": {
      "status": "ok",
      "response": {...}
    },
    "t": {
      "status": "ok",
      "response": null
    }
  }
}

... and gallery-dl only expected the inner portion. Should be fixed now.

On another note: The master-branch version has direct youtube-dl support. You don't need to do the --write-unsupported "trick" anymore if youtube-dl is installed as a Python package.