Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[twitter][bug] show error sometimes #2216

Closed
peammm opened this issue Jan 22, 2022 · 6 comments
Closed

[twitter][bug] show error sometimes #2216

peammm opened this issue Jan 22, 2022 · 6 comments
Labels
bug

Comments

@peammm
Copy link

peammm commented Jan 22, 2022 

Version 1.20.2-dev
Python 3.7.3 - Linux-4.4.180+-x86_64-with-debian-10.8
requests 2.27.1 - urllib3 1.26.8

first I see thie error

[twitter][error] An unexpected error occurred: KeyError - 'timeline'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .

[twitter][error] An unexpected error occurred: KeyError - 'rest_id'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .

add --verbose it show like this

[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/xxx'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/xxx'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A+%22xxx%22%2C+%22withSafetyModeUserFields%22%3A+true%2C+%22withSuperFollowsUserFields%22%3A+true%7D HTTP/1.1" 200 969
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A+%221104019745%22%2C+%22count%22%3A+100%2C+%22includePromotedContent%22%3A+true%2C+%22withQuickPromoteEligibilityTweetFields%22%3A+true%2C+%22withSuperFollowsUserFields%22%3A+true%2C+%22withBirdwatchPivots%22%3A+false%2C+%22withDownvotePerspective%22%3A+false%2C+%22withReactionsMetadata%22%3A+false%2C+%22withReactionsPerspective%22%3A+false%2C+%22withSuperFollowsTweetFields%22%3A+true%2C+%22withVoice%22%3A+true%2C+%22withV2Timeline%22%3A+false%2C+%22__fs_interactive_text%22%3A+false%2C+%22__fs_dont_mention_me_view_api_enabled%22%3A+false%7D HTTP/1.1" 200 80
[twitter][error] An unexpected error occurred: KeyError - 'timeline'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug] 
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/job.py", line 79, in run
    for msg in extractor:
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 61, in items
    for tweet in self.tweets():
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 1101, in _pagination_tweets
    instructions = (data["user"]["result"]["timeline"]
KeyError: 'timeline'



[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/xxx'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/xxx'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A+%22xxx%22%2C+%22withSafetyModeUserFields%22%3A+true%2C+%22withSuperFollowsUserFields%22%3A+true%7D HTTP/1.1" 200 704
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A+%2210%22%2C+%22count%22%3A+100%2C+%22includePromotedContent%22%3A+true%2C+%22withQuickPromoteEligibilityTweetFields%22%3A+true%2C+%22withSuperFollowsUserFields%22%3A+true%2C+%22withBirdwatchPivots%22%3A+false%2C+%22withDownvotePerspective%22%3A+false%2C+%22withReactionsMetadata%22%3A+false%2C+%22withReactionsPerspective%22%3A+false%2C+%22withSuperFollowsTweetFields%22%3A+true%2C+%22withVoice%22%3A+true%2C+%22withV2Timeline%22%3A+false%2C+%22__fs_interactive_text%22%3A+false%2C+%22__fs_dont_mention_me_view_api_enabled%22%3A+false%7D HTTP/1.1" 200 27872
[twitter][error] An unexpected error occurred: KeyError - 'legacy'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug] 
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/job.py", line 79, in run
    for msg in extractor:
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 61, in items
    for tweet in self.tweets():
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 1141, in _pagination_tweets
    legacy = tweet["legacy"]
KeyError: 'legacy'



[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/xxx'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/xxx'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A+%22xxx%22%2C+%22withSafetyModeUserFields%22%3A+true%2C+%22withSuperFollowsUserFields%22%3A+true%7D HTTP/1.1" 200 262
[twitter][error] An unexpected error occurred: KeyError - 'rest_id'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug] 
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/job.py", line 79, in run
    for msg in extractor:
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 61, in items
    for tweet in self.tweets():
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 397, in tweets
    return TwitterAPI(self).user_tweets(self.user)
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 732, in user_tweets
    "userId": self._user_id_by_screen_name(screen_name),
  File "/usr/local/lib/python3.7/dist-packages/gallery_dl/extractor/twitter.py", line 944, in _user_id_by_screen_name
    return self.user_by_screen_name(screen_name)["rest_id"]
KeyError: 'rest_id'
@nisehime
Copy link

I have rest_id error for banned users. Also I have IndexError for this user for some reasons (other show normal "No results" error)

F:\gallery-dl>gallery-dl.exe -v https://twitter.com/yaranmonisans/media
[gallery-dl][warning] logfile: missing or invalid path (expected str, bytes or os.PathLike object, not NoneType)
[gallery-dl][debug] Version 1.20.2-dev - Executable
[gallery-dl][debug] Python 3.9.9 - Windows-8.1-6.3.9600-SP0
[gallery-dl][debug] requests 2.25.1 - urllib3 1.25.11
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/yaranmonisans/media'
[twitter][debug] Using TwitterMediaExtractor for 'https://twitter.com/yaranmonisans/media'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A+%22yaranmonisans%22%2C+%22withSafetyModeUserFields%22%3A+true%2C+%22withSuperFollowsUserFields%22%3A+true%7D HTTP/1.1" 200 968
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/NeBRhmkLMUxQkGpBtQ4k5w/UserMedia?variables=%7B%22userId%22%3A+%221350082281193369604%22%2C+%22count%22%3A+100%2C+%22includePromotedContent%22%3A+false%2C+%22withSuperFollowsUserFields%22%3A+true%2C+%22withBirdwatchPivots%22%3A+false%2C+%22withDownvotePerspective%22%3A+false%2C+%22withReactionsMetadata%22%3A+false%2C+%22withReactionsPerspective%22%3A+false%2C+%22withSuperFollowsTweetFields%22%3A+true%2C+%22withClientEventToken%22%3A+false%2C+%22withBirdwatchNotes%22%3A+false%2C+%22withVoice%22%3A+true%2C+%22withV2Timeline%22%3A+false%2C+%22__fs_interactive_text%22%3A+false%2C+%22__fs_dont_mention_me_view_api_enabled%22%3A+false%7D HTTP/1.1" 200 132
[twitter][error] An unexpected error occurred: IndexError - list index out of range. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "gallery_dl\job.py", line 79, in run
  File "gallery_dl\extractor\twitter.py", line 61, in items
  File "gallery_dl\extractor\twitter.py", line 1114, in _pagination_tweets
IndexError: list index out of range

@mikf
Copy link
Owner

mikf commented Jan 23, 2022

KeyError: 'timeline' is fixed with 9ca8bb2
KeyError: 'rest_id' and IndexError: list index out of range get fixed in a9f78e6

I'd assume KeyError: 'legacy' is from some sort of deleted Tweet, but I haven't been able to encounter this error myself. Could you provide a concrete example where this error happens, @peamcc ?

@mikf mikf added the bug label Jan 23, 2022
@yekyau
Copy link

yekyau commented Jan 23, 2022 •
edited
Loading

I am encountering KeyError - 'entryId' on Twitter users with pinned posts (e.g. gallery-dl https://twitter.com/nocontextmugen). It is resolved if I change extractor.twitter.pinned to false.

mikf added a commit that referenced this issue Jan 23, 2022
@mikf
[twitter] fix pinned tweets (#2216)
5ed26e1 
caused by the changes in dffa440
@mikf
Copy link
Owner

mikf commented Jan 23, 2022

@yekyau fixed (5ed26e1)

@Klonoa723
Copy link

Hello all, just came here to check regarding a similar problem I'm having with twitter as well.

[gallery-dl][debug] Version 1.20.2 - Executable
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.8
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/xxx'
[twitter][debug] Using TwitterTimelineExtractor for 'https://twitter.com/xxx'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A%22freesmellsinc%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D HTTP/1.1" 200 946
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A%221086688842588086272%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%7D HTTP/1.1" 200 88934

[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/DhQ8lYnLh5T5K8aVUgHVnQ/UserTweets?variables=%7B%22userId%22%3A%221086688842588086272%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%2C%22cursor%22%3A%22HBaCwLiZ3ZTrjCkAAA%3D%3D%22%7D HTTP/1.1" 200 88477

[twitter][error] An unexpected error occurred: KeyError - 'core'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "gallery_dl\job.pyc", line 79, in run
  File "gallery_dl\extractor\twitter.pyc", line 62, in items
  File "gallery_dl\extractor\twitter.pyc", line 1140, in _pagination_tweets
KeyError: 'core'

I'm not sure if maybe this could be caused by my config file maybe not being updated and such?
Since I've been only replacing the exe whenever a new version came out.

mikf added a commit that referenced this issue Jan 25, 2022
@mikf
[twitter] fix several errors (#2212, #2216, #2225)
2bf554a 
- fix Tweets with deleted quotes
- fix suspended Tweets without 'legacy' entry
- fix unified_cards without 'type'
@mikf mikf mentioned this issue Jan 25, 2022
Closed
@mikf
Copy link
Owner

mikf commented Jan 25, 2022

KeyError: 'core' and KeyError: 'legacy' are now fixed in 2bf554a.
Has anyone found more, before I push the next release?

@mikf mikf closed this as completed Jan 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@yekyau @mikf @peammm @nisehime @Klonoa723