Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixed tiktok.py #14

Merged
merged 1 commit into from
Oct 26, 2020
Merged

Fixed tiktok.py #14

merged 1 commit into from
Oct 26, 2020

Conversation

merval
Copy link

@merval merval commented Oct 25, 2020

The JSON formatting of the response changed, corrected.

#TESTING
Successfully downloaded and played a tiktok video. Fixes #12

Fixed tiktok.py
1666c27 
The JSON formatting of the response changed, corrected.

#TESTING
Successfully downloaded and played a tiktok video
@blackjack4494
Copy link
Owner

Will have a look as soon as I am on the computer :)

@merval
Copy link
Author

merval commented Oct 25, 2020

Hmm. tiktok is now returning a 403 Forbidden when trying to access the video directly.

python3 -m youtube_dlc https://www.tiktok.com/@charlidamelio/video/6854565358808993029 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.tiktok.com/@charlidamelio/video/6854565358808993029', '--verbose']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.10.24-5
[debug] Git HEAD: 1666c2709
[debug] Python version 3.7.3 (CPython) - Darwin-19.6.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.3.1, ffprobe 4.3.1, rtmpdump 2.4
[debug] Proxy map: {}
[TikTok] 6854565358808993029: Downloading video webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603659497&l=20201025145802010189071071030393DC&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw%3D%3D&signature=4ae8c0eb394026e306c46217d88732cd&tk=tt_webid_v2&vl=&vr='
ERROR: unable to download video data: HTTP Error 403: Forbidden

The yt-dlc JSON is:

{"comment_count": null, "duration": null, "height": 1024, "id": "6854565358808993029", "like_count": null, "repost_count": null, "thumbnail": "https://p16-sign-va.tiktokcdn.com/tos-maliva-p-0068/6129799af243413a9a9833ffd5e43c84_1595952874~tplv-tiktok-play.jpeg?x-expires=1603656000&x-signature=Ibye24cGceIJogOaJDh2dQAGM2w%3D", "timestamp": 1595952872, "width": 576, "title": "charli d\u2019amelio on TikTok", "creator": null, "uploader": "charlidamelio", "uploader_id": null, "uploader_url": "https://www.tiktok.com/@charlidamelio", "thumbnails": [{"url": "https://p16-sign-va.tiktokcdn.com/tos-maliva-p-0068/6129799af243413a9a9833ffd5e43c84_1595952874~tplv-tiktok-play.jpeg?x-expires=1603656000&x-signature=Ibye24cGceIJogOaJDh2dQAGM2w%3D", "width": 576, "height": 1024, "resolution": "576x1024", "id": "0"}], "upload_date": "20200728", "webpage_url": "https://www.tiktok.com/@charlidamelio/video/6854565358808993029", "description": "momma @heididamelio dc @trvpandre  @dejuane.mccoy @justinplaness", "ext": "mp4", "formats": [{"url": "https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603659554&l=2020102514585901018806120930038701&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw%3D%3D&signature=143c87e656b8d2db05f2baaa8832b33c&tk=tt_webid_v2&vl=&vr=", "ext": "mp4", "height": 1024, "width": 576, "format_id": "0", "format": "0 - 576x1024", "protocol": "https", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.41 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5", "Referer": "https://www.tiktok.com/@charlidamelio/video/6854565358808993029", "Cookie": "tt_webid=6887572486863783429; tt_webid_v2=6887572486863783429"}}], "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.41 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5", "Referer": "https://www.tiktok.com/@charlidamelio/video/6854565358808993029", "Cookie": "tt_webid=6887572486863783429; tt_webid_v2=6887572486863783429"}, "extractor": "TikTok", "webpage_url_basename": "6854565358808993029", "extractor_key": "TikTok", "playlist": null, "playlist_index": null, "display_id": "6854565358808993029", "requested_subtitles": null, "url": "https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603659554&l=2020102514585901018806120930038701&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw%3D%3D&signature=143c87e656b8d2db05f2baaa8832b33c&tk=tt_webid_v2&vl=&vr=", "format_id": "0", "format": "0 - 576x1024", "protocol": "https"}

The page raw JSON is:

{'key': '6854565358808993029', 'serverCode': 200, 'statusCode': 0, 'statusMsg': None, 'itemInfo': {'itemStruct': {'id': '6854565358808993029', 'desc': 'momma @heididamelio dc @trvpandre  @dejuane.mccoy @justinplaness', 'createTime': 1595952872, 'video': {'id': 'awesome', 'height': 1024, 'width': 576, 'duration': 15, 'ratio': '720p', 'cover': 'https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/44a25ff93e034437b2d514760954d194?x-expires=1603659600&x-signature=pqKiloEoKlR7z6Dw5q%2BogwezOEQ%3D', 'originCover': 'https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/6129799af243413a9a9833ffd5e43c84_1595952874?x-expires=1603659600&x-signature=McJNE86rpuZf7P7S7ZtTbQV1KR0%3D', 'dynamicCover': 'https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/8bf3a434271a4206b05cf6ba65e16827_1595952874?x-expires=1603659600&x-signature=odTYheZif9hN3DsRR51oiPCUmr0%3D', 'playAddr': 'https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603659935&l=202010251505200101890360232903CB64&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw%3D%3D&signature=73ea321e4135d70bb78b5297c49fe930&tk=tt_webid_v2&vl=&vr=', 'downloadAddr': 'https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603659935&l=202010251505200101890360232903CB64&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw%3D%3D&signature=73ea321e4135d70bb78b5297c49fe930&tk=tt_webid_v2&vl=&vr=', 'shareCover': ['', 'https://p16-sign-va.tiktokcdn.com/tos-maliva-p-0068/6129799af243413a9a9833ffd5e43c84_1595952874~tplv-tiktok-play.jpeg?x-expires=1603659600&x-signature=EycyYLD%2BP6VCASxECMDicErSNoU%3D', 'https://p16-sign-va.tiktokcdn.com/tos-maliva-p-0068/6129799af243413a9a9833ffd5e43c84_1595952874~tplv-tiktok-play2.jpeg?x-expires=1603659600&x-signature=LvUhPNQ%2B1JOl8LlL%2FMEqYZvUTLo%3D'], 'reflowCover': 'https://p16-sign-va.tiktokcdn.com/obj/tos-maliva-p-0068/889e09a14814c1cf81b41713338b3989?x-expires=1603659600&x-signature=hzykHr8pk27JpSRlMzrz0rOTMoQ%3D'}, 'author': {'id': '5831967', 'uniqueId': 'charlidamelio', 'nickname': 'charli d’amelio', 'avatarThumb': 'https://p16-sign-va.tiktokcdn.com/musically-maliva-obj/0a0242e5f906bf9301f859c568b5ecc4~c5_100x100.jpeg?x-expires=1603724400&x-signature=uTvwcUizJFLVUeFH5bP43NAgqjk%3D', 'avatarMedium': 'https://p16-sign-va.tiktokcdn.com/musically-maliva-obj/0a0242e5f906bf9301f859c568b5ecc4~c5_720x720.jpeg?x-expires=1603724400&x-signature=Haaql4dZSJihgbt7g2%2BkY3qtYXE%3D', 'avatarLarger': 'https://p16-sign-va.tiktokcdn.com/musically-maliva-obj/0a0242e5f906bf9301f859c568b5ecc4~c5_1080x1080.jpeg?x-expires=1603724400&x-signature=8MMCWQGskH%2FEgLe9thoDqHl1v3Q%3D', 'signature': 'i am a megan thee stallion stan for life', 'verified': True, 'secUid': 'MS4wLjABAAAA-VASjiXTh7wDDyXvjk10VFhMWUAoxr8bgfO1kAL1-9s', 'secret': False, 'ftc': False, 'relation': 0, 'openFavorite': False, 'commentSetting': 0, 'duetSetting': 0, 'stitchSetting': 0, 'privateAccount': False}, 'music': {'id': '6816061305397496582', 'title': "It Won't Stop (feat. Chris Brown) - DWYR Radio Edit", 'playUrl': 'https://sf16-sg.tiktokcdn.com/obj/tiktok-obj/f37f9236b381c75f058d3cbd223cb0f4.mp3', 'coverThumb': 'https://p16-sg.tiktokcdn.com/aweme/100x100/tos-alisg-i-0000/2430841ff40c47e898030ff5ba98aea1.jpeg', 'coverMedium': 'https://p16-sg.tiktokcdn.com/aweme/200x200/tos-alisg-i-0000/2430841ff40c47e898030ff5ba98aea1.jpeg', 'coverLarge': 'https://p16-sg.tiktokcdn.com/aweme/720x720/tos-alisg-i-0000/2430841ff40c47e898030ff5ba98aea1.jpeg', 'authorName': 'Sevyn Streeter', 'original': False, 'duration': 32, 'album': "It Won't Stop (feat. Chris Brown) (Remixes)"}, 'stats': {'diggCount': 6900000, 'shareCount': 99700, 'commentCount': 95100, 'playCount': 54600000}, 'isActivityItem': False, 'duetInfo': {'duetFromId': '0'}, 'textExtra': [{'awemeId': '', 'start': 6, 'end': 19, 'hashtagName': '', 'hashtagId': '', 'type': 0, 'userId': '58201502112051200', 'isCommerce': False, 'userUniqueId': 'heididamelio', 'secUid': 'MS4wLjABAAAAGq7rP7UjRmSIgRfMGEpXGfLzKj9Tn1BwkmVED7Jb5zdGyWTd7oYtsz-AFyH1TZz9'}, {'awemeId': '', 'start': 23, 'end': 33, 'hashtagName': '', 'hashtagId': '', 'type': 0, 'userId': '219072163025494016', 'isCommerce': False, 'userUniqueId': 'trvpandre', 'secUid': 'MS4wLjABAAAApvS0-n3ATjFKIDWZfU--sgnlMP_ASVaA5oMg3RMKdm-4NENqHSI92bkC82wxOhzh'}, {'awemeId': '', 'start': 35, 'end': 49, 'hashtagName': '', 'hashtagId': '', 'type': 0, 'userId': '6636346770715672581', 'isCommerce': False, 'userUniqueId': 'dejuane.mccoy', 'secUid': 'MS4wLjABAAAABU4QMBairxmwvrY43kdaBJhhwqkXQ1V-J9MeJ0_kHJYbBhrvAvYIyHaDGW1BbeRf'}, {'awemeId': '', 'start': 50, 'end': 64, 'hashtagName': '', 'hashtagId': '', 'type': 0, 'userId': '6641605022291017734', 'isCommerce': False, 'userUniqueId': 'justinplaness', 'secUid': 'MS4wLjABAAAA_H86cFisI0YGU1xCaP2lA7-d0CXgstUHIADfB5RiQirAp9mEPRDYMvxrmqPwzfke'}], 'secret': False, 'forFriend': False, 'digged': False, 'itemCommentStatus': 0, 'showNotPass': False, 'vl1': False, 'itemMute': False, 'authorStats': {'followingCount': 1064, 'followerCount': 95000000, 'heartCount': 0, 'videoCount': 1616, 'diggCount': 6169, 'heart': 7300000000}, 'privateItem': False, 'duetEnabled': True, 'stitchEnabled': True, 'shareEnabled': True, 'isAd': False}, 'shareMeta': {'title': 'charli d’amelio on TikTok', 'desc': 'momma @heididamelio dc @trvpandre  @dejuane.mccoy @justinplaness', 'image': {}}}, 'predictedLanguage': 'en', 'metaParams': {'robotsContent': 'index, follow', 'isError': False, 'title': 'momma @heididamelio dc @trvpandre  @dejuane.mccoy @justinplaness', 'keywords': 'charli d’amelio,charlidamelio,TikTok, ティックトック, tik tok, tick tock, tic tok, tic toc, tictok, тик ток, ticktock', 'description': "charli d’amelio (@charlidamelio) has created a short video on TikTok with music It Won't Stop (feat. Chris Brown) - DWYR Radio Edit. | momma @heididamelio dc @trvpandre  @dejuane.mccoy @justinplaness", 'canonicalHref': 'https://www.tiktok.com/@charlidamelio/video/6854565358808993029', 'applicableDevice': 'pc, mobile'}, 'videoObjectPageProps': {}, 'feedConfig': {'id': '6854565358808993029', 'pageType': 0, 'secUid': '', 'emptyTip': 'No content'}, 'pageOptions': {'footer': {'hidden': True}, 'header': {'showUpload': True, 'type': 'webapp'}}}

@blackjack4494
Copy link
Owner

Maybe some header related issues again?

@someziggyman
Copy link

maybe, this would be helpful. However, I'm not sure how to test if it works. https://github.com/onemanbuilds/TikTokVideoDownloader

@merval
Copy link
Author

merval commented Oct 25, 2020

maybe, this would be helpful. However, I'm not sure how to test if it works. https://github.com/onemanbuilds/TikTokVideoDownloader

Thanks for the link!
I'm mobile currently, but I'll look into this when I'm at a computer.

@merval
Copy link
Author

merval commented Oct 25, 2020

maybe, this would be helpful. However, I'm not sure how to test if it works. https://github.com/onemanbuilds/TikTokVideoDownloader

Thanks for the link!
I'm mobile currently, but I'll look into this when I'm at a computer.

That is actually using the same JSON that yt-dlc is, same KeyError if you run it unedited.

@someziggyman
Copy link

maybe, this would be helpful. However, I'm not sure how to test if it works. https://github.com/onemanbuilds/TikTokVideoDownloader

Thanks for the link!
I'm mobile currently, but I'll look into this when I'm at a computer.

That is actually using the same JSON that yt-dlc is, same KeyError if you run it unedited.

hm.. maybe passing cookies file (or cookie header) from each session would help. I've tried adding various user-agent headers, but that did not help.

@someziggyman
Copy link

someziggyman commented Oct 25, 2020 •
edited
Loading

maybe, this would be helpful. However, I'm not sure how to test if it works. https://github.com/onemanbuilds/TikTokVideoDownloader

Thanks for the link!
I'm mobile currently, but I'll look into this when I'm at a computer.

That is actually using the same JSON that yt-dlc is, same KeyError if you run it unedited.

hm.. maybe passing cookies file (or cookie header) from each session would help. I've tried adding various user-agent headers, but that did not help.

So, seems that I've got it working!

If cookies are passed it works in 100% of cases. I did the following - opened TikTok link in Chrome, then created cookies.txt file with a help of this extension:
https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid/related?hl=en

Then just passed it to ytdlc with --cookies. Guess, this should be tested more and I have no idea how to implement it inside TikTok.py. Maybe somehow grab cookies on the fly and then pass those along when sending a "download" request.

@merval
Copy link
Author

merval commented Oct 25, 2020

maybe, this would be helpful. However, I'm not sure how to test if it works. https://github.com/onemanbuilds/TikTokVideoDownloader

Thanks for the link!
I'm mobile currently, but I'll look into this when I'm at a computer.

That is actually using the same JSON that yt-dlc is, same KeyError if you run it unedited.

hm.. maybe passing cookies file (or cookie header) from each session would help. I've tried adding various user-agent headers, but that did not help.

So, seems that I've got it working!

If cookies are passed it works in 100% of cases. I did the following - opened TikTok link in Chrome, then created cookies.txt file with a help of this extension:
https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid/related?hl=en

Then just passed it to ytdlc with --cookies. Guess, this should be tested more and I have no idea how to implement it inside TikTok.py. Maybe somehow grab cookies on the fly and then pass those along when sending a "download" request.

Confirmed, using cookies it does work 100% of the time. I didn't even need to sign-in.
Having it grab cookies, is actually not a half bad idea. I'll see what cookie it's creating and see if I can find a way to implement it.

@merval
Copy link
Author

merval commented Oct 25, 2020

I'm not overly familiar with all of the various extractors, but it doesn't look like any of them auto-grab cookies. Best next approach would be to require cookies, and raise an ExtractorError if cookies aren't supplied.
@blackjack4494 What do you think?

@someziggyman
Copy link

someziggyman commented Oct 25, 2020 •
edited
Loading

maybe, this would be helpful. However, I'm not sure how to test if it works. https://github.com/onemanbuilds/TikTokVideoDownloader

Thanks for the link!
I'm mobile currently, but I'll look into this when I'm at a computer.

That is actually using the same JSON that yt-dlc is, same KeyError if you run it unedited.

hm.. maybe passing cookies file (or cookie header) from each session would help. I've tried adding various user-agent headers, but that did not help.

So, seems that I've got it working!
If cookies are passed it works in 100% of cases. I did the following - opened TikTok link in Chrome, then created cookies.txt file with a help of this extension:
https://chrome.google.com/webstore/detail/get-cookiestxt/bgaddhkoddajcdgocldbbfleckgcbcid/related?hl=en
Then just passed it to ytdlc with --cookies. Guess, this should be tested more and I have no idea how to implement it inside TikTok.py. Maybe somehow grab cookies on the fly and then pass those along when sending a "download" request.

Confirmed, using cookies it does work 100% of the time. I didn't even need to sign-in.
Having it grab cookies, is actually not a half bad idea. I'll see what cookie it's creating and see if I can find a way to implement it.

Also, inspected headers for request URL and found cookie header. So, it also works like this:

ytdlc --add-header Cookie:"tt_webid=6887613038293124613; tt_webid_v2=6887613038293124613; MONITOR_WEB_ID=7f640e08-87be-42e5-a556-872feb013f61; s_v_web_id=verify_kgpn3pr8_qpIX4y6D_QLLu_4Q50_AVAf_3l4PZJaSctoF" https://www.tiktok.com/@charlidamelio/video/6854565358808993029

That's probably simpler than generating cookies file and more "python friendly". And you are right, no sign in needed. Fingers crossed :-)

@merval
Copy link
Author

merval commented Oct 25, 2020

Actually, after rooting around a little, I did find a process used by dailymotion to capture cookies. Let me implement it and test.

@merval
Copy link
Author

merval commented Oct 25, 2020

Actually, after rooting around a little, I did find a process used by dailymotion to capture cookies. Let me implement it and test.

Hmm. So it's pulling the right cookies automatically, but it won't work unless I specify a cookies file, then it writes the cookies to the file and the next time I run it, the download works right. As far as I can tell, the cookies are in the cookiejar correctly. Not sure why it's not working. I'm still investigating other extractors to see if I'm missing something.

@blackjack4494 blackjack4494 merged commit 1666c27 into blackjack4494:master Oct 26, 2020
@blackjack4494
Copy link
Owner

There are sites like this being able to download no watermark highest resolution videos. But no one actually got any source code public on how to generate requests for that. Most want money for some private api. There is a way to generate requests but you need some specific values x-khronos and x-gorgon. But it seems that it's not that simple to reproduce and requires an android or ios device with a server software running to communicate.

@merval merval deleted the merval/fix_tiktok branch October 26, 2020 01:54
@blackjack4494 blackjack4494 mentioned this pull request Oct 26, 2020
Merged
@tanliang2
Copy link

when do it works?

@pukkandan
Copy link
Contributor

It works now with cookoes

siikamiika pushed a commit to siikamiika/yt-dlc that referenced this pull request Feb 15, 2021
@pukkandan
Fix typos (Closes blackjack4494#14)
f5546c0 
:skip ci all

Co-authored by: FelixFrog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mmraz mmraz mmraz approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Broken]Tiktok seems to be broken
6 participants
@merval @blackjack4494 @someziggyman @tanliang2 @pukkandan @mmraz