fix tiktok download

[GreyAlien502]

Currently, tiktok downloads fail with status 403.
This request fixes that by downloading the page twice before trying the video.
This seems like a bug on their end since i noticed the same problem in the webapp:
the request for the video file fails with 403 when opening a video in a fresh incognito session, but it works when i reload.

[merval]

I like the approach, but I don't think you need to assign the webpage variable twice. simply calling self._download_webpage does the trick

[blackjack4494]

Unfortunately there is no retry option for downloading webpages or info files like in json.
However the mentioned approach here is poorly implemented.
I don't have any problems even trying multiple times. You could at most implement a solution that checks the http status and retries again after a certain timeout.
Currently you force even users without problems to download the page twice.
Also have you tried something like --force-ipv4 or --force-generic?

[merval]

@GreyAlien502 We should look at implementing what @blackjack4494 is referring to. Check the page status code, and call a second time if we get 503 back.

[GreyAlien502]

@blackjack4494, We aren't retrying anything here,
especially not the webpage/info file download, which never fails,
The second call is preemptive, to avoid getting a 403 later, when we get the video.

I don't have any problem

Oh, i wasn't aware that the current code was working for anyone (without providing cookies).
It always fails for me.

have you tried something like --force-ipv4 or --force-generic

I just tried --force-generic, and it sometimes works for me.
Since it makes multiple requests for each resource it downloads,
it's not too surprising that it would.

[GreyAlien502]

@merval, an InfoExtractor will not know that there's a problem
since the error doesn't occur during the information extraction, right?

[blackjack4494]

Please change this so that it won't download the webpage twice if not necessary. You can check the result and implement some try again but only once if response isn't satisfying.

[GreyAlien502]

If anyone can provide a way of determining if the response is satisfying, i can add such a check.
I don't see any meaningful difference between the first and second responses i receive.
Is there a specific reason why you think such a check would be possible?

[shiwei87tan]

tiktok download is down again.

U:\youtube-dlc -i -o "%(id)s.mp4" -v https://www.tiktok.com/@tiktok_stage/video/6885152542095887617
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-i', '-o', '%(id)s.mp4', '-v', 'https://www.tiktok.com/@tiktok_stage/video/6885152542095887617']
[debug] Loading archive file None
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dlc version 2020.11.07
[debug] Python version 3.9.0 (CPython) - Windows-10-10.0.19041-SP0
[debug] exe versions: ffmpeg git-2020-08-07-fab00b0, ffprobe git-2020-08-07-fab00b0
[debug] Proxy map: {}
[TikTok] 6885152542095887617: Downloading video webpage
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://v16-web.tiktok.com/video/tos/alisg/tos-alisg-pve-0037c001/9b3784d6dec84c75b0233ca0d90572c5/?a=1988&br=1272&bt=636&cr=0&cs=0&cv=1&dr=3&ds=3&er=&expire=1604852024&l=202011081013320101151511041701C791&lr=tiktok&mime_type=video_mp4&policy=2&qs=0&rc=M2x3OWhzaTtpeDMzaTgzM0ApZjhoZzczPGU7NzdnNGc5O2dtLmhoaGtxZzNfLS02LzRzc2JeNGM2MzUxLjAzXzVjMDM6Yw%3D%3D&signature=7773bd3019bd845e9f8ee935e9ef478b&tk=tt_webid_v2&vl=&vr='
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
File "youtube_dlc\YoutubeDL.py", line 2011, in process_info
File "youtube_dlc\YoutubeDL.py", line 1866, in dl
File "youtube_dlc\downloader\common.py", line 373, in download
File "youtube_dlc\downloader\http.py", line 349, in real_download
File "youtube_dlc\downloader\http.py", line 114, in establish_connection
File "youtube_dlc\downloader\http.py", line 110, in establish_connection
File "youtube_dlc\YoutubeDL.py", line 2316, in urlopen
File "urllib\request.py", line 523, in open
File "urllib\request.py", line 632, in http_response
File "urllib\request.py", line 561, in error
File "urllib\request.py", line 494, in _call_chain
File "urllib\request.py", line 641, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

[GreyAlien502]

btw, this was fixed in ytdl-org/youtube-dl by adding the extra call to tiktok in the _real_initialize method.

[tanliang2]

how do you have fixed the bug downloading tiktok resposnse with 403? i have the same probem to download the url
"https://v16-web.tiktok.com/video/tos/alisg/tos-alisg-pve-0037c001/5d1561cb4b5c4e94b68c2d8aefc2526b/?a=1988&br=4898&bt=2449&cd=0%7C0%7C1&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1609504918&l=2021010106413301011515316709B85BC4&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=anU0bHY6bG46eDMzNTczM0ApNTRlaTczNmU8N2lkZmc8PGdpYmkyMm9rXmJfLS0vMTRzcy40MTAwNF82YmFjMy41LzM6Yw%3D%3D&signature=8028848b03a13b99813cb39eb52e5b11&tk=tt_webid_v2&vl=&vr="

[pukkandan]

@tanliang1 Provide the full verbose output (with -v)

Also, you might want to try it on the master of yt-dlp. I have applied a tiny patch on top of this PR