Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[youtube] Age-gate bypass does not work because of get_video_info returning 404 #29086

Closed
6 tasks done
Lesmiscore opened this issue May 20, 2021 · 48 comments · Fixed by whatdaybob/sonarr_youtubedl#20 or #31043
Closed
6 tasks done

Comments

@Lesmiscore
Copy link
Contributor

Checklist

  • I'm reporting a bug report
  • I've verified that I'm running youtube-dl version 2021.05.16
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the bugtracker for similar bug reports including closed ones
  • I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-sv', 'Tq92D6wQ1mg']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.05.16
[debug]        from commit 9aad8cd18
[debug]           based on a72600998
[debug] Git HEAD: 8248d5fba
[debug] Python version 3.9.5 (CPython) - Linux-5.11.0-051100-generic-x86_64-with-glibc2.31
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] Tq92D6wQ1mg: Downloading webpage
[youtube] [debug] Fetching webpage from https://www.youtube.com/watch?v=Tq92D6wQ1mg&bpctr=9999999999&has_verified=1
[youtube] trying regex ;ytplayer\.config\s*=\s*({.+?});ytplayer
[youtube] trying regex ;ytplayer\.config\s*=\s*({.+?});
[youtube] trying regex (?m)window\["ytInitialPlayerResponse"\]\s*=\s*({.+});$
[youtube] trying regex ytInitialPlayerResponse\s*=\s*({.+?})\s*;\s*(?:var\s+meta|</script|if\s*\(window\.ytcsi\)|\n)
[youtube] ytInitialPlayerResponse\s*=\s*({.+?})\s*;\s*(?:var\s+meta|</script|if\s*\(window\.ytcsi\)|\n) found
[youtube] Tq92D6wQ1mg: Refetching age-gated info webpage
[youtube] [debug] Fetching webpage from https://www.youtube.com/get_video_info?video_id=Tq92D6wQ1mg&eurl=https%3A%2F%2Fyoutube.googleapis.com%2Fv%2FTq92D6wQ1mg
WARNING: unable to download video info webpage: HTTP Error 404: Not Found
ERROR: Sign in to confirm your age
This video may be inappropriate for some users.
Traceback (most recent call last):
  File "/home/lesmi/youtube-dl/youtube_dl/YoutubeDL.py", line 876, in wrapper
    return func(self, *args, **kwargs)
  File "/home/lesmi/youtube-dl/youtube_dl/YoutubeDL.py", line 907, in __extract_info
    ie_result = ie.extract(url)
  File "/home/lesmi/youtube-dl/youtube_dl/extractor/common.py", line 553, in extract
    ie_result = self._real_extract(url)
  File "/home/lesmi/youtube-dl/youtube_dl/extractor/youtube.py", line 1743, in _real_extract
    raise ExtractorError(reason, expected=True)
youtube_dl.utils.ExtractorError: Sign in to confirm your age
This video may be inappropriate for some users.

Description

Given an age-restricted video like this one, youtube-dl used to workaround it by using get_video_info call. But it seems it's not available anymore.

This is not lethal because when I provided --cookies option then it works normally.

@coletdjnz
Copy link
Contributor

coletdjnz commented May 20, 2021

Noticing this too. This seems like similar behavior of what happened when the old API endpoint used to get continuation pages was being turned off. So I think the get_video_info API endpoint might be seeing the same fate too.

This also affects PR #28778.

@StanNieuwmans
Copy link

@colethedj

Sorry for the double post but this is the first issue a came by when i googled so:

I dont know if this helps but if you want to let get_video_info link to work for now you can add : &html5=1 to the link.
You will get something like this: https://www.youtube.com/get_video_info?html5=1&video_id={videoID}.

@oneandonlyjason
Copy link

I guess that this error probably has the same Reason:

root@vmd57733:~# youtube-dl https://www.youtube.com/watch?v=v4xrvwHa-bU --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://www.youtube.com/watch?v=v4xrvwHa-bU', u'--verbose']
WARNING: Assuming --restrict-filenames since file system encoding cannot encode all characters. Set the LC_ALL environment variable to fix this.
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out ANSI_X3.4-1968, pref ANSI_X3.4-1968
[debug] youtube-dl version 2021.05.16
[debug] Python version 2.7.16 (CPython) - Linux-5.4.0-0.bpo.2-amd64-x86_64-with-debian-10.9
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1
[debug] Proxy map: {}
[youtube] v4xrvwHa-bU: Downloading webpage
[youtube] v4xrvwHa-bU: Refetching age-gated info webpage
WARNING: unable to download video info webpage: HTTP Error 404: Not Found
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/bin/youtube-dl/main.py", line 19, in
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 475, in main
File "/usr/local/bin/youtube-dl/youtube_dl/init.py", line 465, in _real_main
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2069, in download
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
File "/usr/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1502, in _real_extract
File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2644, in compat_parse_qs
File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2614, in _parse_qsl
AttributeError: 'bool' object has no attribute 'split'

@3unjee
Copy link

3unjee commented May 20, 2021

Yes, it seems youtube.com/get_video_info now always returns 404 from Youtube.

I was relying exclusively on it (with MotionBox) and I'm now transitionning to a youtube.com/watch based implementation.

@StanNieuwmans
Copy link

Yes, it seems youtube.com/get_video_info now always returns 404 from Youtube.

I was relying exclusively on it (with MotionBox) and I'm now transitionning to a youtube.com/watch based implementation.

Also with &html5=1 in the link?

pukkandan pushed a commit to yt-dlp/yt-dlp that referenced this issue May 20, 2021
@3unjee
Copy link

3unjee commented May 20, 2021

Yes, it seems youtube.com/get_video_info now always returns 404 from Youtube.
I was relying exclusively on it (with MotionBox) and I'm now transitionning to a youtube.com/watch based implementation.

Also with &html5=1 in the link?

Oh you're right it works with the following https://www.youtube.com/get_video_info?html5=1&video_id={videoID} !

Still wanted to ditch get_video_info because I couldn't find higher resolutions int it (2k 4k).

@Lesmiscore Lesmiscore changed the title [youtube] get_video_info returns 404 [youtube] Age-gate bypass does not work because of get_video_info returning 404 May 20, 2021
@MikeTester21
Copy link

MikeTester21 commented May 20, 2021

Yes, it seems youtube.com/get_video_info now always returns 404 from Youtube.

I was relying exclusively on it (with MotionBox) and I'm now transitionning to a youtube.com/watch based implementation.

I am new to this, what is youtube.com/watch` based implementation? Does it work with age-restricted videos? Can you link to more info?

adding html5=1& solved the get_video_info issue for now/

@3unjee
Copy link

3unjee commented May 20, 2021

Yes, it seems youtube.com/get_video_info now always returns 404 from Youtube.
I was relying exclusively on it (with MotionBox) and I'm now transitionning to a youtube.com/watch based implementation.

I am new to this, what is youtube.com/watch` based implementation? Does it work with age-restricted videos? Can you link to more info?

adding html5=1& solved the get_video_info issue for now/

Yep adding html5 solved. What I was talking about was off topic since it's related to another program I'm working on.

@MikeTester21
Copy link

3unjee, no problem if it is off topic, , I'd like to know if there is any other way to get info from age-restricted videos , like this youtube.com/watch method? Is there an youtube.com/watch URL that takes video ID and returns data, like get_video_info method?

@3unjee
Copy link

3unjee commented May 20, 2021

On my side, for age restricted I'm using https://www.youtube.com/get_video_info?html5=1&video_id={videoID}&el=embedded. The embedded flag seems to matter.

@MikeTester21
Copy link

3unjee , for me it works even without embedded flag, but html5=1 is a must.

The reason I am asking about alternatives (like the youtube.com/watch you mentioned) is because I think it is just a matter of time google kills html5=1 workaround too.

So how will you transition to youtube.com/watch or what will you use if get_video_info with html5=1 workaround gets killed?

@rubyFeedback
Copy link

I also recently had problems with the age-ghetto wall. It was precisely that error:

"WARNING: unable to download video info webpage: HTTP Error 404: Not Found"

Google really wants to be annoying...

@rubyFeedback
Copy link

For me it did not work. :(

E. g. an URL such as

https://www.youtube.com/get_video_info?html5=1&video_id=pawkdEOyvIk&el=embedded

And I got:

[youtube:tab] get_video_info: Downloading webpage

ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Hmm. If a solution was found, perhaps the commandline could display
that as well? Ideally with a copy/pasteable example? I tried several
variants named above but nothing worked for me. Can anyone else
conform a variant and show the command that was used too? I seem
to no longer be able to bypass the google ghetto wall (and that video
from ozzy man is so harmless - why do they censor this crap; I removed
my youtube account not long ago due to google being soooooo annoying.
I'd wish we could have a decentralized youtube ... ).

@coletdjnz
Copy link
Contributor

3unjee, no problem if it is off topic, , I'd like to know if there is any other way to get info from age-restricted videos , like this youtube.com/watch method? Is there an youtube.com/watch URL that takes video ID and returns data, like get_video_info method?

There is the player endpoint of the newer innertube API (www.youtube.com/youtubei/v1/player), which returns the player data that would be normally contained in ytinitialdata from a /watch url request. Youtube seems to be transitioning to this youtubei endpoint for its API calls.

As for the age-gate bypass, this endpoint will return an error if the video is age-restricted if you do not supply auth. So I'm not sure if there is anyway to workaround that.

@coletdjnz
Copy link
Contributor

For me it did not work. :(

E. g. an URL such as

https://www.youtube.com/get_video_info?html5=1&video_id=pawkdEOyvIk&el=embedded

And I got:

[youtube:tab] get_video_info: Downloading webpage

ERROR: Unable to extract yt initial data; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Hmm. If a solution was found, perhaps the commandline could display
that as well? Ideally with a copy/pasteable example? I tried several
variants named above but nothing worked for me. Can anyone else
conform a variant and show the command that was used too? I seem
to no longer be able to bypass the google ghetto wall (and that video
from ozzy man is so harmless - why do they censor this crap; I removed
my youtube account not long ago due to google being soooooo annoying.
I'd wish we could have a decentralized youtube ... ).

This request is done internally in the YouTube extractor, you can't pass it directly to youtube-dl.

@MikeTester21
Copy link

colethedj , by supply auth do you mean gmail/youtube credentials or youtubr API3 credentials? Because google screwed many with their api credentials too, reducing the number of request to very low or even 0 for so many developers. So which auih credentials do you need/are you talking about? Considering that you need only google account to see age-restricted videos is it this what is needed?

@coletdjnz
Copy link
Contributor

coletdjnz commented May 21, 2021

colethedj , by supply auth do you mean gmail/youtube credentials or youtubr API3 credentials? Because google screwed many with their api credentials too, reducing the number of request to very low or even 0 for so many developers. So which auih credentials do you need/are you talking about? Considering that you need only google account to see age-restricted videos is it this what is needed?

This is the internal API Youtube uses ("Innertube") - it is separate from the official dev api.Youtube-dl uses this API is used for playlist continuation pages, if you want to have a look. The auth for it isn't actually implemented in youtube-dl yet (hence #28362). I described the auth required in that issue, but essentially it's your cookies.txt exported from Youtube plus a few headers (SAPISIDHASH etc.)

You can already access age-restricted videos if you pass cookies to youtube-dl - as that is just a normal request to the watch html page (API requests is done internally YouTube's side in this case).

@MikeTester21
Copy link

colethedj, I am trying the -cookies way and it does not work. Using a video example from first post (I just want to get video URL, not download it), this is what I use:

youtube-dl -g --cookies cookies.txt https://www.youtube.com/watch?v=Tq92D6wQ1mg

WARNING: unable to download video info webpage: HTTP Error 404: Not Found

I updated to latest version 2021.05.16

Apologies for question, I am new to youtube-dl, but is youtube-dl utilizing get_video_info method for downloading/getting youtube video info? Because right now I am using a different plugin that calls get_video_info, but if youtube-dl uses the same thing for age-restricted vidos then if get_video_info dies then youtube-dl will sadly also not help.

I just received updated YT terms and it says : "YouTube’s right to monetize: YouTube has the right to monetize all content on the platform and ads may appear on videos from channels not in the YouTube Partner Program" . I am really disliking google, they keep closing door after door for developers. and they ae rally getting greedier with every move.

@coletdjnz
Copy link
Contributor

coletdjnz commented May 21, 2021 via email

user234683 added a commit to user234683/youtube-local that referenced this issue May 27, 2021
get_video_info now returns 404 error. Adding html5=1 fixes it
(for now). See
ytdl-org/youtube-dl#29086 (comment)

Also handles 404 error if it arises so it will be non-fatal
@dstftw dstftw closed this as completed in 24297a4 May 29, 2021
@StanNieuwmans
Copy link

Yes, it seems youtube.com/get_video_info now always returns 404 from Youtube.
I was relying exclusively on it (with MotionBox) and I'm now transitionning to a youtube.com/watch based implementation.

Also with &html5=1 in the link?

i feel like the get_video_info with html5 is broken ;(

@pukkandan
Copy link
Contributor

i feel like the get_video_info with html5 is broken ;(

@StanNieuwmans What makes you say that?

Tested with Tq92D6wQ1mg v4xrvwHa-bU pawkdEOyvIk and they are all still working

@StanNieuwmans
Copy link

i feel like the get_video_info with html5 is broken ;(

@StanNieuwmans What makes you say that?

Tested with Tq92D6wQ1mg v4xrvwHa-bU pawkdEOyvIk and they are all still working

Did you use this tool? because if you did this tool already did an transition to the innertube api, if i am correct. Then there are no issue's.

To the people who do use the html5 "hack", I think it doesn't work anymore. Also I think google is shutting down the www.youtube.com/get_video_info?video_id={videoID} link.

@Lesmiscore
Copy link
Contributor Author

Lesmiscore commented Jun 18, 2021

I rewrote the code to use innertube instead, and age-gate bypassing worked for embeddedable videos. It'll fail if the video can't be embedded, since it's mimicking embedded client.
ytdl-patched/ytdl-patched@7371d30

@rjdg14
Copy link

rjdg14 commented Jun 19, 2021

I rewrote the code to use innertube instead, and age-gate bypassing worked for embeddedable videos. It'll fail if the video can't be embedded, since it's mimicking embedded client.
ytdl-patched/ytdl-patched@7371d30

Age restricted videos haven't been able to be embedded since about last September, as they'll display an error saying that they're age restricted and only viewable on YouTube's website. It's possible that this is different to the "playback on other websites has been disabled by the video owner" error, though.

@pukkandan
Copy link
Contributor

@rjdg14 Not all age-gated videos are non-embeddable. See yt-dlp/yt-dlp#328 (comment)

Also, see #29333 (comment) for another workaround

coletdjnz added a commit to yt-dlp/yt-dlp that referenced this issue Jun 29, 2021
@IoanPaleru
Copy link

Youtube-dl doesn't download age restricted youtube videos anymore like this one "https://www.youtube.com/watch?v=xcRUPMf6scs" (NSFW). Is there a way to get it to download those videos without signing in? Thanks.

@liamengland1
Copy link

works in yt-dlp. @IoanPaleru

@IoanPaleru
Copy link

Thanks, but how do I get it to work in youtube-dl though? Isn't this the page for that? Also, I don't know how or where to download a pre-compiled version of yt-dlp from that works in windows 7 & 10, without having to install anything extra to get it to work. If there isn't a pre-compiled version, I don't know how to compile programs for windows from source code. Are the command options in yt-dlp be the same as youtube-dl? I'm just an average user how knows how to use some simple command line commands with the help of a help guide. Is there some kind of program that will just download those kinds of youtube videos properly without having to configure anything other than just doing regular updates? Youtube-dl used to work properly but it doesn't anymore, so why doesn't it just work properly after updating it? Why does it keep having problems downloading youtube videos properly?

@auouymous
Copy link

@IoanPaleru The problem is that youtube keeps changing its undocumented API. #29333 (comment) fixes it, but you'll need to wait for it to be merged and released.

@coletdjnz
Copy link
Contributor

coletdjnz commented Jul 11, 2021

It seems like the (tier 2) age-gate workaround has been fixed here (03ab027), however there hasn't been a release in over a month, so the latest doesn't include it...

@coletdjnz
Copy link
Contributor

@colethedj that's using the old get_video_info, I am talking about the new API. I have my own implementation, but until now it only worked the embeddable age gate videos. I also learned with Firefox, the cookies are stored in a simple SQLite file, so you should be able to pull from that.

Yes, if you use cookies and you can play the video with such account then you should be able to access the video regardless.
Unless you're talking about something different?

@coletdjnz
Copy link
Contributor

coletdjnz commented Jul 11, 2021

I'm talking about downloading age gate videos using the new API, what are you talking about?

😕 Same thing? Yes, this applies to API requests too (the html watch page basically does the API requests on Youtube's side and mashes all the player&next response data into ytinitialdata if I'm not mistaken). Though as you found there are some extra headers required when using cookies with the API.

@digitalblossom
Copy link

It appears to be fixed in youtube dlp, i just tried it and while standart youtube dl didn't work, the info was fetched correctly and the video downloadble using youtube dlp

@jherazob
Copy link

It appears to be fixed in youtube dlp, i just tried it and while standart youtube dl didn't work, the info was fetched correctly and the video downloadble using youtube dlp

This seems to be their issue where they patched it, looks like it has two different methods to get the data, if one doesn't work they try the other. Of course, when they close those endpoints there will be no way unless a workaround with the new APIs is found.

@coletdjnz
Copy link
Contributor

In summary:

The current known age-gate bypasses (get_video_info & embedded API clients) don't work on all videos. We consider these videos that don't work "Tier 3" - some more information here: yt-dlp/yt-dlp#328 (comment)

This is assuming you're not in the EU, which if you are the above probably does not apply and you require cookies from an account that is age-verified regardless (correct me if I'm wrong - not in the EU myself!).

Both youtube-dl and yt-dlp have working support for tier 2 bypass. However, the tier 2 bypass in youtube-dl current release is broken. It is fixed in master, but for whatever reason there hasn't been a release with it :/


Also good to note get_video_info appears to only be disabled on web clients at this stage. Other non-web clients appear to work with it (can confirm myself that TVHTML5 & ANDROID work with it atm)

@MerkaST
Copy link

MerkaST commented Jul 15, 2021

@colethedj I'm in the EU, I can access tier 1 & 2 videos with yt-dlp just fine. Tier 3 doesn't work here either. I however can indeed not access any of those videos on the YouTube website itself without verifying my age.

@coletdjnz
Copy link
Contributor

Tier 3 workaround:

https://www.youtube.com/get_video_info?
c=TVHTML5&
cver=7.20210428.10.00&
el=detailpage&
html5=1&
video_id=Cr381pDsSsA&
access_token=ya29.a0ARrdaM_CTfzZC0eE_Yxk04oowauAra_Z8Zh6mO5UzN84XYNot32JyYqcCG...

I'd create a new issue regarding these findings, since it's not particularly relevant to the age-gate.

for anyone reading: it's an alternative method to providing cookies, not an age-gate workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet