Add ability to set po_token and visitor data #4789
Conversation
unixfox
requested review from
TheFrenchGhosty,
SamantazFox and
a team
as code owners
|
Tried `Title: Missing hash key: "videoDetails" (KeyError) Backtrace
Missing hash key: "videoDetails" (KeyError) |
|
You'll also need to setup #4772 |
|
My instance "got hit by" sign in requirement yesterday. I've used unixfox:potoken-config, merged #4772 into it and built inv_sig_helper. inv_sig_helper starts successfully, invidious connects to it and successfully starts. (I've replaced visitorData & poToken with XXX, but they matched the generated ones.) Maybe I'm missing something, but I just thought I will let you know. |
|
@MMaster thanks for the feedback. I personally think that something changed since this PR was created with how youtube treat the poToken validity on a youtube embed page. For the moment on yewtu.be, I'm using a poToken generated from www.google.com search and it works fine. I'll try to adapt youtube-trusted-session-generator for this other kind of potoken generation. Will get back to you once it's done! |
Strangely I just did exactly what the trusted session generator does manually in private window and copied visitorData and poToken from the play button POST request and it looks like this time it got further, but based on the logs it still looks like it requires login which in turn errors out on I haven't changed anything except for po_token and visitor_data in config - its still the same docker image I've built before. EDIT: Just noticed that the android request is missing "signatureTimestamp", I'm not sure if that's expected. |
|
Last message, I promise :) EDIT: I figured out that youtube-trusted-session-generator returns poToken and visitorData even if youtube expects you to login there. Those are shorter than the proper ones. (visitorData 48 vs 80 chars, poToken 164 vs 204 chars) PS: Thank you so much for maintaining this project. Much appreciated! I hope my findings will help at least a little bit :) |
|
hello @MMaster, please try again the script https://github.com/iv-org/youtube-trusted-session-generator with the latest commit Without your latest patch in order to confirm that it works great with the PR #4772 thank you.
That's not a good idea if it only works with TvHtml5ScreenEmbed because it means that ONLY videos that can be embedded will work. My new commit in the script should fix this issue. |
|
@unixfox Thank you for getting back to me! I've tested it, but unfortunately I'm still getting the "This helps protect our community." with visitor_data and po_token generated using that script in docker. It doesn't matter if I run it from the instance or other machine where the generated tokens are longer. Unfortunately I currently don't have machine where I would be willing to install all the dependencies of chromium without docker, but I can get one if you think it would be worth it. Now even if I open the original embed video url using private browser from instance IP it doesn't show the sign in dialog and I can play it directly (previously I couldn't play the embed video without signing in). The tokens I get manually from play button submit POST request work as long as I skip Android. I understand this is not good - just saying. |
|
@MMaster Please try yet again, this time it should work! But in Docker for headfull, this requires a graphical environment (X11) so now the image is a bit bigger but at least it works. |
|
@unixfox Bingo. This time the tokens generated from the script work just as the ones I got manually. |
the po_token generated only works with the WEB client. It will never work for ANDROID client since this token has been generated for a webpage, not on the youtube app for android. Hence you need #4772 too. |
I got #4772 merged and invidious is using it and adding the signatureTimestamp to WEB requests. But when you look into parser.cr extract_video_info:
As is visible in traces here: #4789 (comment) |
|
I have pushed the merged code (unixfox:potoken-config with merged #4772) with quick and dirty Dockerfile to compile and then run inv_sig_helper in background in docker here: https://github.com/MMaster/mm-invidious/tree/potoken-config-with-sig (it doesn't contain the latest wording changes in config, but those don't matter for the functionality anyway) I've added inv_sig_helper directly to it (I said it's quick and dirty :) ) I hope people won't mind - I will remove the branch when it won't be needed anymore. https://github.com/MMaster/mm-invidious/tree/potoken-config-with-sig |
| @@ -55,7 +55,7 @@ def extract_video_info(video_id : String) | |||
| client_config = YoutubeAPI::ClientConfig.new | |||
|
|
|||
| # Fetch data from the player endpoint | |||
| player_response = YoutubeAPI.player(video_id: video_id, params: "", client_config: client_config) | |||
| player_response = YoutubeAPI.player(video_id: video_id, params: "2AMB", client_config: client_config) | |||
There was a problem hiding this comment.
Not sure if I should put 2AMB in a static variable somewhere for using it in try_fetch_streaming_data too but I don't know where.
|
@MMaster I'm so so sorry, to be transparent with you, I didn't try #4772 yet, and I thought that in this PR the client would be switched from Android to WEB. But I was wrong. In fact it makes perfect sense because some IP addresses are not yet blocked but sig decipher is becoming a must for almost all youtube clients. So in this PR, #4789, I just pushed a new commit. It's quite simple. If
I wouldn't recommend doing that because we specifically created a separate process for isolating the invidious process from possible remote code executions from Google itself! So the final plan is to offer inv_sig_helper as a separate docker container for an ideal isolation. Thank you for the feedback, though! Without you, I think we wouldn't have found this issue! @SamantazFox @syeopite please re-review the PR. |
|
@unixfox No worries. I already thought it's because the token doesn't work on Android yesterday, but I just got steered by your comment that Android is required and thought that you got it working so I suspected it's the signature. I've tested the changes and it works properly now. I've also merged the changes to that temporary branch in my repo if anyone else wants to try it in docker. I'm happy I could help!
Yeah, I know. I will definitely do that when the inv_sig_helper communication gets finalized. But inv_sig_helper still created just unix domain socket and invidious is automatically using it. So running it in the same container was the easiest and fastest way to get it working for testing since sharing UDS between containers can be tricky and I wasn't going for permanent solution. EDIT: I also plan to automate the potoken generation to get and configure new token with each invidious restart. |
Please consult iv-org/youtube-trusted-session-generator#1 if you want to help us. |
At the very least, you can run inv_sig_helper in a seperate user, with really strict rules set on it (in fact, this is how i designed it originally without thinking about docker). |
|
Seems like videoplayback requests will still fail (403) if invidious does not provide the po_token as pot parameter: LuanRT/YouTube.js#708 (comment) Can anyone replicate the issue? |
From my testing over last 3 weeks:
I can try to test if adding |
After quick hack to add I tested right before adding it and I was getting 403s to all DASH videoplayback requests, as soon as I added I checked with regular youtube and that one is also adding |
Fixes api requests iv-org/invidious#4789
|
I have just added |
| @@ -110,7 +110,7 @@ struct Video | |||
| fmt["url"] = JSON::Any.new("#{fmt["url"]}#{DECRYPT_FUNCTION.decrypt_signature(fmt)}") | |||
| end | |||
|
|
|||
| fmt["url"] = JSON::Any.new("#{fmt["url"]}&host=#{URI.parse(fmt["url"].as_s).host}") | |||
| fmt["url"] = JSON::Any.new("#{fmt["url"]}&host=#{URI.parse(fmt["url"].as_s).host}&pot=#{CONFIG.po_token}") | |||
There was a problem hiding this comment.
This should be behind a if CONFIG.po_token conditional, but I'll fix that when merging the two PRs.
There was a problem hiding this comment.
Yeah but an empty value for a query string won't really hurt.
SamantazFox
added
ready
and removed
enhancement
SamantazFox
changed the title
Support poToken/visitorData challenge, based on:
https://github.com/yt-dlp/yt-dlp/wiki/Extractors#po-token-guide
yt-dlp/yt-dlp#10648
iv-org/invidious#4789
By following the steps in the yt-dlp wiki page (linked above), I was
able to get a working (poToken, visitorData) pair in Chromium. For some
reason, I could not get a working pair in Firefox Nightly; maybe my
profile in the latter is fingerprinted and soft-blocked somehow?
Support poToken/visitorData challenge, based on:
https://github.com/yt-dlp/yt-dlp/wiki/Extractors#po-token-guide
yt-dlp/yt-dlp#10648
iv-org/invidious#4789
By following the steps in the yt-dlp wiki page (linked above), I was
able to get a working (poToken, visitorData) pair in Chromium. For some
reason, I could not get a working pair in Firefox Nightly; maybe my
profile in the latter is fingerprinted and soft-blocked somehow?
Helps with #4734
This adds the ability to set po_token and visitordata ID statically in the config.yaml
Also, if
po_tokenparameter is passed, then WEB client is used, if not Android client is used.TvHtml5ScreenEmbedwill still be used as a fallback.Script for generating po_token and visitor_data: https://github.com/iv-org/youtube-trusted-session-generator
There are plans to automate the tokens creation: iv-org/youtube-trusted-session-generator#1 but not yet implemented. At least this unlocks people for keep using invidious.