Feature innertube client fix #219

alive4ever · 2024-10-28T02:52:50Z

The major change is the addition of mweb innertube client, which includes some refactoring of how base_js is handled and a new trick to decrypt n signature by extracting the relevant decryption code from base_js file, using technique similar to iv-org/inv_sig_helper.

This pull request also introduce three dependencies: fake-useragent to simplify user agent header creation of mobile and desktop browser, flpc to parse nsig decryption regex, and dukpy to execute the extracted nsig decryption code.

Also add the ability to parse visitorData in the YT Api response and specifying own visitorData and poToken pair using properly formatted json in the data/po_token_cache.txt file.

Also several more fixes for android and ios client and make innertube client selectable.

Also several changes in settings.py, notably to allow reloading of tv_embedded client in case of missing player urls and showing Download placeholder via use_video_download option, which credits ~heckyel/yt-local.

This will hopefully fix #218

alive4ever · 2024-11-08T01:32:25Z

Hi, it's been a while without any feedback.

For n signature solving, dukpy is used here. I tried several python js bindings before coming to this setup.

First, I tried importing JSInterpreter from yt-dlp project. This add yt-dlp as dependency, which is huge (more than 20MB) and the js execution time is slow. The nice thing is that JSInterpreter can execute js functions from base.js without extracting the specified function before.
The second is py-mini-racer. It is slightly faster than JSInterpreter from yt-dlp but the package is rather huge (18 MB of site-package files). It requires the js function to be extracted.
The third is dukpy, which I consider very good with smaller site-package than py-mini-racer (9MB). It also needs the js function to be extracted.
The fourth is combining quickjs with jsengine, which is fastest and smallest (2MB). It can call js function directly from base.js. The downside is there is no pre-built wheel for quickjs module on aarch64 which results in source install for aarch64, so I stick to dukpy for this reason (pre-built wheels and faster aarch64 install.

user234683 · 2024-11-08T04:28:42Z

Great work on this; still need to finish reviewing. Should have something back to you this weekend

alive4ever · 2024-11-08T08:15:21Z

Thanks for the feedback. It gives a peace of mind for me.

No need to rush. I am open for any improvement suggestions to this pull request.

user234683 · 2024-11-10T02:49:19Z

youtube/util.py

+                print('Unable to access ' + player_file)
+
+        signature_timestamp = None
+        signature_timestamp_cache = settings.data_dir + '/sts_' + player_version + 'txt'


Recommend using os.path.join here. Also, .txt, not txt

Sorry for late reply.

I've used os.path.join in place of string concatenation in this file and similar places.

user234683 · 2024-11-10T03:00:27Z

youtube/util.py

+    response_dict = json.loads(response)
+    if settings.use_visitor_data:
+        if not settings.use_po_token:
+            if response_dict['responseContext'].get('visitorData'):


We shouldn't assume 'responseContext' will be present - otherwise it will raise an exception when youtube changes something.

I've put this inside try ... except block, with specific KeyError exception message.

user234683 · 2024-11-10T03:07:53Z

youtube/util.py

+    if settings.use_visitor_data:
+        if not settings.use_po_token:
+            if response_dict['responseContext'].get('visitorData'):
+                if not os.path.exists(visitor_data_file):


Not sure how the visitor system works - but do we want to refresh this file ever? Maybe YouTube issues an updated token for example and marks the old one as invalid?

I added os.path.getmtime check to make sure that the visitorData.txt file is less than 86400 seconds old before using its content. Otherwise, the visitor data file will be deleted and replaced with new one.

user234683 · 2024-11-10T03:08:55Z

youtube/util.py

+        else:
+            if os.path.exists(visitor_data_file):
+                print('Removing visitor_data file')
+                os.remove(visitor_data_file)


For anonymity's sake - do we want to consider refreshing the visitor data every day? Again, not really sure what constraints go into it

Done, as I mentioned above.

user234683 · 2024-11-10T03:10:25Z

server.py

+                    with open(visitor_data_file, "r") as file:
+                        visitor_data = file.read()
+                        file.close()
+                except:


Use except Exception, otherwise you'll catch KeyboardInterrupt and SystemExit: https://stackoverflow.com/questions/54948548/what-is-wrong-with-using-a-bare-except

I added except OSError to notify if there is a file access error which prevents access to the visitor data file.

user234683 · 2024-11-10T21:49:44Z

youtube/yt_data_extract/watch_extraction.py

+
+def extract_nsig_func(base_js):
+    for i, member in enumerate(NSIG_FUNCTION_ARRAYS):
+        func_array_re = regex.compile(member.replace('$', '\\$'))


I recommend against this; I would just put the three \ escapes you need into your regex instead of modifying it at runtime

I only tried doing what inv_sig_helper does, copying exactly the same regex pattern with runtime replacement of escaping dollar sign which only done once for as long as the extracted nsig_func_{player_version}.js file exists.

The resulting n_sig_code is cached as data/nsig_func_{player_version}.js and loaded as info['nsig_func'] = { player_version: js_nsig_decrypt_code } during runtime of the youtube-local` session.

So the n_sig_function extraction is only done once and the subsequent access to it is either loaded directly from the info['nsig_func'] dict or loaded from nsig_func_{player_version}.js file if the file is already exists.

user234683 · 2024-11-10T21:50:27Z

youtube/yt_data_extract/watch_extraction.py

+    func_body_re = []
+    for i, member in enumerate(NSIG_FUNCTION_ENDINGS):
+        func_body_re_item = ''
+        func_body_re_item += func_context.group(1)


You need to do a re.escape() on this before appending it to your regexes

user234683 · 2024-11-10T21:52:02Z

youtube/yt_data_extract/watch_extraction.py

+    print('jscode len is: ' + str(len(jscode)))
+    dukpy_session = dukpy.JSInterpreter()
+    # Loading the function into dukpy session
+    dukpy_session.evaljs(jscode)
+    print('n_sig = ' + n_sig)
+    #n_sig_result = dukpy_session.evaljs('decrypt_nsig("' + n_sig + '")')
+    n_sig_result = dukpy_session.evaljs("decrypt_nsig(dukpy['n'])", n=n_sig)
+    print('n_sig_result = ' + n_sig_result)
+    return n_sig_result


Have we verified that dukpy has limited execution privileges? For instance, can javascript code executed with Dukpy make network requests or open files? If so it would be a massive security hole

Also recommend removing these debugging print statmenets when you're done

dukpy is just a python wrapper for duktape js engine.

The pypi package of dukpy has dukpy-install command which is able to download npm packages from the internet.

Unless told to do so, dukpy module doesn't access the internet for as far as I know. The nsig_func doesn't need access to the internet during runtime, which I have verified doing manual n_sig decryption using various python js bindings.

I also consider dukpy as just-work lightweight js engine for python, since it has wheels for arm64 on pypi and armhf on piwheels.org so if anyone runs this on their single board computers, they will hopefully meet no problems during runtime.

Confirming that this works on arm64.

user234683 · 2024-11-10T21:56:54Z

requirements.txt

@@ -6,3 +6,6 @@ urllib3>=1.24.1
 defusedxml>=0.5.0
 cachetools>=4.0.0
 stem>=1.8.0
+fake-useragent>=1.5.1
+flpc>=0.2.5


Did you do any performance testing that suggested the need for this? Is there a noticeable speedup? Would rather avoid dependencies if possible

With no break statement, the re module will hang for some time. flpc will not hang in finding the specified regex, even without break statement in the for loop.

I added the break statement in the for loop so the regex engine will be freed from work (i.e. testing another regex pattern) after a match is found, which mitigates hanging on the built-in re module.

I've removed flpc from the requirements to use the built-in re module as you wish, with very small or no performance degradation during my extended testing.

user234683 · 2024-11-10T22:01:19Z

youtube/util.py

+    print("Debugging headers")
+    for item in headers:
+        print(item)
+    print("Debugging data payload")
+    print(json.dumps(data, indent=4))


Recommend removing these debugging print statmenets

sapnolas · 2024-11-22T16:29:30Z

Any chance you could make a version of this similar to the youtube local main branch that uses Python 3.6 or earlier so it can run on Windows 7?

alive4ever · 2024-11-23T05:38:18Z

Any chance you could make a version of this similar to the youtube local main branch that uses Python 3.6 or earlier so it can run on Windows 7?

I actually tried several times to also build packages for oldwin (32 bit) as found on my github-actions history with no success. Tried several version of python, from oldwin to 3.8.7 with failure regarding finding _socks module.

So I reverted my github action recipe to only build for python 3.11 on Windows.

Add a setting option to enable video download. Downloading is disabled by default.

Use fake useragent library instead of hardcoded useragent.

Allow selecting innertube client via settings page.

Update ios and web innertube client context.

Make the loading of age_restricted contents configurable. Disabled by default.

Add functional mweb client. Also several features included: - Fix sig decryption code - Add nsig decryption with function body extraction taken from iv-org/inv_sig_helper, using dukpy to execute nsig js decryption code. - Add support to use visitorData from api response - Add support to use poToken by saving a json file as 'po_token_cache.txt': { "visitorData": "long_base64_visitorData_value", "poToken": "long_base64_poToken_value" } - More consistent request headers around several module: server.py, channel.py, and comments.py

Avoid removing dist-info directories so fake-useragent can start, since the module will call importlib.metadata on init. Also use HEAD instead of master during archive copying step to allow generate_release.py to run on branch other than master.

- use os.path.join to access visitorData.txt and signature timestamp cache file - remove visitorData.txt if file is more than 24h old. - try accessing responseContext and raise exception on KeyError

- use os.path.join to define visitorData.txt file - use OSError exception instead of blank exception

- remove flpc import - add re.escape to extracted function name before appending to list - remove debugging messages

remove flpc from requirements, since built-in python regex engine is good enough.

Return error message when n signature is None

Remove debugging messages during yt-api request

Only send visitor data header to specific google domains.

Update ios innertube client context.

Update to require fake-useragent>=2.0.0

Update player version for innertube clients.

Emergency fix for player 3bb1f723 to fix both signature and `n` query parameter decryption. Credits to /yt-dlp team.

Fix wrong indentation in po_token conditinal statements.

alive4ever · 2024-12-14T10:10:59Z

Update for today:

Currently experiencing 403 errors with mweb client after ±1 minutes of video playback on formats other than integrated 360p format. Still haven't found its cause.

Full video can only be played on mweb with integrated 360p format (itag 18).

alive4ever · 2024-12-17T10:22:34Z

Update for today:

Currently experiencing 403 errors with mweb client after ±1 minutes of video playback on formats other than integrated 360p format. Still haven't found its cause.

Full video can only be played on mweb with integrated 360p format (itag 18).

Update for today:

It seems that the cause of 1 minute playable stream is mweb client started to require po_token. According to BgUtils author, this is case 2 of mandatory po_token use.

When to Use a PoToken

YouTube's web player checks the "sps" (StreamProtectionStatus) of each media segment request (only if using UMP or SABR; our browser example uses UMP) to determine if the stream needs a PoToken.

Status 1: The stream is either already using a PoToken or does not need one.
Status 2: The stream requires a PoToken but will allow the client to request up to 1-2 MB of data before interrupting playback.
Status 3: The stream requires a PoToken and will interrupt playback immediately.

Adding data/po_token_cache.txt and enabling settings.use_po_token solves this issue for me.

Btw, po_token_cache.txt can be extracted from browser or created using bgutils or similar tools.

Any improvement suggestion is appreciated.

alive4ever force-pushed the feature-innertube-client-fix branch 2 times, most recently from 1da876d to 5276825 Compare November 5, 2024 15:12

user234683 reviewed Nov 10, 2024

View reviewed changes

MBCX mentioned this pull request Nov 22, 2024

Error getting mssing player: This video is unavailable #221

Closed

alive4ever added 15 commits November 24, 2024 01:05

settings: add option to enable video download

e5f74b0

Add a setting option to enable video download. Downloading is disabled by default.

misc: use fake useragent library

2c83c72

Use fake useragent library instead of hardcoded useragent.

settings: add innertube client selection

c19ee02

Allow selecting innertube client via settings page.

util: update innertube client context

c1346e5

Update ios and web innertube client context.

player: disable loading of age_restricted content

e731ca9

Make the loading of age_restricted contents configurable. Disabled by default.

util: more suggested fixes

7c4acd2

- use os.path.join to access visitorData.txt and signature timestamp cache file - remove visitorData.txt if file is more than 24h old. - try accessing responseContext and raise exception on KeyError

server: more fixes

1f51512

- use os.path.join to define visitorData.txt file - use OSError exception instead of blank exception

util: fix signature_timestamp function

5a78bfa

yt_data_extract: remove flpc import

96774a0

- remove flpc import - add re.escape to extracted function name before appending to list - remove debugging messages

requirements: remove flpc

b63842c

remove flpc from requirements, since built-in python regex engine is good enough.

yt_data_extract: return error message if no n signature present

92734fb

Return error message when n signature is None

util: remove debugging messages

418511a

Remove debugging messages during yt-api request

server: only send visitor data to specific domains

c7e1cac

Only send visitor data header to specific google domains.

alive4ever force-pushed the feature-innertube-client-fix branch from f50cae3 to c7e1cac Compare November 24, 2024 02:05

alive4ever added 5 commits November 26, 2024 09:41

util: update ios client

f5e27d5

Update ios innertube client context.

update fake-useragent module

5e2a156

Update to require fake-useragent>=2.0.0

util: update player version

d0c6db1

Update player version for innertube clients.

yt_data_extract: fix signature decryption

03cc5f9

Emergency fix for player 3bb1f723 to fix both signature and `n` query parameter decryption. Credits to /yt-dlp team.

server: fix po_token use

c2d15df

Fix wrong indentation in po_token conditinal statements.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature innertube client fix #219

Feature innertube client fix #219

alive4ever commented Oct 28, 2024

alive4ever commented Nov 8, 2024

user234683 commented Nov 8, 2024 via email

alive4ever commented Nov 8, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

pelag0s Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

user234683 Nov 10, 2024

alive4ever Nov 23, 2024

sapnolas commented Nov 22, 2024

alive4ever commented Nov 23, 2024

alive4ever commented Dec 14, 2024

alive4ever commented Dec 17, 2024 •

edited

Loading

Feature innertube client fix #219

Are you sure you want to change the base?

Feature innertube client fix #219

Conversation

alive4ever commented Oct 28, 2024

alive4ever commented Nov 8, 2024

user234683 commented Nov 8, 2024 via email

alive4ever commented Nov 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sapnolas commented Nov 22, 2024

alive4ever commented Nov 23, 2024

alive4ever commented Dec 14, 2024

alive4ever commented Dec 17, 2024 • edited Loading

alive4ever commented Dec 17, 2024 •

edited

Loading