[YoutubeDL] Fix function signature parser (refs ytdl-org/#30363) #30366

lanegramling · 2021-12-15T04:47:37Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

Searched the bugtracker for similar pull requests
Read adding new extractor tutorial
Read youtube-dl coding conventions and adjusted the code to meet them
Covered the code with tests (note that PRs without tests will be REJECTED)
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)
This is NOT my code. This code is copied from PSlava's merged fix to the same issue (Link to issue: youtube_dl.utils.ExtractorError: Could not find JS function 'na' #30363) observed in the yt-dlp repository using the same regexes for signature parsing. Link to analogous yt-dlp bugfix: [youtube] fix for signature decode function detection yt-dlp/yt-dlp#641. (This also doubles as a qualifier that this code is released under Unlicense by the original author)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

Relevant issue: #30363
Referring to the discussion in the comments of the aforementioned PR on yt-dlp, along with what is apparent upon scrutinizing Youtube's base.js (https://www.youtube.com/s/player/850eb2bc/player_ias.vflset/en_US/base.js), function signatures are not always 2 characters. An adjustment is made to the regex here to reflect this. I don't know when Youtube changed this, but I have had ~>80% of requests failing for the last 24 hours approx., due to a failure in parsing the function in youtube_dl/extractor/youtube.py::_parse_sig_js. Applying this fix to my local branch has resolved the ExtractorError exceptions and everything works fine now.

Resolves #30363
Resolves #30364
Resolves #30367
Resolves #30369.
Resolves #30370
Resolves #30372.

philippkeller · 2021-12-15T09:42:49Z

I tested this change and it worked for me, thanks for fixing, @lanegramling

phreaknerd

Working well. Thanks

sobhan-m94 · 2021-12-15T10:58:26Z

thanks for fixing, @lanegramling

stefaner1 · 2021-12-15T11:30:20Z

How can I approve it? Can I download it using python pip before the changes are committed? It works for me and I need it fast.

dirkf · 2021-12-15T11:40:43Z

Thanks for this. To help updating while releases are on hold I've pulled the commit into PR #30184 so that a single youtube.py update fixes both issues.

Please add this text to your PR description, to link the potentially resolved issues through GH magic:

Resolves #30363
Resolves #30364
Resolves #30367
Resolves #30369.
Resolves #30370
Resolves #30372.

dirkf · 2021-12-15T12:24:34Z

youtube_dl/extractor/youtube.py

+             r'\bm=(?P<sig>[a-zA-Z0-9$]{2,})\(decodeURIComponent\(h\.s\)\)',
+             r'\bc&&\(c=(?P<sig>[a-zA-Z0-9$]{2,})\(decodeURIComponent\(c\)\)',
+             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\);[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}\(a,\d+\)',
+             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\)',


These 2 regexes ought to be merged, though maybe not in the scope of this PR:

r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function$\s*a\s*$\s*{\s*a\s*=\s*a\.split$\s*""\s*$(?:;[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}$a,\d+$)?',

👍work for me, modify the file by hand

works also for me 👍

Question What do windows peeps do?

Suffer.

In addition to that they can install python and run this themselves quite fine.

nicolasbouthors · 2021-12-15T13:38:02Z

This works fine here 👍

LooOOooM · 2021-12-15T14:28:18Z

Can someone please also compile a windows version or pointing us the the path, where the related file usually sits? I am using Yoube-DLG on windows in version 0.4. I cannot find the file youtube.py anywhere to motify it by hand.
THX!

dirkf · 2021-12-15T14:34:30Z

Read issue #29326 for hints.

lanegramling · 2021-12-15T18:36:07Z

Thanks for this. To help updating while releases are on hold I've pulled the commit into PR #30184 so that a single youtube.py update fixes both issues.

Please add this text to your PR description, to link the potentially resolved issues through GH magic:
Resolves #30363
Resolves #30364
Resolves #30367
Resolves #30369.
Resolves #30370
Resolves #30372.

Added. Thanks for taking the time to combine the PRs.

wiyco · 2021-12-16T00:48:26Z

youtube_dl/extractor/youtube.py

-             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\)',
+             r'\bm=(?P<sig>[a-zA-Z0-9$]{2,})\(decodeURIComponent\(h\.s\)\)',
+             r'\bc&&\(c=(?P<sig>[a-zA-Z0-9$]{2,})\(decodeURIComponent\(c\)\)',
+             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\);[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}\(a,\d+\)',


Suggested change

r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function$\s*a\s*$\s*{\s*a\s*=\s*a\.split$\s*""\s*$;[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}$a,\d+$',

r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function$\s*a\s*$\s*{\s*a\s*=\s*a\.split$\s*""\s*$;[a-zA-Z0-9$]{2,}\.[a-zA-Z0-9$]{2,}$a,\d+$',

This works for me.👍

I made this observation as well. I was torn here - Wasn't sure whether it would be necessary, so I did some tests without these extra commas and everything worked just fine, so I left them out. I will elect to the suggestion of the greater democracy for opinions on this.

This worked for me.

Thanks heaps, worked a charm. I always wondered how the updates fixed these kinds of errors in the past. Love open source, love learning, love sharing.

snepderg · 2021-12-16T02:06:48Z

How would I go about decompiling this properly so I can edit the file myself? I'm using Chocolatey.
(I should note I've tried to do my own research but I have never opened a .exe before, and that's the only file I think I can get at with Chocolatey)

lanegramling · 2021-12-16T02:39:45Z

I would suggest the reverse route: cloning a source repository with the included fix found in #30366 , and performing the build yourself based on the instructions available in the readme. That is, if you do need to use a binary version to agree with your existing environment.

alvinbengeorge · 2021-12-16T05:41:39Z

Could you please accept this soon and push it to pypl

pianodude1011 · 2021-12-16T06:15:27Z

I tried the fix by uninstalling youtube-dl and reinstalling from this branch, then running youtube-dl https://www.youtube.com/watch?v=2GG8xQDH2cI and still get the same errors. I'm erring more on the side that I somehow messed up but I figured it might be helpful to note this in case others are also having issues.

If it helps, I ran
pip3 uninstall youtube-dl
pip3 install git+https://github.com/lanegramling/youtube-dl@fix-function-sig-parser#egg=youtube-dl

supersu-man

This fixed the issue

supersu-man · 2021-12-16T07:41:57Z

Could you please accept this soon and push it to pypl

@alvinbengeorge pip install git+https://github.com/lanegramling/youtube-dl@fix-function-sig-parser should do the job.

mattalibodie · 2021-12-16T08:34:50Z

I tried the fix by uninstalling youtube-dl and reinstalling from this branch, then running youtube-dl https://www.youtube.com/watch?v=2GG8xQDH2cI and still get the same errors. I'm erring more on the side that I somehow messed up but I figured it might be helpful to note this in case others are also having issues.

If it helps, I ran pip3 uninstall youtube-dl pip3 install git+https://github.com/lanegramling/youtube-dl@fix-function-sig-parser#egg=youtube-dl

Thanks! It work for me.

pianodude1011 · 2021-12-16T17:35:34Z

Thanks! It work for me.

I'm glad it worked for you but it didn't work for me :v

Chi-K24 · 2021-12-16T18:04:05Z

pip3 install git+https://github.com/lanegramling/youtube-dl@fix-function-sig-parser#egg=youtube-dl

Thanks my dude, it worked like a charm!

000011111111 · 2021-12-16T19:39:42Z

Using yt-dlp worked for me. Workflow here!

https://youtu.be/dAc5VIDIs5k

snepderg · 2021-12-16T21:00:22Z

I decided to just switch to yt-dlp, someone was kind enough to PR youtube-dl-explorer (which is what I use):
notthebee/ytdl-explorer#5

Best of luck folks!

* commit 'afaec67d45df19642eb4af230b506127af84ceff': Fixes 404 Issue Currently Present On XVIDEOS.COM release 2021.12.17 [ChangeLog] Actualize [ci skip] [youtube] Update signature function patterns (closes ytdl-org#30363) (ytdl-org#30366)

…tdl-org#30366)

Playback fixed based on similar change in youtube-dl: ytdl-org/youtube-dl#30366

[youtube] Fix function signature parser (refs ytdl-org/ytdl-org#30363)

1a09168

lanegramling changed the title ~~[youtube] Fix function signature parser (refs ytdl-org/#30363)~~ [YoutubeDL] Fix function signature parser (refs ytdl-org/#30363) Dec 15, 2021

lanegramling mentioned this pull request Dec 15, 2021

youtube_dl.utils.ExtractorError: Could not find JS function 'na' #30363

Closed

6 tasks

phreaknerd approved these changes Dec 15, 2021

View reviewed changes

dirkf mentioned this pull request Dec 15, 2021

[YouTube] Randomly slow youtube download speed #29326

Closed

dirkf mentioned this pull request Dec 15, 2021

[YouTube] Unthrottle downloads by responding to the "n" parameter challenge #30184

Closed

11 tasks

dirkf reviewed Dec 15, 2021

View reviewed changes

lanegramling mentioned this pull request Dec 15, 2021

Update youtube.py #30379

Closed

11 tasks

wiyco reviewed Dec 16, 2021

View reviewed changes

rcy mentioned this pull request Dec 16, 2021

upstream youtube-dl is broken rcy/mixchat#12

Closed

paulomanrique approved these changes Dec 16, 2021

View reviewed changes

supersu-man approved these changes Dec 16, 2021

View reviewed changes

EKINGcoolbreeze approved these changes Dec 16, 2021

View reviewed changes

dstftw merged commit b5242da into ytdl-org:master Dec 16, 2021

Roki100 mentioned this pull request Dec 17, 2021

Problem with bot splamy Splamy/TS3AudioBot#953

Closed

dirkf linked an issue Jan 29, 2022 that may be closed by this pull request

yotube dl error Signature extraction failed: Traceback (most recent call last): #30376

Closed

5 tasks

Lesmiscore mentioned this pull request Feb 20, 2022

Add build workflow #30644

Open

14 tasks

NoSuck pushed a commit to NoSuck/youtube-dl-1 that referenced this pull request Apr 5, 2022

[youtube] Update signature function patterns (closes ytdl-org#30363) (y…

595164a

…tdl-org#30366)

dirkf mentioned this pull request Jun 16, 2022

[YouTube] Implement optional YouTube server-imposed throttling bypass #28859

Closed

3 tasks

dirkf pushed a commit to dirkf/youtube-dl that referenced this pull request Jun 27, 2022

[youtube] Update signature function patterns (closes ytdl-org#30363) (y…

3b35892

…tdl-org#30366)

giantorth added a commit to giantorth/LMS-YouTube that referenced this pull request Oct 10, 2022

Update regex in Signature.pm to fix playback

83fd7df

Playback fixed based on similar change in youtube-dl: ytdl-org/youtube-dl#30366

giantorth mentioned this pull request Oct 10, 2022

Update regex in Signature.pm to fix playback philippe44/LMS-YouTube#41

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YoutubeDL] Fix function signature parser (refs ytdl-org/#30363) #30366

[YoutubeDL] Fix function signature parser (refs ytdl-org/#30363) #30366

lanegramling commented Dec 15, 2021 •

edited

Loading

philippkeller commented Dec 15, 2021

phreaknerd left a comment

sobhan-m94 commented Dec 15, 2021

stefaner1 commented Dec 15, 2021

dirkf commented Dec 15, 2021 •

edited

Loading

dirkf Dec 15, 2021

sinfrecu Dec 15, 2021

rdxmb Dec 16, 2021

CagesThrottleUs Dec 16, 2021

Asday Dec 24, 2021

nicolasbouthors commented Dec 15, 2021

LooOOooM commented Dec 15, 2021

dirkf commented Dec 15, 2021

lanegramling commented Dec 15, 2021

wiyco Dec 16, 2021

lanegramling Dec 16, 2021

Oluwasetemi Dec 16, 2021

ghost Dec 16, 2021

snepderg commented Dec 16, 2021 •

edited

Loading

lanegramling commented Dec 16, 2021

alvinbengeorge commented Dec 16, 2021 •

edited

Loading

pianodude1011 commented Dec 16, 2021 •

edited

Loading

supersu-man left a comment

supersu-man commented Dec 16, 2021

mattalibodie commented Dec 16, 2021

pianodude1011 commented Dec 16, 2021

Chi-K24 commented Dec 16, 2021

000011111111 commented Dec 16, 2021 •

edited

Loading

snepderg commented Dec 16, 2021 •

edited

Loading

	r'(?:\b\|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s=\sfunction\(\sa\s\)\s{\sa\s=\sa\.split\(\s""\s\);[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}\(a,\d+\)',
	r'(?:\b\|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s=\sfunction\(\sa\s\)\s{\sa\s=\sa\.split\(\s""\s\);[a-zA-Z0-9$]{2,}\.[a-zA-Z0-9$]{2,}\(a,\d+\)',

[YoutubeDL] Fix function signature parser (refs ytdl-org/#30363) #30366

[YoutubeDL] Fix function signature parser (refs ytdl-org/#30363) #30366

Conversation

lanegramling commented Dec 15, 2021 • edited Loading

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

philippkeller commented Dec 15, 2021

phreaknerd left a comment

Choose a reason for hiding this comment

sobhan-m94 commented Dec 15, 2021

stefaner1 commented Dec 15, 2021

dirkf commented Dec 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolasbouthors commented Dec 15, 2021

LooOOooM commented Dec 15, 2021

dirkf commented Dec 15, 2021

lanegramling commented Dec 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snepderg commented Dec 16, 2021 • edited Loading

lanegramling commented Dec 16, 2021

alvinbengeorge commented Dec 16, 2021 • edited Loading

pianodude1011 commented Dec 16, 2021 • edited Loading

supersu-man left a comment

Choose a reason for hiding this comment

supersu-man commented Dec 16, 2021

mattalibodie commented Dec 16, 2021

pianodude1011 commented Dec 16, 2021

Chi-K24 commented Dec 16, 2021

000011111111 commented Dec 16, 2021 • edited Loading

snepderg commented Dec 16, 2021 • edited Loading

lanegramling commented Dec 15, 2021 •

edited

Loading

dirkf commented Dec 15, 2021 •

edited

Loading

snepderg commented Dec 16, 2021 •

edited

Loading

alvinbengeorge commented Dec 16, 2021 •

edited

Loading

pianodude1011 commented Dec 16, 2021 •

edited

Loading

000011111111 commented Dec 16, 2021 •

edited

Loading

snepderg commented Dec 16, 2021 •

edited

Loading