[YouTube] Parse and execute deobfuscation code without JavaScript #529

Stypox · 2021-01-28T09:32:34Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
[not needed] I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

I don't know if this can be merged safely, it is more of an experiment. @FireMasterK

This is the JS this PR parses, which is built by loadDeobfuscationCode():

var Aw = {
	bQ: function(a,b) {
		var c = a[0];
		a[0] = a[b%a.length];
		a[b%a.length] = c
	},
	RY: function(a) {
		a.reverse()
	},
	aD: function(a,b) {
		a.splice(0,b)
	}
};
var Bw = function(a) {
	a = a.split("");
	Aw.bQ(a,35);
	Aw.aD(a,3);
	Aw.RY(a,30);
	Aw.bQ(a,44);
	Aw.aD(a,1);
	return a.join("")
};

Debug APK: app-debug.zip

FireMasterK · 2021-01-28T09:55:17Z

There's a strange unrelated error when I try that apk:

Exception

User Action: searched
Request: test
Content Country: IN
Content Language: en-IN
App Language: en_IN
Service: YouTube
Version: 0.20.8
OS: Linux Android 10 - 29

Crash log

java.lang.RuntimeException: This is a runtime error, k = 29
	at org.schabi.newpipe.DownloaderImpl.execute(DownloaderImpl.java:235)
	at org.schabi.newpipe.extractor.downloader.Downloader.get(Downloader.java:70)
	at org.schabi.newpipe.extractor.services.youtube.YoutubeParsingHelper.getJsonResponse(YoutubeParsingHelper.java:605)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeSearchExtractor.onFetchPage(YoutubeSearchExtractor.java:66)
	at org.schabi.newpipe.extractor.Extractor.fetchPage(Extractor.java:54)
	at org.schabi.newpipe.extractor.search.SearchInfo.getInfo(SearchInfo.java:29)
	at org.schabi.newpipe.util.ExtractorHelper.lambda$searchFor$0(ExtractorHelper.java:80)
	at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$qyxpuXgomWa-cbONQns-pd7zxm0.call(Unknown Source:8)
	at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
	at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4813)
	at io.reactivex.rxjava3.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
	at io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:614)
	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)
	at io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:919)

The implementation code wise seems to be good. (From a glance)

Stypox · 2021-01-28T10:23:14Z

@FireMasterK oh, I uploaded the wrong apk, sorry. I updated it now.

FireMasterK · 2021-01-28T11:00:40Z

It works great! It also appears that it might also decipher these videos faster too. (Likely due to not needing to recreate a script context everytime.)

Stypox · 2021-01-28T13:14:16Z

It works great! It also appears that it might also decipher these videos faster too. (Likely due to not needing to recreate a script context everytime.)

Yeah, that's the whole point :-D

FireMasterK · 2021-01-30T19:22:11Z

I just want to add, Invidious does this already so I think it should be safe to merge
See: https://github.com/iv-org/invidious/blob/master/src/invidious/helpers/signatures.cr

XiangRongLin · 2021-02-16T14:34:28Z

@Stypox can you rebase and see if the test pass?

Stypox · 2021-02-16T15:38:21Z

@XiangRongLin it passed! The stream urls could still lead to 404 errors due to a wrong deobfuscation, since that's not tested, but the deobfuscation process would have probably raised an IndexOutOfBound exception at some point if something was wrong. So I think everything is fine.

XiangRongLin · 2021-02-16T16:25:15Z

I'm going to use the app with this extractor for the next few days. If nothing comes app, it can be merged

FireMasterK · 2021-02-16T19:05:57Z

YouTube has already updated their deobfuscation code once since this PR was opened, it should be safe imo if it still works :D

XiangRongLin · 2021-02-19T19:26:08Z

...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java

+            final String[] stringSteps = deobfuscationCode.split(containerName + "\\.");
+            boolean first = true;
+            cachedDeobfuscationFunctions = new ArrayList<>();
+            for (final String step : stringSteps) {


Since this is an array you can just use a for i loop with i=1 instead of the weird first=true setup

XiangRongLin · 2021-02-23T06:33:37Z

This occured in my preunified version with this extractor. I searched for the video and clicked on it. then the error came.

it works on my preunified with extractor v0.20.10 and yesterday it worked too on deobf extractor

Exception

User Action: requested stream
Request: https://www.youtube.com/watch?v=l0GN40EL1VU
Content Country: US
Content Language: en-US
App Language: en_US
Service: YouTube
Version: 0.20.9
OS: Linux Android 11 - 30

Crash log

org.schabi.newpipe.extractor.stream.StreamInfo$StreamExtractException: Could not get any stream. See error variable to get further details.
	at org.schabi.newpipe.extractor.stream.StreamInfo.extractStreams(StreamInfo.java:194)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:72)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:64)
	at org.schabi.newpipe.util.ExtractorHelper.lambda$getStreamInfo$3(ExtractorHelper.java:116)
	at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$YTHJjScxCJNO1LTCqs3IKy35iyY.call(Unknown Source:4)
	at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.single.SingleDoOnSuccess.subscribeActual(SingleDoOnSuccess.java:35)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.maybe.MaybeFromSingle.subscribeActual(MaybeFromSingle.java:41)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.drain(MaybeConcatArray.java:153)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.request(MaybeConcatArray.java:78)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onSubscribe(FlowableElementAtMaybe.java:66)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray.subscribeActual(MaybeConcatArray.java:42)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeToSingle.subscribeActual(MaybeToSingle.java:46)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:923)

XiangRongLin · 2021-02-23T06:36:22Z

this one too https://www.youtube.com/watch?v=WDH_nJM3djc.

Search query: summer joe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YouTube] Parse and execute deobfuscation code without JavaScript #529

[YouTube] Parse and execute deobfuscation code without JavaScript #529

Stypox commented Jan 28, 2021 •

edited

Loading

FireMasterK commented Jan 28, 2021

Stypox commented Jan 28, 2021

FireMasterK commented Jan 28, 2021

Stypox commented Jan 28, 2021

FireMasterK commented Jan 30, 2021

XiangRongLin commented Feb 16, 2021

Stypox commented Feb 16, 2021

XiangRongLin commented Feb 16, 2021

FireMasterK commented Feb 16, 2021

XiangRongLin Feb 19, 2021

XiangRongLin commented Feb 23, 2021 •

edited

Loading

XiangRongLin commented Feb 23, 2021 •

edited

Loading

FireMasterK commented Feb 26, 2021

[YouTube] Parse and execute deobfuscation code without JavaScript #529

[YouTube] Parse and execute deobfuscation code without JavaScript #529

Conversation

Stypox commented Jan 28, 2021 • edited Loading

FireMasterK commented Jan 28, 2021

Exception

Stypox commented Jan 28, 2021

FireMasterK commented Jan 28, 2021

Stypox commented Jan 28, 2021

FireMasterK commented Jan 30, 2021

XiangRongLin commented Feb 16, 2021

Stypox commented Feb 16, 2021

XiangRongLin commented Feb 16, 2021

FireMasterK commented Feb 16, 2021

XiangRongLin Feb 19, 2021

Choose a reason for hiding this comment

XiangRongLin commented Feb 23, 2021 • edited Loading

Exception

XiangRongLin commented Feb 23, 2021 • edited Loading

FireMasterK commented Feb 26, 2021

Stypox commented Jan 28, 2021 •

edited

Loading

XiangRongLin commented Feb 23, 2021 •

edited

Loading

XiangRongLin commented Feb 23, 2021 •

edited

Loading