Fix ReCaptcha activity and correctly save obtained cookies #3035

Stypox · 2020-01-29T19:17:53Z

[ x] I carefully read the contribution guidelines and agree to them.

Description

This pr aims to solve the many issues regarding the ReCaptchaActivity. In #2527 I tried to fix it, though I was unable to test the changes since I couldn't reproduce any ReCAPTCHAs anymore, so it kept crashing. After @B0pol in #2979 provided a way to reproduce, I was able to redesign the activity. Before the cookies were extracted from the page on close, but unfortunately this didn't imply that the captha was solved, since there could have been redirections or the page didn't close on resolution. Now the user is provided with a "Done" button to press when he finishes solving the captcha, that sets the cookies to the downloader.
Sometimes services return a recaptcha challenge error code (429) just to indicate Too Many Requests, so in that case the ReCaptchaActivity is opened, but the only thing the user has to do is press "Done" since there are no recaptchas to solve. ~~This is the only scenario I was able to test, and I am confident the Youtube cookie extraction method works since I didn't change it,~~ (it didn't work, but it should be fixed now) ~~though I am not entirely sure that the handleCookies function is always called (someone who knows how WebViewClients work?).~~ (now it is always called on close).

Also renamed some recaptcha-related strings to have them match the naming conventions, leading to many changed strings.xml files.

Follow-ups for next pull requests

Maybe we should keep a hash-table with all cookies and their values and update that table every time we need, instead of overwriting the whole cookie string every time. Since, for now, the only service with recaptchas is YouTube, there is no problem to always overwrite the old Downloader cookies, but for the future we might need to change this.
Anyway in the handleCookies function I left the space for other services (other than Youtube) to try and extract cookies.

Fixes & debug apk

Fixes #3023 fixes #2979 fixes #2924 fixes #2788 fixes #2484
Debug apk: app-debug.zip

B0pol · 2020-01-29T19:35:49Z

I car confirm it's fixing fine #2979.
Just a question : could you make background agree with theme?

This is the page I had with peertube.co.uk , my theme was default (grey)

TobiGr · 2020-01-29T20:05:27Z

Just a question : could you make background agree with theme?

This should be quite easy. Maybe we can pass either some js to the webview after loading the site. For converting R.color ids which are used in the themes to hex, you might use the the getHexRGBColor method from here. However, it should be move the to an util class.

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java

Stypox · 2020-01-30T19:55:30Z

@B0pol I am not sure it is safe to change the background of the webview. It would surely require some (potentially slow) JavaScript code to be run, and it could create graphical problems on complex recaptcha pages (I use an extension to change colors on my Firefox ("Dark Background and Light Text"), but on some websites it doesn't work properly).
I don't think it is really a problem to have an unthemed recaptcha activity, because after all it is shown very rarely. But anyway if we decide to implement it, I think it would be better suited for a separate PR, to make merging this faster ;-).

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java

Even if the page didn't auto-close

juanfra684 · 2020-02-01T17:30:10Z

I'm testing the apk file attached to the first comment but it doesn't fix my problems with YT captchas. I always get a captcha when I try to watch a YT video. I solve the captcha and can watch the video in the webview of NewPipe. I press ✔️ when I solve the captchas but then I get again another captcha with the next video which I watch.

Stypox · 2020-02-01T17:54:29Z

@juanfra684 could you test this apk, based on the last commit?
app-debug_1.zip

If even the apk above doesn't work, please use the following one. When you press :heavy_check_mark (after solving) it throws an error on purpose (so that I can see the content of cookies): please paste the bug report content here.
app-debug_2.zip

Stypox · 2020-02-01T17:55:11Z

@TobiGr this is how the Activity looks like now. Is it ok? I decided to move the button to the right for better UI, since it's an action.

juanfra684 · 2020-02-01T18:09:09Z

@juanfra684 could you test this apk, based on the last commit?
app-debug_1.zip

app-debug_1.zip is corrupted. I can extract the apk but I can't install it. app-debug_2.zip installs fine.

Stypox · 2020-02-01T20:36:10Z

@juanfra684 Could you try again with this apk (as a replacement for app-debug_1.zip)? Thank you :-)
app-debug.zip

juanfra684 · 2020-02-01T22:03:50Z

I've the same problems with the new apk. This is the output of app-debug_2 with the captcha solved:

{
   "user_action": "ui error",
   "request": "App crash, UI failure",
   "content_language": "GB",
   "service": "none",
   "package": "org.schabi.newpipe.debug",
   "version": "0.18.2",
   "os": "Linux Android 7.1.2 - 25",
   "time": "2020-02-01 21:58",
   "exceptions": [
      "java.lang.NullPointerException: Found cookies: YSC=AeJKpj2WGsA; s_gl=e3ea30620a3c9e68880a30237b86c6cfcwIAAABHQg==; VISITOR_INFO1_LIVE=fflNk6rzzmc; GPS=1; CONSENT=WP.283253; YSC=AeJKpj2WGsA; s_gl=e3ea30620a3c9e68880a30237b86c6cfcwIAAABHQg==; VISITOR_INFO1_LIVE=fflNk6rzzmc; GPS=1; CONSENT=WP.283253; YSC=AeJKpj2WGsA; s_gl=e3ea30620a3c9e68880a30237b86c6cfcwIAAABHQg==; VISITOR_INFO1_LIVE=fflNk6rzzmc; GPS=1; CONSENT=WP.283253\n\tat org.schabi.newpipe.ReCaptchaActivity.saveCookiesAndFinish(ReCaptchaActivity.java:134)\n\tat org.schabi.newpipe.ReCaptchaActivity.onOptionsItemSelected(ReCaptchaActivity.java:121)\n\tat android.app.Activity.onMenuItemSelected(Activity.java:3248)\n\tat androidx.fragment.app.FragmentActivity.onMenuItemSelected(FragmentActivity.java:384)\n\tat androidx.appcompat.app.AppCompatActivity.onMenuItemSelected(AppCompatActivity.java:219)\n\tat androidx.appcompat.view.WindowCallbackWrapper.onMenuItemSelected(WindowCallbackWrapper.java:109)\n\tat androidx.appcompat.view.WindowCallbackWrapper.onMenuItemSelected(WindowCallbackWrapper.java:109)\n\tat androidx.appcompat.app.ToolbarActionBar$2.onMenuItemClick(ToolbarActionBar.java:64)\n\tat androidx.appcompat.widget.Toolbar$1.onMenuItemClick(Toolbar.java:207)\n\tat androidx.appcompat.widget.ActionMenuView$MenuBuilderCallback.onMenuItemSelected(ActionMenuView.java:781)\n\tat androidx.appcompat.view.menu.MenuBuilder.dispatchMenuItemSelected(MenuBuilder.java:840)\n\tat androidx.appcompat.view.menu.MenuItemImpl.invoke(MenuItemImpl.java:158)\n\tat androidx.appcompat.view.menu.MenuBuilder.performItemAction(MenuBuilder.java:991)\n\tat androidx.appcompat.view.menu.MenuBuilder.performItemAction(MenuBuilder.java:981)\n\tat androidx.appcompat.widget.ActionMenuView.invokeItem(ActionMenuView.java:625)\n\tat androidx.appcompat.view.menu.ActionMenuItemView.onClick(ActionMenuItemView.java:151)\n\tat android.view.View.performClick(View.java:5637)\n\tat android.view.View$PerformClick.run(View.java:22433)\n\tat android.os.Handler.handleCallback(Handler.java:751)\n\tat android.os.Handler.dispatchMessage(Handler.java:95)\n\tat android.os.Looper.loop(Looper.java:154)\n\tat android.app.ActivityThread.main(ActivityThread.java:6121)\n\tat java.lang.reflect.Method.invoke(Native Method)\n\tat com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:889)\n\tat com.android.internal.os.ZygoteInit.main(ZygoteInit.java:779)\n"
   ],
   "user_comment": ""
}

Stypox · 2020-02-02T20:04:56Z

I've the same problems with the new apk.

Do you mean that you're not able to install it, or that solving the recaptcha does nothing? @juanfra684

This is the output of app-debug_2 with the captcha solved:

Thank you :-)

juanfra684 · 2020-02-02T20:20:03Z

I've the same problems with the new apk.

Do you mean that you're not able to install it, or that solving the recaptcha does nothing? @juanfra684

I can install it but YT always shows a captcha with every video. I can watch any video within the webview after of solve the first captcha, so my IP is not blacklisted. The problem only happens when I go to a video from NewPipe directly.

I've tried a few times solving the captchas and touching ✔️ but YT stills shows captchas with every new video.

Stypox · 2020-02-02T20:37:46Z

@juanfra684
Ok, I should have fixed the problem with the latest commit, thanks to the report you provided.
Sorry if I keep asking you to test APKs :-), here is hopefully the last one: app-debug.zip

juanfra684 · 2020-02-02T20:58:24Z

@juanfra684
Ok, I should have fixed the problem with the latest commit, thanks to the report you provided.
Sorry if I keep asking you to test APKs :-), here is hopefully the last one: app-debug.zip

Don't worry. NewPipe is one of my favorite apps, I've not problem helping to test the fixes.

Looks like your last commit fixed the problem. I only was asked to solve the captcha one time. After that everything worked as usual.

Thanks a lot for fixing this!

Stypox · 2020-02-02T21:06:22Z

Thank you very much for testing!
@TobiGr this should be ready :-D

TobiGr

Thanks! Looks good to me

mauriciocolli · 2020-02-04T01:59:11Z

For reference in the future, there's a very easy way to get reCAPTCHAs to test this issue: route your traffic through tor or any popular free proxy. Chances are good that a abuser/spammer already made it blacklisted.

But this appears to have fixed it.

PS: One thing though that was already mentioned, we are setting the cookies for EVERY request, even for those that the service don't use it, what about we don't do that? Maybe something like a cookie jar for services?

I think this could be done before the next release.

Stypox · 2020-02-27T13:48:52Z

For reference in the future, there's a very easy way to get reCAPTCHAs to test this issue: route your traffic through tor or any popular free proxy. Chances are good that a abuser/spammer already made it blacklisted.

How can I route my NewPipe's traffic through tor? @mauriciocolli

TobiGr · 2020-02-27T14:10:33Z

@Stypox Install Orbot. That app is in the GuardianProject repository which is enabled by default in F-Droid.

Stypox · 2020-02-27T14:34:45Z

Thanks

TobiGr added the bug Issue is related to a bug label Jan 29, 2020

TobiGr reviewed Jan 29, 2020

View reviewed changes

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java Show resolved Hide resolved

TobiGr added this to the 0.18.3 milestone Jan 29, 2020

TobiGr reviewed Jan 30, 2020

View reviewed changes

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java Outdated Show resolved Hide resolved

Stypox mentioned this pull request Jan 31, 2020

Recaptcha crash on app open #3023

Closed

TobiGr reviewed Feb 1, 2020

View reviewed changes

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java Outdated Show resolved Hide resolved

mqus and others added 9 commits February 1, 2020 18:24

fixes TeamNewPipe#3021, see also TeamNewPipe/NewPipe-legacy#21

c0519d8

Change recaptcha string names to match style

61d102d

Add "Done" drawable (only white since it is used on toolbar)

a3d8848

Fix ReCaptchaActivity crash and save cookies correctly

daa4fd5

Remove space between "Done" button and ReCaptchaActivity title

4e1638f

Improve formatting

fe138f6

Move "Done" button and make it theme conpliant in ReCaptcha

0cc890a

Try to extract cookies just before closing recaptcha activity

9b09028

Even if the page didn't auto-close

Remove left-behind Log

1bf55c2

Stypox force-pushed the recaptcha branch from 1a6303d to 1bf55c2 Compare February 1, 2020 17:24

Merge branch 'dev' of github.com:TeamNewPipe/NewPipe into recaptcha

f95d51b

Stypox added 2 commits February 2, 2020 21:33

Fix addYoutubeCookies functions (Yt changed things lately)

6da9096

Merge branch 'dev' into recaptcha

3372bac

Remove unused imports and clean up comment style

b684115

TobiGr approved these changes Feb 2, 2020

View reviewed changes

TobiGr merged commit 471ce4a into TeamNewPipe:dev Feb 2, 2020

This was referenced Feb 4, 2020

Accessing YouTube videos needs completing Captcha. #3050

Closed

reCAPTCHA challenge #3062

Closed

Stypox deleted the recaptcha branch February 9, 2020 10:48

Stypox linked an issue Feb 9, 2020 that may be closed by this pull request

reCAPTCHA challenge #3062

Closed

3 tasks

This was referenced Feb 9, 2020

stuck recaptcha after solving #2896

Closed

reCAPTCHA challenge requested #3093

Closed

This was referenced Feb 19, 2020

Release v0.18.4 #3113

Merged

Release v0.18.4 #3114

Closed

Non Authorized Apps Captcha Verification #3134

Closed

Stypox mentioned this pull request Mar 25, 2020

ReCaptcha #3272

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ReCaptcha activity and correctly save obtained cookies #3035

Fix ReCaptcha activity and correctly save obtained cookies #3035

Stypox commented Jan 29, 2020 •

edited

Loading

B0pol commented Jan 29, 2020 •

edited

Loading

TobiGr commented Jan 29, 2020

Stypox commented Jan 30, 2020 •

edited

Loading

juanfra684 commented Feb 1, 2020

Stypox commented Feb 1, 2020 •

edited

Loading

Stypox commented Feb 1, 2020 •

edited

Loading

juanfra684 commented Feb 1, 2020

Stypox commented Feb 1, 2020 •

edited

Loading

juanfra684 commented Feb 1, 2020

Stypox commented Feb 2, 2020

juanfra684 commented Feb 2, 2020

Stypox commented Feb 2, 2020

juanfra684 commented Feb 2, 2020

Stypox commented Feb 2, 2020

TobiGr left a comment

mauriciocolli commented Feb 4, 2020

Stypox commented Feb 27, 2020 •

edited

Loading

TobiGr commented Feb 27, 2020

Stypox commented Feb 27, 2020

Fix ReCaptcha activity and correctly save obtained cookies #3035

Fix ReCaptcha activity and correctly save obtained cookies #3035

Conversation

Stypox commented Jan 29, 2020 • edited Loading

Description

Follow-ups for next pull requests

Fixes & debug apk

B0pol commented Jan 29, 2020 • edited Loading

TobiGr commented Jan 29, 2020

Stypox commented Jan 30, 2020 • edited Loading

juanfra684 commented Feb 1, 2020

Stypox commented Feb 1, 2020 • edited Loading

Stypox commented Feb 1, 2020 • edited Loading

juanfra684 commented Feb 1, 2020

Stypox commented Feb 1, 2020 • edited Loading

juanfra684 commented Feb 1, 2020

Stypox commented Feb 2, 2020

juanfra684 commented Feb 2, 2020

Stypox commented Feb 2, 2020

juanfra684 commented Feb 2, 2020

Stypox commented Feb 2, 2020

TobiGr left a comment

Choose a reason for hiding this comment

mauriciocolli commented Feb 4, 2020

Stypox commented Feb 27, 2020 • edited Loading

TobiGr commented Feb 27, 2020

Stypox commented Feb 27, 2020

Stypox commented Jan 29, 2020 •

edited

Loading

B0pol commented Jan 29, 2020 •

edited

Loading

Stypox commented Jan 30, 2020 •

edited

Loading

Stypox commented Feb 1, 2020 •

edited

Loading

Stypox commented Feb 1, 2020 •

edited

Loading

Stypox commented Feb 1, 2020 •

edited

Loading

Stypox commented Feb 27, 2020 •

edited

Loading