SponsorBlock Implementation #3205
Conversation
app/src/main/java/org/schabi/newpipe/views/MarkableSeekBar.java
Outdated
Show resolved
Hide resolved
|
i was expecting linus as example, not dissapointed 😆 |
There was a problem hiding this comment.
@TheAssassin We need to update our privacy policy when merging this. From https://sponsor.ajay.app/about/:
When you submit a sponsor, a few things are stored. The data for the sponsor, the video id, and a hashed version of your user ID is stored. Your user ID is a randomly generated UUID generated by your browser. As well as these, a hashed + salted (5000 times) version of your ip address is saved. This is to be able to rate limit one ip sending way too much spam. Why is this hashed? This just adds an extra layer of security if there is a database breach. After hashing it this many times, it becomes extremely difficult to reverse it and find what your actual IP address is, but I can still verify that the current connection is from that hashed ip. By hashing it 5000 times and salting it, brute forcing would take longer to do.
As well as these, whenever you submit a sponsor, it tells the server and the server records one "view" on that sponsor. No IP data or userID data is collected for this. This is to make it so that you can see how many people you have helped and I can see how much the extension is being used. However, I perfectly understand why someone wouldn't want this data logged, so there is an option to disable this if you would like.
Above is not mentioned that we transfer the id of the video the user is currently watching to their services.
Here is a link to their privacy policy.
@TheAssassin I suggest to show a dialog similar to the one when submitting a crash report when enabling this feature in the settings. What do you think?
|
For this kind of tracking, a simple update to the privacy policy is likely not sufficient. We would first of all need a risk assessment ("Datenschutzfolgeabschätzung", EU-GDPR Article 9). IANAL, I cannot perform such an assessment. Also, this might be a commissioned data processing requiring a contract. Even if all that was provided at some point, this probably couldn't be opt-out, but would have to be strictly opt-in. I think the amount of data that needs to be sent to this service, and given the fact the service doesn't provide the essential privacy documents (imprint, privacy policy compliant with the EU-GDPR) makes all my privacy alarm bells ring. We need to discuss this internally and evaluate whether it's possible or desirable for us to merge this pull request. Note for everyone: Please try to get in touch before implementing anything with a relevance for data protection or privacy. |
|
Hey all thanks very much for the feedback - I'm still learning my way around things. I'll be sure to make changes soon. @TobiGr @TheAssassin I should have taken a closer look at the privacy policy. EDIT: actually it sends the video id for both submitting sponsors and requesting sponsors (duh)... not sure what we want to do about this... |
|
I have had a look at that "privacy policy" @TobiGr linked to. Not only it isn't linked to anywhere on the website, it does not comply with EU laws at all. I don't think this service can just be included in this app. |
|
In this case we might need to contact them first and ask them to get a correct privacy policy. |
|
You can download the sqlite database and use it locally. This would not require sending any personal information. The only downside is that the data is only as recent as your last download. |
|
Hm. That are 33 MB. That's too much for mobile phones. Maybe we can host our own instance and update the database every hour or so. What do you think @TheAssassin? I am at university atm. Let's have a discussion about this in our IRC channel in the evening. |
|
Also, if you have suggestions on the privacy policy, feel free to tell me. I thought it was gdpr compliant. |
|
@ajayyy thanks for reaching out. IANAL, but I'll create a list of the most important problems I see later. @TobiGr hosting an API server is problematic, too, because of the data we would be sent. Hosting the database file would be possible, and would also mean we'd be in compliance with the GDPR, as our privacy policy provides all necessary information (unlike the upstream). The most privacy-friendly way to implement this would be to download and cache this sponsor database and delta-update it every couple of days. |
|
It turned out that we only need three columns from the The idea is to set up a new db at our/@TheAssassin's server which contains the data mentioned above. That's ~3MB on disk. When SponsorBlock is enabled, that data is downloaded once and synced after that either once a day or when viewing new videos after an hour using zsync or something similar. A different approach could be a new plugin-like app which provides relevant sponsorBlock data to other apps. This could be either NewPipe, SkyTube or other services. @gzsombor Are you guys at SkyTube interested in integrating SponsorBlock locally? |
|
30 minutes should be fine if it's just the server fetching it and not every individual client. |
|
@TobiGr : thanks for asking and notifying about this interesting development! I would like to have this implemented in SkyTube too. If you could implement the API in NewPipeExtractor - probably in a new module, that would be awesome. About your privacy concerns - yes, it's problematic, but downloading ~3 MB file, every day, wont be fast, and lightweight. Even if we only do on wifi. The other problem which I see, is if you host that db dump on your server, then the question is, how could the user report new sponsor blocks? Calling the SponsorBlock API directly? Or calling your server, which forwards and anonymize the request? Both cases, there will be some latency, when the user submits a block, and another user could receive - and enjoy - that information, due to the need to re-download the db dump. |
|
This is how you could remove the videoID from requests while not storing the whole db: ajayyy/SponsorBlockServer#25 I haven't implemented it yet though. This still isn't perfect, as the server still knows a list of possible videos. |
|
@gzsombor The idea is to download the db once when sponsorBlock was enabled in the settungs and use zsync or a similar algo to only fetch the deltas to keep sync costs low.
I am not sure where we place that functionality. It should be in the extractor, but I don't know where exactly. My first idea was to put the sponsor blocking (SB) info into the
@TheAssassin and I came to the conclusion that reporting new sponsor blocks is hard to realize for us without having a lawyer who'd write a proper privacy policy and risk assessment as people transport personal and harmful data with the video id. Sending hashes instead of video ids sounds like a good idea and could solve legal problems partly. @TheAssassin What do you think about that? The idea from ajayyy/SponsorBlockServer#25 is to send the first digits of a sha or other sum to retrieve a set of video id's from the server when asking for sponsor blocks. Submitting only the sha of the videoId, start and end time for new SB blocks might be also GDPR compliant. What do you think? |
|
I dont see how this feature invades the user's privacy any further than requesting the content from Google/SoundCloud/CCC/Framasoft in the first place and those aren't mentioned in the privacy policy either. Of course it's an additional invasion that would be unnecessary if the user didn't want the feature, so opt-in is appropriate but since this feature will probably be seen as unethical by a significant portion of users, that would be a good idea anyways. FYI, from the GDPR:
IANAL either but I don't think NewPipe can be considered to be a commercial application and we're not the ones processing the personal data either. If we hosted the db and made our users send personal data to our service I think we could reasonably be considered a processor but @ajayyy's hosting it and they had no connection to NewPipe whatsoever prior to being asked about privacy concerns with their service afaik. k-anonymity would be awesome but, as long as the user is made aware that their entire activity is sent to Sponsorblock's servers in clear-text if this feature is enabled, I don't think we should wait for that to be implemented. |
|
Just because NewPipe doesn't host it doesn't mean NewPipe isn't responsible. This is a subject that needs to be 100% confirmed that it doesn't require changes to the policy before merging. Also, I'm personally against this. Sponsor reads are about the best form of ad possible for everyone involved. The viewer can easily skip it (and even gets their favorite YouTuber to read it and make it funny) and the money goes directly to the creator, instead of a bizarre algorithm. Having these ads doesn't harm the privacy of the viewer, which is why I and many others use NewPipe. I'd like ad reads to be present for as long as possible because they are the best solution. |
|
Also I don't believe that NewPipe by any means could be considered a personal or household activity. It's registered as something in Germany I believe(?) |
The privacy policy doesn't mention any of the other services we send data to either. That might be an issue on its own but not specific to the Sponsorblock feature.
That's fine but not an argument against the addition of an option, especially not an opt-in one.
From the Website's contact page:
But again, IANAL. |
|
It must have just been an idea thrown out at one point to register as something or other I am not sure. The rest I agree now. |
are videos accessed through Newpipe even counted in Youtube's view statistics? because if they don't, the content creator won't get any more revenue even if you watch the ad segments. |
|
I mean in the long run that companies will stop paying for the sponsored ads in the first place, not anything to do with the statistics they see. |
|
Can you update the version to .19.3 I really want to use this on my TV |
|
@lucaboox There is a Kodi plugin: https://github.com/siku2/script.service.sponsorblock |
|
https://github.com/Atemu/NewPipe/tree/sponsorblock-0.19.2 (rebase only, works well) |
|
@Atemu thanks for that! |
|
I think that if we removed the ability to submit timestamps and added a warning that by enabling Sponsorblock, a request with the ID of every video you watch is sent to Sponsorblock's servers in addition to Google's, there should be no significant privacy or legal concerns. The timestamp submission involves a user ID and permanently storing their IP which I think is a bit more of an issue for privacy, so it might be a good idea to split that off into a separate PR. |
|
NewPipe is only available on Android, so that's a huge chunk of the mobile market removed (without iOS). The rest of us are capable of making the opt-in decision for themselves. To think that dozens of us will make any noticeable dent in the advertisements market for youtube is just not tenable...
Edit: This is a second most discussed pull request, and the most discussed issue for the entire project. A lot of people seem to want it. |
I just gave some examples how this could hurt creators. Just because you're ignorant enough and miss all my replies here doesn't mean I want to repeat myself yet again. In any case: we based our decision not on whether we like sponsoring or not. We chose to remain neutral with regards to that, and base it on whether the technology respects privacy or not. I will leave the discussion now, as this doesn't lead anywhere. |
You misunderstand. |
|
I would like to use this opportunity to thank @polymorphicshade for his awesome work for all this time, I also would ask him if there's away to continue his work, that is if it's not too much to ask. Again thank you sooo much 🌷 |
|
So @polymorphicshade senpai, please tell us that you're forking the project and that releases from now on will be posted there? |
|
@Fatcakes yes I'll be maintaining my fork and posting APKs there from now on. |
|
To get this discussion into a more productive way: What do you think about a plugin system? Just like in your browser, where you could install that adblocker plugin (let's call it like that), it would (as far as I see/IMHO) solve all your problems. You could still stand by your opinion that you do not want to be a gatekeeper and do not want to integrate this feature as you only want to block privacy-invading technologies, but users who want more – like an adblocker in their browser, because they want to block ads (which is of course their freedom/free decision to use) –, anyone, who wants to get a different feature not useful for NewPipe (/the majority of users) in general, can just use a plugin. After all, it would be the most flexible solution IMHO that honors your decision/opinion of what an alternative YouTube client should do/behave like, while allowing other's to extend/modify it (and keep getting upstream updates). SponsorBlock could then integrate it and offer it as an additional APK or whatever. You do only have to maintain an API, not a whole big feature you do not like. Of course, this can be used for several purposes – whatever people want to do with YouTube e.g. there are plenty of browser add-on's for YouTube and (just an example) what if you want to automatically like videos. Things like that… In any case, this can solve the conflict that seems to be developing here, possibly avoid a fork, which would be a good thing IMHO, and just please everyone. Browsers do it like this/it's possible to do on desktops, so why should not it be like this on mobile too? |
Sponsors also have to take users skipping sponsored segments manually into account and I'm sure they already do. They know people can and will just skip them. YT collects data on what parts of the video viewers actually watch and exposes it to the uploader. This is the only way sponsors could get to know how many people watch or skip their segments. Why? I would've skipped the segments anyways. Again, all SponsorBlock really does is automate the skipping. If I wasn't skipping sponsorships manually already, I wouldn't have a reason to activate SponsorBlock. I think SponsorBlock therefore very much falls under "features we consider useful". *This is assuming NP users are even accounted for in this statistic and that uploaders share it with their sponsors.
I have a feeling this is a much larger amount of work than you think it is. A plugin system for NP would probably have to rely on an unstable ABI and at that point the plugin maintainer has the same burden as a fork maintainer. In any case, I think that's a discussion for a separate issue/feature request. |
@rugk Such dynamic feature would be an overkill of implementation compared to its benefits so I'm definitely sure that no developer here who would consider spending time on it even if it sounds like a cool feature. Maintaining a fork like polymorphicshade has promised is therefore the more secure and inferior variant. @ all: If you like the feature and want to use it, stick on his version. Super easy. There is no need to implement it in the official version, you'll get the same experience like the official version except of the additional SponsorBlock feature since he keeps it uptodate and will probably release a version on the fork's release page whenever NewPipe releases a new version. |
Jeez, I just asked a normal question and didn't insult you in any way. I even said sorry in case it was already answered. Why the hostility? This whole PR seems to be incredibly emotionally charged. |
Is there an "official" fork already? (except of this PR and test APKs) |
|
@rugk Here: https://github.com/polymorphicshade/NewPipe |
|
Ah, sorry missed @polymorphicshade's reply here:
If so, then you should IMHO enable issues in your repo and possibly also publish the APK on F-Droid. |
|
I agree with publishing onto F-Droid, or at least setting up a custom repository (as F-Droid takes a while to update, and on the occasions Youtube breaks Newpipe, it can take a week or so before F-Droid pushes the update) Again, shame we can't just have sponsorblock in Newpipe without needing to fork. I see no reason why most people wouldn't want the option for sponsorblock quite frankly, but I am happy we have the fork thank you Maybe this can be merged in with Newpipes existing custom f-droid repo? |
|
Has it been confirmed if Newpipe views count on Youtube? |
To make the views count, NewPipe would have to send a request to the watchtime endpoint. |
If newpipe's views dont count in youtube, whats the point in allowing sponsors ?, creators get paid depending on how many views the video gets. If you want to support the creators, you shouldn't use newpipe in the first place |
|
i wrote earlier #3205 (comment) and since then many things have been written and i am putting some more views forward. coming to implementation of sponsorblock, there should be absolutely no difference between an ad on a newspaper and skipping a sponsored ad in a video and skipping ads. with a traditional newspaper, all the advertiser gets is "this paper has x circulation and is sold in x demographic'. thats it. they dont know how many people looked at a particular page, how long they viewed it, did they act on it (indirectly) and yet everyone is fine with that. why should that be a problem be on newpipe which is used by a tiny tiny internet population ? second. there were talks about "giving positive signals about which sponsors your favorite youtubers have". this is a half baked idea. anyone who uses newpipe has NO direct way of contacting and interacting with a youtuber because there is no chat, no likes/dislikes, no feedback at least directly from the app. i subscribe to simply and rss feed of a youtuber. a one way street. there is no feedback. about making a positive impact on the community, how about start giving peertube a more first class treatment because that way if a youtuber is losing money with youtube, they would be compelled to join peertube because the positive feedback would be better. to that end, i have already suggested a feature to bring "support" button on peertube #5031, video intent #4853 and "background play" #3671 and sepia search #4470 and adding views #3092. i hope, with improving peertube, one day newpipe might default to peertube first and youtube second because the community might find it more engaging and creators more fruitful to use peertube. |
I disagree -- creators offer a quantum of supporting options (e.g. donations, merch, Patreon subscription) and its uncommon to have no method of any more or less direct payment. |
|
Oh shi here we go again. @TheAssassin already gave a big no on implementing this feature. It's very difficult to convince a stubborn person otherwise. The only options is as someone commented to have newpipe forks like NewPipeAdv, NewPipeX, etc with these (and other features) that get rejected Well... I could be wrong and we could somehow convince the maintainers to include this patch on official newpipe, but tbh my expectations are low |
I was talking in the context of youtube adverts |
Absolutely. I have Patreon subscriptions to creators I watch a lot. That's a lot more valuable to them then whatever small amount of ad revenue they might (or might not) receive even if I watched the video directly on YouTube without ad blockers. |
@polymorphicshade That's good to hear. Again. Sorry for rejecting your PR that late. When putting your fork on F-Droid, can you please change the name to something other than "NewPipe", so there is no confusion about which version is doing what. |
Would "NewPipe Sponsorblock" be distinct enough? |
|
Not quite. This still implies Team NewPipe has something to do with this. I'm sure you'll find a better name Also, please move your discussions to that repository, so you don't generate all these notifications any more in this repository. |
I recommend just using this name and the Newpipe logo unless the Newpipe license forbids doing so. EDIT: You can always rebrand later. |
|
The idea that this decision to not include SponsorBlock is somehow a "neutral decision" is laughable. For those who don't know, NewPipe already blocks ads. If @TheAssassin truly cared about "creators being hurt," he wouldn't be contributing to an app that facilitates circumventing YouTube's ad system for content creators. The hypocracy here is laughable. Imagine letting your app become overtaken by a fork over this xD |
|
While I don't 100% agree with this decision either, @TheAssassin made it clear that they will not implement this multiple times now. I don't think there's a need to discuss this further and try to convince the maintainers otherwise, especially since @polymorphicshade already said that they will continue to maintain a fork with SponsorBlock implemented. |
|
sponsorblock functionality should absolutely be included in newpipe. A fork shouldn't be necessary. |
A simplified implementation of SponsorBlock (https://sponsor.ajay.app/about).
Closes #2554
[✓] I carefully read the contribution guidelines and agree to them.
Test APK: newpipe_0.20.4_sponsorblock.zip
Previews