Set up own F-Droid repo to push urgent updates to users

[AltNico]

At 35C3 there was a F-Droid meetup where @theScrabi showed up and explained your need at NewPipe to sometimes quickly push updates to F-Droid. This is totally understandable and we at F-Droid would really like to offer quicker updates through our infrastructure, but at the moment we can't improve publication time without reducing F-Droid's security standards.

To give some insights: as you all know, at F-Droid everything gets built from source. Unlike Google Play, this means that the building process isn't decentralized at developers' infrastructure, but we rather have a central place where everything is built. This is a great target for people to infiltrate F-Droid's infrastructure and get onto hundreds of thousands of people's devices. Therefore, we have quite high security standards here and run builds only on fully trusted hardware, maintained by fully trusted F-Droid members.

There are plans to decentralize F-Droid's build infrastructure to let builds run on multiple semi-trusted servers run by semi-trusted F-Droid members, but those plans require a lot of work and aren't reachable within the next months. Anyway, what's needed for them are apps to be built reproducibly. So most of the work proposed in this issue needs to be done anyway when F-Droid starts to offer quicker updates.

What I propose to NewPipe (and any other app that wants to distribute quicker updates for apps installed through F-Droid) mainly consists of two parts:

• Set up a F-Droid repo where you publish official builds of NewPipe that can be reproducibly built by F-Droid
• Include F-Droid's update channels library into NewPipe to inform users about urgent updates

Let me explain this a little bit further.

People around F-Droid worked hard to make reproducible builds possible. What this means is that F-Droid tries to build an app and then compares this build with the app provided by upstream. If those builds match, F-Droid publishes the upstream .apk. As of now, there are two reproducibly built apps in F-Droid: Öffi thanks to @schildbach's and Briar thanks to @grote's work.

This alone doesn't solve F-Droid's slow publication time yet. It still needs the same amount of time to publish apps, in some cases it could even take more time if there are problems with apps' reproducibility.

However, one important aspect changed: the apps are now signed by upstream, meaning upstream can publish updates which can be installed seamlessly by users.

Here, F-Droid's update channels come into play: if F-Droid for whatever reason needs too long to publish an update or upstream wants to publish an urgent update almost instantly, that update channel library can be used to notify users about an available update that's so far not yet in F-Droid's official repo, but which can be installed from upstream's repo.

The implementation and UX would be like this:

• On the app's first run, users can opt-in to get notified by upstream about urgent updates.
• The app then regularly checks if there's an urgent update available by upstream. This can be a simple JSON file inside NewPipe's source code repo which contains the version of the app that every user should at least have.
• If that version is newer than the version installed on the user's device, F-Droid's update channel runs and notifies the user about an update. The user can then click on that notification which results in the update being downloaded. It can then be installed with another click, opening Android's package manager. The UX is the same like inside F-Droid's app, only that it's inside NewPipe and doesn't touch F-Droid at all.

Be aware that implementing F-Droid's update channel could result in the app being kicked out of Google Play. This isn't a problem for NewPipe but I state it here for everyone else reading this. Even if the mechanism is deactived, an app that includes update channel's code violates Google Play's guidelines. For example, Öffi got kicked out of Google Play for including a deactived code which would show a donation button to its users.

By using F-Droid's update channels, you also get another feature for free: when people install .apks from F-Droid without installing the F-Droid app, they normally don't get updates for it. With that library, they would get notified about updates and could install it conveniently with some taps on the popping up notification.

Of course you can also use this F-Droid repo of NewPipe for more cool stuff. People can add it to F-Droid like any other repo and you could push beta and nightly builds to that repo, with those builds automatically checking for updates and proposing them to users all the time.

Some more interesting links:

• At verification.f-droid.org you can see if your app is already reproducible and what's missing in case it's not. Just search for NewPipe and watch out for the build's date.
• The build scripts of Briar and Öffi. Note the Binaries entry and the use of disorderfs at Briar's side. This is an essential hack because as of now, it's not possible to build Android apps reproducibly with standard Android SDK tools! (This was reported to Google several months ago, but so far they didn't react.)
• On GitLab F-Droid's nightly repo is hosted.

The F-Droid documentation also contains a lot of useful information:

• Setup an F-Droid App Repo
• Build Server Setup
• Reproducible Builds
• Verification Server
• Publishing Nightly Builds

I'm CCing @eighthave and @Bubu as they were also at the meetup and possibly know more about my proposed stuff. Also, so far nobody has done this. I'm proposing it to NewPipe because it doesn't have that problem with being on Google Play and could serve as an example app for other apps. This is of course also interesting for Conversations, where @iNPUTmice attended the F-Droid meetup, but they have the problem of being on Google Play and so far don't publish official builds outside of Google Play.

---

• I carefully read the contribution guidelines and agree to them.
• I checked if the issue/feature exists in the latest version.
• I did use the incredible bugreport to markdown converter to paste bug reports.

[schildbach]

Thanks for this extensive writeup! I'd like to add that Bitcoin Wallet is reproducable as well and it's interesting for the fact that it uses the backwards-compatible way with dual signatures. Existing users will get updates signed by F-Droid; new installs will get the developer signed version.

The configuration metadata can be found in this directory: https://gitlab.com/fdroid/fdroiddata/tree/master/metadata/de.schildbach.wallet/signatures

[eighthave]

@uniqx and I did a lot of work to get the multi-signature support working, happy to hear that it is appreciated

[licaon-kter]

This is the right time...

[TheAssassin]

We've set up a demo repository a couple of hours ago, based on the binaries from the GitHub release page. Of course, those binaries don't have the same quality as the original F-Droid ones, therefore I plan to import the existing F-Droid builds and cross-sign them as suggested by @schildbach. Eventually, we might even set up a continuous integration system on infrastructure and build proper binaries from the metadata using the F-Droid tools.

Infos will follow soon!

[eighthave]

One easy way to start with a repo would be to setup an automatic nightly build repo based on travis/github Pages or gitlab-ci: https://f-droid.org/en/2017/12/11/publishing-nightly-builds-with-gitlab-ci.html You can also easily host fdroid repos on github/gitlab Pages, while generating the repo anywhere. https://gitlab.com/fdroid/fdroidserver/blob/master/examples/config.py#L186

[theScrabi]

@TobiGr you where interested in this topic. Her would be the thread to go :)

[TheAssassin]

The repo URL I mentioned last night is https://archive.newpipe.net/fdroid/repo. We've tested it a bit, and it seems to work. Right now, it only provides the latest GitHub binaries, and you'll have to remove the old version to see the new one. We'll try to make sure that from then on, we also provide F-Droid binaries and ensure smooth upgrades by cross-signing those packages and including them in our repository. I guess I'll bother @eighthave once I find some time to set this up.

The repository is somewhat more secure than a completely automated one, as I update it manually, and the builds I want to include from F-Droid will be signature-checked before inclusion. Also, old GitHub builds will be dropped as soon as "better" versions from F-Droid are available. The repository is running on private infrastructure (i.e., we control it).

[TheAssassin]

@theScrabi can you please investigate whether we can make use of https://gitlab.com/fdroid/update-channels to allow users to update more quickly next time?

[eighthave]

Right now, using Update Channels, you can make NewPipe get updates from your repo, then prompt the user to install it. I think we'll need to do a small update to Update Channels to fully address your use case. It seems to me that Update Channels should have an option to prompt the user to add the NewPipe repo to F-Droid, if F-Droid is already installed.

[theScrabi]

We just merged the app updated notification PR. This one has the purpose to notify the user about new releases of newpipe at the release page of github. So I think with this we already have quite a need solution. While Update channels will definitely be an improvement, I think our main goal atm should be to get reproducable builds running.

[TheAssassin]

Reproducible as in "exactly compatible to F-Droid binary-wise"?

[TheAssassin]

@theScrabi I think the update channels lib is worth implementing even if we will provide "only" one channel for now. The notifications are more reliable, as they are based on the F-Droid repository. As @eighthave said, it could use our own repository.

[yausername]

@eighthave
I can't find recent newpipe releases on https://verification.f-droid.org
Are these verification builds not getting triggered for newpipe?

[eighthave]

the buildserver instance on verification was stuck, so I kicked it. Let me know if it still isn't working.

[yausername]

I still can't find builds after versionCode 69. Is https://verification.f-droid.org updated via some jenkins job?

[TobiGr]

It looks like the latest build was verified. This means, we can proceed to set up our own repo. We need to upload the latest binaries and update the indices.

[eighthave]

If you want to do the dev signature in f-droid.org via reproducible builds, then you'll need to make sure you sign your builds with v1-only signatures. We don't yet support v2/v3 signatures via the process where you provide the signatures (we welcome contributions though!). FYI, you can subscribe to all verified builds here: https://verification.f-droid.org/verified.json

[opusforlife2]

@TobiGr What work is left to do here apart from what eighthave mentioned?

[opusforlife2]

Done, finally. :)

[TobiGr]

https://newpipe.schabi.org/blog/announcement/f-droid/pinned/f-droid-repo/