Add support for Lineageos 17.1

[SolidHal]

This PR includes all the changes I found necessary to build lineageos 17.1.

The only thing that would need to be changed to make this fully functional would be updating the muppets manifests repos to have 17.0 and 17.1 branches. To work around this in my own builds I specified the muppets repos in local_manifests.

This also includes a commit that adds an environment variable to save the boot.img in the zips folder, as flashing this with fastboot is necessary as it contains the recovery image, and custom recoveries for android 10 are not yet mature.

[tobiasKaminsky]

Thanks for updating MicroG to 17.1 🎉
I checked out this branch, did "docker build ." and now when starting your cicd version I get:

 > > [Sun Mar  1 07:16:41 UTC 2020] SIGN_BUILDS = true and not empty $KEYS_DIR, but "$KEYS_DIR/networkstack.pk8" is missing

How do I get this? (I extracted the keys from my previous lineageos-16, so that I could update from lineage-16 to microG.)

EDIT: I used the one I found in your commit message: https://android-review.googlesource.com/c/platform/build/+/901518

[SolidHal]

@tobiasKaminsky Sorry, should have mentioned that a new key is required for signing the network stack. Generating just the networkstack key should fix the issue. Probably should add logic to do so.

[ghost]

Like every other Android 10 ROM, UnifiedNLP won't work on Lineage 17.1 because GmsCore doesn't request the necessary permissions. Solution is to use a fork, OmniROM has one for their microG builds (https://github.com/omnirom/android_prebuilts_prebuiltapks/tree/android-10) or if you want to build it yourself, https://github.com/NoGooLag/android_packages_apps_GmsCore has the correct patches. I'd recommend not building Lineage 17.1 until everything works.

[SolidHal]

@p4t44 I'll update the script to bring in the omnirom prebuilts for now. In the future it might be nice to add an option to build our own.
That said, everything besides unifiedNLP works on 17.1 with this PR. (and unifiedNLP still works pretty well, I've been using it for ~2 months now built using this docker)

[SolidHal]

@p4t44 the readme is now updated to suggest building the NoGooLag fork of MicroG instead of the prebuiltapk that it was suggesting, meaning everything is now functional

EDIT: In testing, I have found that building the apk in fails due to the many issues discussed here: microg/GmsCore#729
I'm going to see if I can get something working based off of the comments in that issue, when I do I'll submit a PR to https://github.com/NoGooLag/android_packages_apps_GmsCore
Otherwise I'll drop building it into the rom, and instead go the method of a prebuilt-apk.

EDIT 2: In attempting to rework the Android.mk, and not getting very far with it as I am still new to working with gradly I found this issue by mar-v-in microg/GmsCore#859 in which he discusses getting rid of the Android.mk file altogether as it break with every new version of Android AOSP...

So instead in the interest of having a working system I have decided to go the route of including the prebuilts from omnirom. These can be swapped out with prebuilts provided in https://github.com/lineageos4microg/android_prebuilts_prebuiltapks when the ones there have the Android 10 patches, either built ourselves or from mar-v-in when he has a chance to accept the PR.

@corna I can build and upload https://github.com/NoGooLag/android_packages_apps_GmsCore to https://github.com/lineageos4microg/android_prebuilts_prebuiltapks if you like.

[nikolowry]

Any ETAs on when this could potentially land now that LineageOS 17.1 has been officially released?

[rakshazi]

Hello, guys
Could you provide nightly/testing builds with this changes, please?
Some kind of "beta update channel", can't wait to use LOS 17 with MicroG :)

[JuniorJPDJ]

It's already forked, everyone can just build it himself basing on branch from this pull ;D

[Findus23]

When I tried to build again today, after a while all git fetches seem to be failing like this (with all kinds of github repositories):

fatal: unable to access 'https://github.com/LineageOS/contributors-cloud-generator/': transfer closed with outstanding read data remaining
LineageOS/contributors-cloud-generator:
fatal: unable to access 'https://github.com/LineageOS/contributors-cloud-generator/': transfer closed with outstanding read data remaining

fatal: unable to access 'https://github.com/LineageOS/android_prebuilts_tools-lineage/': transfer closed with outstanding read data remaining
LineageOS/android_prebuilts_tools-lineage:
fatal: unable to access 'https://github.com/LineageOS/android_prebuilts_tools-lineage/': transfer closed with outstanding read data remaining

Maybe github has an issue or has rate-limited my IP address, but I don't really have an idea. Cloning from the docker host works without issue and even small repos like https://github.com/LineageOS/contributors-cloud-generator seem to be failing.

Even adding RUN git config --global http.postBuffer 1048576000 (SolidHal#7) to the dockerfile and rebuilding the image, didn't change anything.

Maybe I'll start again in a few days with a fresh vserver and IP.

[yeshwanthvshenoy]

Can someone clear if we need these changes from @SolidHal to build for Lineage 17.1 or can we use the new MicroG APKs and build using master itself?

[DPTJKKVH]

Hal said don't build microG yourself for the time being. Grab the prebuilts from github or just download it via fdroid after flash.

[yeshwanthvshenoy]

Yes but since Lineage doesnt have signature spoofing. I am planning to build Lineage 17.1 for my device using the docker scripts here and the latest prebuilt microg. So my question is that do I need to base it off the scripts from Hal's repo or just fork this and change the prebuilts version to whatever and proceed as in this repo?

[kleini]

@yeshwanthvshenoy another possible solution is to use Nanolx. You start then with the plain LineageOS and Nanolx installs on top FDroid and MicroG as system apps and patches the necessary signature spoofing. I am still trying to get this working, some points still seem to be missing for me.

[Tutul-]

Does Nanolx patch signature spoofing for only system apps ?

[chilliger]

@DPTJKKVH I was able to integrate the latest microG release. The Android.mk file simply looks like this:

LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)

LOCAL_MODULE := GmsCore
LOCAL_MODULE_TAGS := optional
LOCAL_PACKAGE_NAME := GmsCore
LOCAL_CERTIFICATE := platform
LOCAL_SRC_FILES := gmsCore.apk
LOCAL_MODULE_CLASS := APPS
LOCAL_MODULE_SUFFIX := $(COMMON_ANDROID_PACKAGE_SUFFIX)

include $(BUILD_PREBUILT)

I did not add this overlay https://github.com/SolidHal/docker-lineage-cicd/blob/master/src/build.sh#L179
but still added the unifiednlp patch and signature spoofing patch.

[DPTJKKVH]

@chilliger Hey, thanks for letting us know!

I'm currently building a rom without microG to test what SolidHal has been proposing. However including a prebuilt gmsCore.apk in the build process is much more preferable. Does your build script automatically pull in the latest gmsCore.apk from GitHub or do you manually download the apk and point your build script to the local location? Or do you actually build gmsCore.apk from source despite marvin telling us that this isn't supported anymore?

Actually, would you mind creating a pull request including all the changes you did to make this work? I think there is more to it than what I can discern from your latest post.

[rugk]

Should be enabled by default IMHO.
Microg users likely want to use or at least be able to use UnifiedNLP.

[spolack]

I agree.

[projectgus]

@DPTJKKVH FWIW I just built with this PR and the latest MicroG release without any other changes apart from prebuilts (I think @chilliger is also using a prebuilt). The changes I needed are here: https://github.com/SolidHal/android_prebuilts_prebuiltapks/pull/2 EDIT: Merged into SolidHal's fork now.

Can build with this version right now by updating the local overlay to point to my fork (suggest checking my copy of the APK first of course...) Or remove the local overlay entry, make your own prebuilts directory with the changes you want, and overlay mount this into the src directory (skipping repo) by appending -v "/path/to/android_prebuilts_prebuiltapks:/srv/src/LINEAGE_17_1/packages/apps/prebuilt/:ro"

[DPTJKKVH]

@projectgus Hey! Thanks for the heads up! I will try to do a build like suggested. Thank you for making a pull request to solid hals repo!

I will report back here as soon as I tested it. (Build take up to 9 hours to finish on my crappy build rig. 😅

[spolack]

Could you please check, whether we can now use our prebuilts? I've updated GmsCore to latest.

[spolack]

@SolidHal Thank you for your effort. I'm glad to see people putting work into that project. Unfortunately you PR contains quite some unnescessary commits with changing things forth an back.

I suggest you to create a new Pull Request based on current master, only containing the nescessary changes. If not, i'll do so, but it will require 1-2 Weeks until i find some time. Hopefully we can bring up the builds soon again!

a new PR shall be created in favor of this.

[DPTJKKVH]

@spolack I will verify @projectgus suggestion today.

Build went through fine already but I must test location and notifications.

Will flash as soon as I'm done with my workload for today.

After that I will gladly help clean up this pull request.

[spolack]

I shifted my priorities and plan to work in the changes of the PR through by cherry picking and modify some things in a different fashion, but wont accept it as full "package"

Though this PR can stay open for discussion until i'm ready.

[DPTJKKVH]

@projectgus I wanted to give you my sincere thanks both for doing the pull request at SolidHal's repo and for addressing me directly and explaining things in more detail.

I was able to successfully build, using SolidHals (rather YOUR) prebuilts.

Both notifications and location are working fine, exactly like they did before the microG upgrade.

@spolack I'm looking forward to see how your repo will differ from SolidHal's. I pulled in almost all of SolidHal's pull request (minus the latest two changes) and modified them afterwards to my needs. I use way more RAM and cache for example.

[projectgus]

@DPTJKKVH No worries, glad it was useful.

BTW, I realised when I updated prebuilts I forgot to change LOCAL_CERTIFICATE := platform to LOCAL_CERTIFICATE := PRESIGNED, meaning that GmsCore probably can't be updated via APK from a future official release (need to build the whole image again or re-sign the APK with your platform signing key.)

Could you please check, whether we can now use our prebuilts? I've updated GmsCore to latest.

@spolack FWIW I checked over the prebuilts when I built this PR, and I think the only other "custom" one is MozillaNlpBackend which has been rebuilt with background location permission added in SolidHal/android_prebuilts_prebuiltapks@2ca2cb3 (this is the only change, based on dumping both APKs with jadx and having a quick peek.)

If it's helpful then I probably have some time this weekend to try building with lienageos4microg prebuilts, and I also don't mind trying to make a cleaner branch - but if you or @SolidHal is working on this then I'm just as happy not to. :)

[DPTJKKVH]

Just a small motivation for everyone working on this.

Surely I am the one with the least knowledge of everyone actively (and visibly) trying to keep this project alive.

Every time I update my DockerHub image there are many people who download it afterwards. I talk three figure numbers. I bet this is the same if not more for SolidHals and everyone elses public DockerHub images.

So there seem to be lot of (silent) people who are relying on us keeping this project alive to have a secure smartphone.

I hope this is additional motivation for everyone to continue doing their best. 😊

On a personal note: I grew a lot working on this project both in using git and building Android images in general. I want to sincerely thank everyone that has added to keeping this repo alive with special thanks to @SolidHal for laying the foundation for that and everyone who answered my questions along the way. I write this with a tear in my eye because this is my first true OpenSource community experience and this is what I always imagined OpenSource should be. I think we help a lot of people to keep their devices operational and secure especially in those times where many simply can't afford to buy a new phone.

Thank you for your time and work and thank you for taking me along and help me grow! I will try to improve further and use my new found knowledge to help here and in other projects were I can.

[projectgus]

I rebased this branch on the latest master, cleaned up the commit log a bit, and was able to build with the official prebuilts - everything seems to work as expected. My branch is here, can make a PR to this repo if that's useful. Thanks for all your effort making it easy to build lineageos4microg @spolack & @SolidHal 😁 (and appreciate the kind words @DPTJKKVH!)

[spolack]

Update: I sucessesfully created and tested some builds with the current master and now triggered a full build run for the current LineageOS roster. In a couple of days all images should be available again.

[spolack]

@SolidHal Please create PRs for other features i did not pulled in yet, if you still require them.

[DPTJKKVH]

Allow me one last question: I thought only Corna had "admin rights" on the microG Project. How comes that you can close pull requests and push merge stuff?

Where you given the permission in the past or did Corna decide to hand this project over to you?

Just wondering, I'm glad that someone motivated and skilled picked up the torch.

[spolack]

@DPTJKKVH There were multiple contributing developers and engineers to the LineageOS for microG Project.

Previously @corna took care of the actual android build process and the docker container.
@n1zzo crafted the website, which is still running.
I (former account was @nexxxus) spent my time mostly with running the underlay infrastructure, aquiring server ressources etc.

As you can see, such a project can be hardly a one man show.

Best
Simon

[EchedelleLR]

Sorry if this is not the correct thread but idk where should I report this. I tried in the website repo too but is a non-frequented repository, so I thought this would be a better place.

Just after the announcement that LineageOS 17.1 could be compiled and first builds were uploaded to be downloaded I saw that old LineageOS builds for smartphones whose LineageOS support was abandoned time ago had dissapeared.

I considered LineageOS for microG also what LineageOS was not, an archive of last builds for smartphones with abandoned support.

Is the current situation going to change in the near future?

[brevilo]

Update: I sucessesfully created and tested some builds with the current master and now triggered a full build run for the current LineageOS roster. In a couple of days all images should be available again.

@spolack Quick question: is there a plan to return to regular (weekly?) automated builds?

Cheers for all the hard work guys!