GCM -> FCM Update needed for push messaging

[chris42]

---- EDITED 26. August for summarizing the history of original issue and needed solution

Google has updated the push messaging infrastructure to FCM and will retire the old setup called GCM in 2019 (https://firebase.googleblog.com/2018/04/time-to-upgrade-from-gcm-to-fcm.html).

Hence apps are moving over to use FCM and stop working with current GCM features of microg. This results in different app behaviors. Some apps do not receive any push messages anymore, others (like Whatsapp) fall back to a polling mechanism, that uses lots of battery.

To use updated apps again with microg, the GCM functionalities mimicking google service framework will need to be updated to FCM.

---- Original Issue that lead to missing FCM functionality issue
Hi there,

I have a strange behaviour on my phone with GCM on AOSP Oreo. Some apps can register for GCM, even push notification tester works flawless. However Whatsapp and Threema won't register for the service.
I regularly get the message "GmsGcmRegisterSvc(6107): registration not allowed".

So far I have the feeling, that the "ask for new apps" function is not working properly. As it was enabled in the beginning (When installing Threema and Whatsapp the first time), but not anymore.

Is there a way to clear the list of registered apps and start over? This is just a hunch now and want to test this, before digging into other things.

Regards
Chris

P.S.: If you have any other idea, let me know. Can also produce some logfiles.

[chris42]

So I wiped the phone and did a complete reinstall. Same behaviour as before. Apps can register, push notificaton tester and other apps work.
However Whatsapp or Threema still do not show up in the registration list and do not receive messages via GCM. Within Threema I can reask the registration with GCM, however it receives a timeout.

Any idea? Any logs needed?

[MartinX3]

WhatsAPP won't register at my side.

[chris42]

So I checked with an older version (~mid 2017) of Whatsapp (2.17.223) and that will register fine with GCM.

So it seems that something has changed in the last versions and will need some adaptation in GMS Core.

[ArchangeGabriel]

I don’t know a lot of things about this, but maybe they moved to FCM instead of GCM.

[MartinX3]

I updated WhatsAPP and since the last update, it registered fine.
Maybe, it was because TitaniumBackup Restoration or was there a bug in a previous WhatsAPP version?

[tomasz1986]

Restoring an app with TitaniumBackup is known to break its GCM, unless the app is capable of re-registering itself. The app's GCM connection may still work though if everything has taken place within the same Android installation, but if it is a different one or a different ROM, then it will always break (as the old token is no longer valid).

[chris42]

Not sure about FCM. I am not a developer, but it is at the core GCM. They will have changed it, but can't tell to what extend.

I do not use TitaniumBackup Restore and have this phenomenon also with Threema.
So far left options are: bug in app or needed adaptation in microg.
Will check if older version of Threema solves this as well, then it should be needed adaptation in microg. (As I can't imagine Threema and Whatsapp to share code)

[chris42]

I guess the issue gets more pressing now, as e.g. Whatsapp does not allow you to install older versions. It forces you to update before registration.

Hence the question: Is microg dead? I haven't seen updates aside from the google service number for a year now and more and more things are getting unusable.

I know about open source developers and "don't bother them, as they are doing it in their free time". But I am not asking to finish this at a certain time, more for a clear status on who is actually still developing this or if no one finds time?

[Ra72xx]

I'd like to ask that question, too. I'm grateful as for several years MicroG has allowed me to be almost Google-free, but I just now try to setup a new smartphone, and I do not longer get MicroG to work. As there is not too much activity on GitHub, it would be great to know if there is a fix in work or if it's time to switch back to Google :-(. As chris42 says, all the usual caveats concerning oss developpers apply ;-).

[chris42]

I changed the Title to something more meaningful and added a $150 Bounty on Bountysource.

After numerous other issues referenced and talks about it in the forum. What is essentially needed to fix this issue:
Rework of GMSCore to update GCM push messaging to FCM. I would consider it solved by the following:
Current functionalities re-enabled

• Registering apps in Microg with current functionality of whitelist should work again
• Message counter as usual, to verify, that messaging is working
• Push messages are delivered on time and reliable
• Message notification is working as usual

I don't think keeping backwards compatibility makes sense, as GCM is to be deactivated. Not sure if a transfer to FCM is feasible for old GCM (not updated) apps?

[ale5000-git]

If I'm not wrong FCM is just an extension of GCM (I'm not sure) so backwards compatibility is implied.

[chris42]

Yes and No. FCM apparently has a lot of GCM in its core, however the GCM API and servers will be deprecated in April 2019.
See here https://firebase.googleblog.com/2018/04/time-to-upgrade-from-gcm-to-fcm.html

[pikatchu2k3]

Nice idea with the bounty starting for fixing this problem.
I will add some bucks too, after the holidays.

[dos1]

Note that client SDKs and GCM tokens will continue to work indefinitely. However, you won't be able to target the latest version of Google Play Services in your Android app unless you migrate to FCM.
(https://developers.google.com/cloud-messaging/faq)

So, basically, microG has to reimplement its push messaging system to use Firebase servers and to add a client-side Firebase API alongside the existing GCM one, so both old and new apps can still work.

[chris42]

@dos1
Yes pretty much. The GCM part could also be put into an extra proxy app. I think this was done with c2dm as well, when GCM was introduced.
The developer would need to see if this is possible and worth to put effort in. First focus should be to adapt FCM API and handling of FCM Servers as this is missing for a lot of people.

[Ra72xx]

Stupid question: Who is the said developper? What's the matter with Marvin? I reverted to GApps because MicroG would no longer work properly, but I eagerly watch the developments here.

[FOSSFOREVER]

This project is very promising. Thanks to all the backers and developers for this. I'll eagerly be watching the developments, too!

[kwokkeunglam]

anyone already started working on it?

[chris42]

@Larx
"The developer" is any developer interested to fix it. Payment is done via bountysource.

@kwokkeunglam
"denim2x" just set a bounty goal on bountysource (https://www.bountysource.com/issues/51518216-gcm-fcm-update-needed-for-push-messaging). This does not necessarily mean he started, but apparently he is interested.

Happy to help anyone who is looking into this.

[kwokkeunglam]

I prefer a new way of push, just teach developers to use it.

Save battery + Rich content , yet still free.

anyone support this?

[chris42]

@kwokkeunglam
Could you please clarify what you mean?

[dropicallus]

As mentioned GCM will be replaced by a firebase implemention until feb. 2019.
I also think that would be a show stopper of using this project. But from a java developer perspective I think it could be a hard work to find informations about the new apis. Especially if google is still extending their cloud service infrastructure. A second problem is that this project has still only one maintainer, which was still mentioned in the other discussions. But that will slow down the development in the future.

[chris42]

@kwokkeunglam and @dropicallus
If your idea is to create a new push service aside GCM/FCM, this is admirable, but way out of scope of this issue and project.
Microg was created to mimic the Google APIs, so apps that are utilizing Google APIs would run on a phone without the Google framework. To now create a new push system, you would need a new API, infrastructure and convince every app out there to use it.
Feel free to setup anything to do so, but please understand that this issue and bounty is about FCM within the Google API.

[kwokkeunglam]

@chris42

1. a push notification ecosystem completely independent of google

2. I have a plan to use UDP, each message have bytes (8 byte header + 65,527 bytes of data) and fit inside one datagram. ( * fit inside one datagram means you either receive the whole message or you don't receive it. )
   after some data used for protocol , such as QoS, resend, deduplicate, encryption, signature, etc;
   still have around 60000 bytes available for use.

( encryption is a must as UDP do not have it, need to do by ourselves )
Encryption and sign can be done using developer's certificate, like APNS.

There can be feedback UDP message send back to server after receive, or the message will try to resend again.

3. the foot-print shall be small enough. Works both as an embedded service or as a central service.
   The SDK will ask user to install a push-service from F-Droid / Any store;
   If central service is installed, all apps share one UDP connection to backend, which can feature save energy.
   Otherwise, the app can start embedded service to receive push internally.

UDP socket itself is quite energy efficient.

4. Cost & Developer acceptance:
   I am not a business man, but are we using letsencrypt for free?
   I am quite sure the cost is lower than running letsencrypt.

Push only require some server farm to run, no need to buy a C.A. cert.

[Mafus1]

I freshly installed Lineage OS for microg 15.1 with GsmCore Version 0.2.6.13280 and i checked that:

• google device registration is active
• all the green checkmarks in microg diagnostics are there
• nothing is blocking network traffic to mtalk.google.com (e.g. afwall, adaway, ...)

But never the less I can't get any app to register to Google Cloud Messaging. I even checked with notification tester but it fails to "Register for Push Notification". Any idea what might cause the problems? I attached a log file. push_notification_tester_log_1.txt

EDIT: today out of the blue, everything seems to work just fine! All the apps registered correctly. Maybe it took some time for the google device registration to go through?

[karcherm]

I got the symptoms you described until I added a GMS Google account in the android settings. I don't remember whether I had to reboot after that for push notifications to work.

[mar-v-in]

There seems to be an issue with automatic device registration in latest version. If you have similar issues as described by @Mafus1, open the system phone app and dial *#*#2432546#*#* to do a manual device registration.

[Mafus1]

I didn't have to add a GMS Google account like @karcherm said, but maybe this works for others. If I have the Problem again, I will try what @mar-v-in suggested.

[WolfangAukang]

I was in the same condition as @Mafus1 (LineageOS 15.1 with microg 0.2.6.13280, clean installation). Followed @karcherm strategy with a reboot and now GCM is connected.

Excuse any formatting issues, did this on mobile.

[dschrempf]

I can confirm that I had to do a manual device registration with the code provided above on LOS 14.1, official microG 13280.

[wooque]

I have LineageOS 15.1 with microG 0.2.6.13280 and had to dial code that @mar-v-in suggested to connect Google Cloud Messaging

[NvrBst]

I think there might be some minor FCM logic missing from the latest version in relation to gms checkin. I can tell using the "Push Notification Tester" app on microG's download page that v2.6 of the tester app does not work on clean installs (it does work after a device registration succeeded), but v2.5 works fine after a clean install. In the changelog it says v2.6 is when the author added FCM support.

I think microG does a checkin automatically when a GCM token is requested, I think maybe this is being skipped with FCM token requests? Else, if fcm tokens don't occur through microG, maybe make it so it'll always attempt a gcm checkin (if needed) whenever something in the gcm pipline happens (connection, registration, etc).

[ale5000-git]

Related to #313

[ghost]

This is obviously not fixed. Why was a bounty paid out and this issue closed?

[chris42]

@lifranc
Please read the issue thread, the comments of the backers and observe the timeline.

[ale5000-git]

@lifranc
Most FCM apps are now working (which is what this ticket was about).
For the other different or new problems you need to open a different ticket.

[BurhanBudak]

Hi is this thread open for discussion, as you know RCS and Jibe doesn use FCM but uses Play Services to register RCS. I kind of did activate RCS by backup up a app from a Google enabled devices and transferred the zip to a LineageOS phone. Now I have RCs enabled with MicroG but the service doesn't use FCM.

[ale5000-git]

@BurhanBudak
This should be fixed from a lot of time; if you still have problems please update microG to the latest version and open a new ticket.