upload / download never finishes, sometimes with a crash -> update to v43.2

[mcliquid]

EDIT by @matkoniecz : see #4028 (comment) for details what is going on.

See "edited by" above to see original version.

---

How to Reproduce
I have the App on manual upload and when I try to upload it crash's. In Wifi and in 4G.

Versions affected
Latest version 43.1 from GitHub and also the Google Play version

Crash report

de.westnordost.osmapi.common.errors.OsmBadUserInputException: Bad Request (400) - Bad OAuth request. at de.westnordost.osmapi.OsmApiErrorFactory.createError(OsmApiErrorFactory.java:40) at de.westnordost.osmapi.OsmConnection.handleResponseCode(OsmConnection.java:349) at de.westnordost.osmapi.OsmConnection.makeRequest(OsmConnection.java:195) at de.westnordost.osmapi.OsmConnection.makeRequest(OsmConnection.java:148) at de.westnordost.osmapi.map.MapDataApi.getOneElement(MapDataApi.java:289) at de.westnordost.osmapi.map.MapDataApi.getWay(MapDataApi.java:271) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataApiImpl.getWay(MapDataApiImpl.kt:94) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditUploader.fetchElement(ElementEditUploader.kt:36) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditUploader.upload(ElementEditUploader.kt:21) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditsUploader.uploadEdit(ElementEditsUploader.kt:54) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditsUploader.access$uploadEdit(ElementEditsUploader.kt:26) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditsUploader$upload$2$1$1.invokeSuspend(ElementEditsUploader.kt:45) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditsUploader$upload$2$1$1.invoke(Unknown Source:8) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditsUploader$upload$2$1$1.invoke(Unknown Source:4) at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89) at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165) at kotlinx.coroutines.BuildersKt.withContext(Unknown Source:1) at de.westnordost.streetcomplete.data.osm.edits.upload.ElementEditsUploader$upload$2$1.invokeSuspend(ElementEditsUploader.kt:45) at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) at kotlinx.coroutines.internal.LimitedDispatcher.run(LimitedDispatcher.kt:39) at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:95) at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)

[mcliquid]

Tried also to relogin. Seems to be the OAuth Server?

[westnordost]

In a nutshell

Fix this by updating to v43.2.

The update is now posted to Google Play, for F-Droid it may take up to a few days until the update appears.

On v43.2, you need to log in again. Don't worry, your edits are safe and the statistics (star count etc.) will be synced shortly after you log in again.
In the meantime, while you wait for the update to appear, you can log out. Then, you should be able to use the app as normal, i.e. the download should work as normal and the edits will be saved locally until you log in.

What happened?

So, this was totally my fault. I accidentally deleted the OAuth profile for StreetComplete on the openstreetmap website instead of editing it for an upcoming feature. There is no way to undo this as far as I know. I created a new profile which will be used for v43.2 onwards.
The OpenStreetMap API will return a HTTP error 400 when a user tries to communicate with the API with an unknown (because: deleted) OAuth profile and the app does not handle that properly. I am not sure what exactly is causing this "endless upload/download", it could be just a security feature of the OSM API that the API will take a really really long time to respond when there is an app with spurious OAuth details. No matter the exact reason, the solution is to force logout all users and let them login again with the new OAuth profile.

Thanks to @matkoniecz for helping me to find, debug the issue and test the solution. On a Sunday, no less!

[mcliquid]

@westnordost Thank you for taking care so fast! With the new version it work again.

[MagicLike]

I actually installed Streetcomplete today for the first time via F-Droid, but I am not able to update via the apk nor via the PlayStore (Aurora Store). I can install the apk, but the version number doesn't change & the bug ist still there. On the Aurora Store I get the error message "INSTALL_FAILED_UPDATE_INCOMPATIBALE: Package de.westnordost.streetcomplete signatures do not match previously installed version; ignoring!".

[Helium314]

see

for F-Droid it may take up to a few days until the update appears.

You cannot update from other sources if you installed the app from F-Droid because F-Droid uses different signing keys. You either have to wait, or uninstall and install form a different source.

[MagicLike]

@Helium314 Ok, thank you! Didn't thought of that...

@westnordost What about making a StreetComplete F-Droid repo, which updates with every new Version immediately?

[matkoniecz]

What about making a StreetComplete F-Droid repo, which updates with every new Version?

Releases directly from @westnordost are already provided in https://github.com/streetcomplete/StreetComplete/releases

F-Droid compiles apps on their own, and it adds their own delay. Sometimes days, sometimes weeks.

App store has own process where they verify app and sometimes flag it and delay release for nonexisting or poorly explained reasons.

[daPhipz]

Sorry to sort of hijack here, but: You gotta love FOSS software for exactly this reason! The Issue is reported, and within TWO HOURS, a fix is already made available to the end user. On a weekend, as well!
Thank you so much for your ongoing hard work!
I'll set up my donation to StreetComplete right now, and I recommend that anyone who reads this should consider doing the same, if they are able to do so. This awesome work should be honoured appropriately!

[mmd-osm]

So it seems this is even visible in Prometheus: https://prometheus.openstreetmap.org/d/5rTT87FMk/web-site?orgId=1&from=1652574165743&to=1652645318685

Starting at around 16:30 with 100-200 requests/s w/ HTTP error code 400. The delay was probably caused by some rate limiting which is in place.

[mmd-osm]

Looping in @tomhughes: root cause for excessive HTTP 400 requests on the API has been identified, fix is already being rolled out.

[HolgerJeromin]

Can we restore the api key to support older app versions?
Perhaps from a db backup on osm.org ?

[matkoniecz]

And sorry for a confusion! BTW, is it expected from API to hand out 400 error in such case? It seems that it was supposed to be given for "too large area downloaded", not "authorization failed".

@mmd-osm Note that fix is not yet deployed. It may take some time before Google Play and F-droid users are updated :(

[mmd-osm]

@matkoniecz : do you know the exact API endpoint(s) returning HTTP 400 in that case?

[westnordost]

@mmd-osm At least:

• /api/0.6/[node|way|relation]/#id
• /api/0.6/notes?bbox

... but I suspect any request that is OAuth-signed but with an unknown OAuth app secret+key

[mmd-osm]

@westnordost : thanks, this means both CGImap and Rails port (for the notes endpoint) show a similar behavior. I believe you're still on OAuth 1.0a, right? According to https://datatracker.ietf.org/doc/html/rfc5849#section-3.2, HTTP 401 would be a more appropriate response code for invalid tokens.

We probably need a follow up discussion on https://github.com/openstreetmap/openstreetmap-website on how to proceed here, also keeping in mind that we will likely spend only limited efforts on a soon to be deprecated OAuth1.0a implementation.

[tomhughes]

Any error almost certainly comes from the oauth plugin and nobody is going to be interested in, or able to, do anything much about it.

Frankly I'm surprised you got a 400 as I thought a missing client would probably generate a 5xx as there's a famous bug in the plugin where it doesn't handle a missing client and throws an exception.

[meiphoo7-Mae]

see

for F-Droid it may take up to a few days until the update appears.

You cannot update from other sources if you installed the app from F-Droid because F-Droid uses different signing keys. You either have to wait, or uninstall and install form a different source.

I'm also using the app from F-Droid but I would like to change the source. Please can you tell me what happens if I delete the version from F-Droid and replace it by installing the apk here from Github? My device claims it will delete all data that is associated with the app however I don't want to loose my progress. But now I'm logged out my star count is zero so I suspect that the app is not keeping tracking of my progress by itself but stores it somewhere else. So can I safely switch to the github apk without loosing the statistics?

[westnordost]

You won't lose progress because it will be re-synced after you log in again. What you lose is any settings you made (turning auto-sync on or off, enablement or disablement of quests etc)

[westnordost]

But as mentioned earlier, you can simply stay logged out and use the app as normal, then log in again after the app updated to v43.2 to upload your changes.

[meiphoo7-Mae]

Wow! Thanks, that's a fast response. Yes, I did read that but I would like to change any way because I feel sometimes it takes a long time for F-Droid to have the new version. A few years a go I installed the version from F-Droid to be sure to not get an infected app but in the meantime I'm convinced that the apk from this github is absolutely safe so I rather go for the 'original' app.

[matkoniecz]

Note that you will lose on reinstall also not yet uploaded answers.

As mentioned star count and achievements will recover, but it may take a bit to synchronize.

[meiphoo7-Mae]

Thanks for mentioning that but I upload answers whenever I can so I never keep them around for a long time.

[peternewman]

Can we restore the api key to support older app versions? Perhaps from a db backup on osm.org ?

I guess if someone would have access to the OSM DB for that, they could just do an INSERT on that API tokens table and insert the token again, I'd guess @westnordost still has the old access token saved somewhere… (Then everything should work again.)

It's very hackish, but I'd likely work.

This probably wants looking at doesn't it, for all the people who can't or won't update to v43.2 of SC for whatever reason (e.g. old phone). From my experience being on v43.0, it broke for me so I assume it will for everyone.

Although I've personally no idea of how many people are using old versions.

[tomhughes]

Restoring it from backup is completely impractical - we'd have to drop one of the slave databases and do a restore that would probably take several days, all to recover a single row.

[HolgerJeromin]

Restoring it from backup is completely impractical - we'd have to drop one of the slave databases and do a restore that would probably take several days, all to recover a single row.

Ok.
I only use text based sqldumps where such a thing is easy to do. But my DBs do not contain such amount of data.

[mmd-osm]

Users would have to log on and re-authorize the app in any case as the old entries underneath “My Authorised Applications” have been removed as well.

I’m wondering if we could register yet another oauth application and replace oauth key and secret by their previous values according to: 933f26c

[tomhughes]

It's not a text dump so actually thinking about it I think we can restore specific tables though it would still be quite involved to recover everything needed.

[tomhughes]

I think I have managed to restore both the application record and any granted tokens that existed as of the 9th May when the last backup was done.

[westnordost]

Awesome, you are a life saver! The update is not yet available on F-Droid and then people protact updating anyway, so this saves a lot of people the trouble and also this app from getting a bad reputation. Thank you!

(v43.2+ will still use the new oauth profile because reverting it now to the old one would cause yet another disruption. For the new feature in v44.0, people would need to login again anyway.)

[guerda]

Great, it instantly worked for me again (on F-Droid)!

[HolgerJeromin]

http code 400 dropped as expected. :-)

https://prometheus.openstreetmap.org/d/5rTT87FMk/web-site?orgId=1&from=1652614870056&to=1652787670056

[westnordost]

I'll close and unpin this then

[Filbuntu]

Great, it instantly worked for me again (on F-Droid)!

@guerda How did you manage to get an update on F-Droid? I have updated F-Droid to 1.15 but still no StreetComplete 43.2.
On the homepage it is also still 43.0 (https://f-droid.org/en/packages/de.westnordost.streetcomplete/)

[guerda]

Great, it instantly worked for me again (on F-Droid)!
@guerda How did you manage to get an update on F-Droid?

I didn't. I was able to upload my changes with the current version from F-Droid as the old oauth session was reactivated.

[Filbuntu]

I didn't. I was able to upload my changes with the current version from F-Droid as the old oauth session was reactivated.

That is encouraging. I tried again. Yes, indeed it works now for me again, too. I tried half a day ago and it did not work.