Have a fallback for overpass server

[dbf256]

Use case
Currently overpass api url is hardcoded in https://github.com/westnordost/StreetComplete/search?q=OVERPASS_API_URL&unscoped_q=OVERPASS_API_URL This host is blocked by Russian ISPs so without VPN it is not possible to load quests. Also there is a small risk that main overpass server is down.

Proposed Solution
There are several overpass instances: https://wiki.openstreetmap.org/wiki/Overpass_API
Maybe it is possible to fallback to some of them if primary is not available (or allow to specify host in settings).

[westnordost]

Ok weird, why would they block the overpass api? Also, since when?

Rather than manually changing the overpass api url, another solution is to automatically select the overpass server based on the network location, somehow. Not sure how to find this out though.

Edit: I see there is no Russian-based overpass api. I thought there was.

[matkoniecz]

Ok weird, why would they block the overpass api?

https://en.wikipedia.org/wiki/Internet_censorship_in_Russia blocks pretty wide range of things, hard to guess what triggered this particular block

I see there is no Russian-based overpass api. I thought there was.

Down since https://wiki.openstreetmap.org/w/index.php?title=Overpass_API&diff=1638185&oldid=1634053

[matkoniecz]

select the overpass server based on the network location, somehow

Maybe just detect failed connection and failover to the next one in the list?

[dbf256]

@westnordost, they tried to block Telegram messenger last year and blocked a lot of Amazon/Hetzner/DigitalOcean subnets that were used by Telegram team to setup proxy servers for their application. So there is nothing wrong with overpass, just a side effect of some mindless actions.

So maybe it is possible to have a set of overpass urls and automatically switch between them if current one is not reachable.

[rugk]

Maybe just detect failed connection and failover to the next one in the list?

Depending on what servers you want to include, you need to take care: not all have the whole world data in there (so per location may be the best, for a first try at least).
Also some have other "fair use" limits like maximum of X numbers of queries per day.

I feel a machine-readable version of that table may be useful, so it can determinate/fallback/prioritize the servers based on how much load they want/where they are/etc. (mhh, and ping/response times may also be a factor… 🤔)

[mmd-osm]

Also watch out for servers with broken db updates, e.g. checking timestamp_osm_base in the overpass response is kind of mandatory.

[westnordost]

Not sure If I should implement this. Is it not possible to report the Overpass api website as a false positive to those ISPs?

Also, what would be the http response code in case the url is blocked? Or is it just a DNS block?

[dbf256]

@westnordost , it is not possible - blocklists are provided to ISPs and they must apply them.
I'm not sure that there is any code, most likely it is a failure of https connection (if it is http than provider could return some stub page that this url is in blacklist).

[westnordost]

Well okay, but they might block other overpass instances as well. I'd accept a PR in advanced settings where one can change the URL(s) of overpass Api (, OSM Api, ... etc)

[matkoniecz]

I'd accept a PR in advanced settings where one can change the URL(s) of overpass Api (, OSM Api, ... etc)

What about automatic fallback where new overpass turbo url is used if query failed on a a current one?

This way in case of block user has no need to manually switch it. Main negative is that SC users may fall back to secondary reasons due to flaky internet or requesting manually many queries. Maybe this may overload secondary overpass or some secondary overpass servers have outdated data?

[matkoniecz]

Looking at list of servers it seems to be worse than I remembered

overpass-api.de - already used

---

French Overpass API instance - likely to be not eligible even with manual switch needed

As long as you don't exceed 1000 queries per day per "project". (By "project" I mean if you use this API in a smartphone app and you have 10,000 customer querying once per day then you are above the limit)

---

Swiss Overpass API instance data coverage is "Switzerland only"

---

Kumi Systems Overpass Instance - currently two weeks of replication lag, but better than nothing

---

Taiwan Overpass API - unclear data limits, I will try to contact owners as instructed

[mmd-osm]

Kumi differs quite a bit across their 4 servers, 3 of them are no longer updating at the moment. Attic queries don't work on any of them. Taiwan seems very stable and up to date. Besides, there's an Overpass instance for OSMCha, which is very up to date (but not in the official list).

[matkoniecz]

Kumi differs quite a bit across their 4 servers, 3 of them are no longer updating at the moment.

Thanks! https://3.overpass.kumi.systems/api/ turned out to be updating one.

Attic queries don't work on any of them.

Fortunately this should not be a problem, attic is used only in development (to get stable test not affected by current state of the database).

Taiwan seems very stable and up to date.

Thanks! Hopefully I will manage to contact owners (and lets hope that it is not blocked in Russia).

EDIT: first contact attempt - https://wiki.openstreetmap.org/wiki/Overpass_API#Public_Overpass_API_instances directed me to https://osm.tw/contact/ - for now I used FB group as it was on the first position.

[dbf256]

Thanks! Hopefully I will manage to contact owners (and lets hope that it is not blocked in Russia).

overpass.nchc.org.tw is not blocked.

[matkoniecz]

overpass.nchc.org.tw is not blocked.

Thanks for checking!

What about https://3.overpass.kumi.systems/ ? Is it also OK?

[dbf256]

What about https://3.overpass.kumi.systems/ ? Is it also OK?

Yes, this one is fine also.

[matkoniecz]

Yes, this one is fine also.

Great, so once implemented it will actually help and there is no need to wait for response from Taiwan (though one more additional server available would be great).

I think I will go with manual override given that alternative servers are a bit unstable (only one kumi server updating, Russian server disappeared some time ago etc).

In comparison main Overpass instance is extraordinarily stable and excellently maintained (thank to person/people handling this!).

[mmd-osm]

In comparison main Overpass instance is extraordinarily stable and excellently maintained (thank to person/people handling this!).

Both instances are administrated by Roland (@drolbr)

[westnordost]

Maybe the easiest solution for the Russians would be to setup a simple php (or so) script on a non-blocked webspace that simply forwards the request to the main overpass instance and returns the result.
Combined with that one can edit the URL to the overpass instance in some advanced setting.

[mmd-osm]

IP-based rate limiting is in place on the main Overpass instance, so a simple forwarder would be blocked most of the time.

[matkoniecz]

I contacted Kumi Systems as documented on wiki, and there is reply already that their servers now restarted updates. Though for now there is no sign that updates restarted (now 15 days of a lag).

[mmd-osm]

Probably their database(s) are corrupt, a restart won’t help here.

[kumitterer]

It is quite odd that three out of four databases would be corrupted very shortly after an upgrade to the latest version of the Overpass server software with fresh databases at the time. Nonetheless, we are already in the process of setting those servers up again from scratch, so by the end of the day I hope that all servers will have a current state again.

What is even odder is that you are saying that attic queries don't work on any of the servers – I'm not sure what the reason for that would be, but I'll try to figure that out too.

[mmd-osm]

I can’t say for sure if the databases are really corrupt, however attic queries were returning errors about missing/invalid geometry. Usually that’s a strong indication that something went wrong with some previous update. We had a case where those errors disappeared, but usually the update process would abort once it hits those objects with invalid geometry.

[matkoniecz]

Kumi Systems servers are now up, with replication lag limited to 20 minutes. Thanks!

And I found a checker:

https://isitblockedinrussia.com/?host=https%3A%2F%2Foverpass.kumi.systems%2Fapi%2F

https://isitblockedinrussia.com/?host=https%3A%2F%2Foverpass-api.de%2Fapi%2F

that lists

Decision 27-31-2018/Ид2971-18 made on 2018-04-16 by Генпрокуратура.

as reason for blocking

178.63.0.0/16

[matkoniecz]

@mmd-osm

Besides, there's an Overpass instance for OSMCha, which is very up to date (but not in the official list).

Do you think that it would be OK to ask them for permission to use it? Or is it something that would be pushy/useless/unwelcome?

[matkoniecz]

@westnordost Not sure whatever you are interested in doing this but there are probably some irritated reviews by Russian users that now can get answer with solution that (hopefully) works for them.

[westnordost]

Sure, ok, I'll do that