Replace usage of Natural Earth boundaries data

[matkoniecz]

Currently ne boundaries data is used for z1 to z3. Replacing seems to be a simple change, but unfortunately with some side-effects.

Next usage of NE data will be removed (what will leave builtup layer as sole place where OSM data is not used).

Some places may no longer have a different data (direct consequence of above)

Rendering of z1 to z3 will be much slower (small initial testing indicated that generation time may double or even tripple)

Due to #2164 the same line-width result in different lines (I am not sure how complicated would be to ensure that lines will be at least sort-of similar - lines are not dashed so it is not completely hopeless).

Maritime borders will be rendered (#621), but with ocean-polygons it should be fixable.

before

after

In this case I would be tempted to remove ne data, and search for way to ensure that new borders will not look worse than current ones. But there are significant performance regression (what is quite ironic, as I noticed that ne data is used during looking for easy ways to improve performance). As potential alternative - is there an OSM derived equivalent of ne borders?

But maybe proper method of solving this is to document that ne data is used for performance reasons and to avoid #2164 and #621.

[matkoniecz]

Updated before/after and conlusions, I initially switched files.

[dieterdreist]

sent from a phone

Il giorno 16 giu 2016, alle ore 18:14, Mateusz Konieczny [email protected] ha scritto:

Rendering of z1 to z3 will be much slower (small initial testing indicated that generation time may double or even tripple)

this can be ignored, these tiles are rendered rarely and in both cases one metatile covers the whole mercator world

[pnorman]

Rendering of z1 to z3 will be much slower (small initial testing indicated that generation time may double or even tripple)

z1-z2 isn't a concern, z3 might be an issue but I suspect its slowness is due to placenames-medium

[sommerluk]

I suspect its slowness is due to placenames-medium

placenames.mss does not render placenames-medium on z3 anymore since 4c6c890 but the project.yaml still querries it on z3. So in project.yaml, we could switch to “minzoom: 4” without rendering changes.

[kocio-pl]

I guess rendering time for z1-z3 is not a big issue - they are probably not being refreshed too frequently anyway (once a month on OSMF servers) and it's only 84 of them, which is just a small fraction of even just low zoom tiles, let alone all the rendered tiles. 300% slower might sound scary, but let's not forget about scale of the problem.

[kocio-pl]

Due to #2164 the same line-width result in different lines (I am not sure how complicated would be to ensure that lines will be at least sort-of similar - lines are not dashed so it is not completely hopeless).

Could we just make them wider for these zoom levels or the problem is more complex?

[jeisenbe]

Now that we are using water polygons to render the oceans, it is possible to stop using NE boundary data. However, there are performance advantages and rendering advantages to using pre-processed boundary data.

An alternative would be to develop a new set of generalized country boundaries for low zoom levels and have them available at http://osmdata.openstreetmap.de

I believe this would be a better solution, compared to the current option of rendering administrative boundaries from the full data but using smoothing to get the lines to look better; it would have high performance and would still be based on OSM data.

Is anyone interested in writing a script that could produce the simplified or generalized borders?

This could also be used for zoom levels 4 to 10, if admin_level 3 and 4 were included in a simplified borders file

[kocio-pl]

Please look at #2172 (comment). What performance gain would we get this way (in summary)?

I'm not against using external data when the troubles would be big, but it's always preferable to not rely on them if the problem is small.

[jeisenbe]

I'm still interested in seeing this developed. I don't personally have the skills with PostGIS or whatever would be needed to make this happen, but I suspect that admin boundaries based directly on OSM data would be popular, and a nice complement to the simplified ocean polygons available on https://osmdata.openstreetmap.de

Using full data will be much slower and would lead to poor results in many cases, because the Mapnik simplify algorithms do not work very well with curvy boundaries (For example, those that follow meandering rivers)

[StyXman]

Would it be something along the lines of: once th data has been imported, use ST_Simplify or ST_SimplifyPreserveTopology to copy the data from one table to another? Or would you like something more along the lines of planet_osm_roads, which is generated at import time?

[imagico]

You need to generate the boundary grid topology from the boundary polygons and doing this you will probably also need to apply some heuristics to deal with smaller errors in mapping (i.e. adjacent borders not accurately matching). You would also need topology simplification (i.e. removal of small exclaves) in addition to line simplifications for good results at the lower zoom levels. What would also be important is to split boundaries into land and maritime boundaries and not modify the maritime boundary geometries (although this is not strictly necessary because this can also be done with compositioning).

Note this is not only needed for z1-z3 as this issue indicates - it is also needed beyond (and that means also for admin_level 3/4) because rendering at z4+ is currently faulty due to introduction of not topology aware line simplification in #3103. See #907.

[StyXman]

You would also need topology simplification (i.e. removal of small exclaves)

... but not small countries, I guess?

[imagico]

I changed the title of this issue to definitely aim at replacing and not just explaining the use of non-OSM data.

Reasons are:

• our own goals as well as the Guidelines for new tile layers mandate us to use OSM data where possible.
• Natural Earth is not meant to be used in Mercator projection so the current use is highly suboptimal even ignoring the previous point.
• Natural Earth is often outdated and does not necessarily adjust to a changing reality in a timely fashion like OSM does. This is much more severe with other parts of Natural Earth (which in parts are half a century old!) and not so much for the admin boundaries - but still...
• Natural Earth - as already pointed out back in Order placenames by population and substitute NE usage on low zoom level... #1461 - transports a distinctly US centered perspective and as such is highly unsuitable for our goal to represent the whole OSM community. That has so far not often been a huge practical problem in the past for the national boundaries at z1-3 but there is no guarantee that it won't be in the future and even without practical issues the principle matters, especially with something as politically sensitive as boundaries.

What is needed to solve this has been outlined in #2172 (comment).

Personally i would be in favor of removing the use of Natural Earth (i.e. not showing boundaries at z1-3) until a decent solution for rendering those from OSM data is available. But i know that such a step might be controversial.

[SomeoneElseOSM]

Whilst I understand the logic behind replacing the use of NE data (see #4803 , which I added) I don't understand "as already pointed out back in #1461 - transports a distinctly US centered perspective" in the comment above - the words from "transports a distinctly US centered perspective" don't seem to appear there?

Edit: Thanks for linking to the issue in more detail below.

[imagico]

Sorry for just vaguely linking to the issue, an example of US-centrism was mentioned in #1461 (comment).

To be clear - i do not know what the cause is for Natural Earth defining the boundaries in East Africa as it does - this does not necessarily have to do anything with the general cultural/political bias in Natural Earth. My point is that the bias is a reason (among others) to get rid of Natural Earth and a solution to the Somaliland problem (within NE or as a patch of it) would not remove that argument.

[matkoniecz]

do not know what the cause is for Natural Earth defining the boundaries in East Africa as it does

They describe own policy at https://www.naturalearthdata.com/about/disputed-boundaries-policy/

A notable edge case is Somaliland which is treated as a sovereign state in Natural Earth because it meets the rubric above.

[imagico]

Thanks for the pointer. On paper this is actually more consistent as a verifiable definition than what we in OSM have for admin_level=2 (which is a completely vague idea of a country).

But note that NE has two different data sets: countries and sovereign states:

https://www.naturalearthdata.com/downloads/110m-cultural-vectors/

The boundary lines we use are derived from the countries file while the text you cite is mostly about the sovereign states. For East Africa there is of course no difference between the two.

Bottom line: We cannot really make a well substantiated claim that NE is any more subjective and biased than OSM w.r.t. country/admin_level 2 boundaries. But given the role and function of this project providing consistent feedback just on the OSM view of the world and not intermixing that with a distinctly different view is most certainly a prudent idea (in addition to the other reasons mentioned).

[matkoniecz]

But given the role and function of this project providing consistent feedback just on the OSM view of the world

Yep. For reference, OSMF just received complaint about Somaliland being depicted as independent on the OSM Carto-powered map. So de facto complaining about this issue, or even more specifically #4803 (and ideally we would only get complaints about actual OSM data, not about both OSM and NE borders)

[imagico]

And i can very much relate to this being a valid complaint. Not on the political level - i lack the local background knowledge to judge that - but regarding the map not accurately representing what is mapped in OSM here.

In #4803 (comment) i explained what is needed to solve this and how any organization interested in supporting OSM could help accomplishing that. In German we call this ein Wink mit dem Zaunpfahl. 😉