Polygons from OSM layer are missing at some scales

[warrenwyf]

I found that some polygons from OSM layer were lost in the tiles at certain scales, so I made a test data reproduction of this phenomenon.

This is the polygon object that I picked from the OSM data that would cause the problem:

Then exported the polygon as a shapefile named "simplify-wrong.shp" and added each of these two data to the config:

{
  "layers":
  {
    "from_osm":
    {
      "minzoom": 0,
      "maxzoom": 14,
      "simplify_below": 13,
      "simplify_level": 0.0003
    },
    "from_shapefile":
    {
      "minzoom": 0,
      "maxzoom": 14,
      "simplify_below": 13,
      "simplify_level": 0.0003,
      "source": "simplify-wrong.shp"
    }
  },
  "settings":
  {
    "minzoom": 8,
    "maxzoom": 14,
    "basezoom": 14,
    "mvt_version": 1
  }
}

And the process script is shown below:

function init_function()
end
function exit_function()
end

ZRES5  = 4891.97
ZRES6  = 2445.98
ZRES7  = 1222.99
ZRES8  = 611.5
ZRES9  = 305.7
ZRES10 = 152.9
ZRES11 = 76.4
ZRES12 = 38.2
ZRES13 = 19.1

-- Process node tags
node_keys = {  }
function node_function(node)
end

-- Process way tags
landcoverKeys   = { wood="wood", forest="wood",
                    wetland="wetland",
                    beach="sand", sand="sand",
                    farmland="farmland", farm="farmland", orchard="farmland", vineyard="farmland", plant_nursery="farmland",
                    glacier="ice", ice_shelf="ice",
                    grassland="grass", grass="grass", meadow="grass", allotments="grass", park="grass", village_green="grass", recreation_ground="grass", garden="grass", golf_course="grass" }

-- Process way tags
function way_function(way)
	local natural  = way:Find("natural")
	local landuse  = way:Find("landuse")
	local leisure  = way:Find("leisure")
	local isClosed = way:IsClosed()

	local l = landuse
	if l=="" then l=natural end
	if l=="" then l=leisure end
	if landcoverKeys[l] then
		way:Layer("from_osm", true)
		SetMinZoomByArea(way)
		way:Attribute("class", landcoverKeys[l])
		if l=="wetland" then way:Attribute("subclass", way:Find("wetland"))
		else way:Attribute("subclass", l) end
	end
end

function SetMinZoomByArea(way)
	local area=way:Area()
	if     area>ZRES5^2  then way:MinZoom(6)
	elseif area>ZRES6^2  then way:MinZoom(7)
	elseif area>ZRES7^2  then way:MinZoom(8)
	elseif area>ZRES8^2  then way:MinZoom(9)
	elseif area>ZRES9^2  then way:MinZoom(10)
	elseif area>ZRES10^2 then way:MinZoom(11)
	elseif area>ZRES11^2 then way:MinZoom(12)
	elseif area>ZRES12^2 then way:MinZoom(13)
	else                      way:MinZoom(14) end
end

Finally, use the command to generate mbtiles:

tilemaker  --input test.osm.pbf --output test.mbtiles --config config-test.json --process process-test.lua --bbox 115.6,39.97,116.13,40.36 --verbose

When rendering this mbtiles, you can see that at some scales, the polygons in the "from_osm" layer are partially missing, but the polygons in the "from_shapefile" layer are intact.

I have packaged the data and configuration related to this test in the attachment:

test.zip

[systemed]

Thanks for a great test case.

This appears to be an issue with clipping. Commenting out lines 127-158 in output_object.cpp mostly fixes it (there's still some weirdness west of the channel). I don't yet know whether our algorithm for generating the clipping box is wrong, or whether we've unearthed an issue with boost::geometry::intersection.

(Note to self: The clipping code was mostly introduced in #234 so that's where to start looking.)

[systemed]

I think #430 will fix this particular issue. There still appears to be some weirdness with this test case but I'm not sure that the relations in this OSM file are well-formed so I'm not concentrating on those!

[warrenwyf]

I think #430 will fix this particular issue. There still appears to be some weirdness with this test case but I'm not sure that the relations in this OSM file are well-formed so I'm not concentrating on those!

Thank you for trying to fix this issue, I built a binary using the "clip_endzoom" branch and regenerated mbtiles, unfortunately this problem seems to persist. Here are the results of generating similar problematic polygons at several levels:

Level 10 & 11:

Level 12:

Level 13:

Level 14:

These pictures are around the coordinates (115.6472,40.2085), but this is not an isolated case, it can easily happen with large and complex polygons, such as the "wood" type of the "landcover" layer.

By the way, I downloaded the data directly from geofabrik in osm.pbf format, without any special modifications.

[systemed]

Hm, sorry about that. Would you be able to provide an .osm.pbf for the affected area, extracted either with --strategy=smart (Osmium) or clipIncompleteEntities=true (Osmosis)? The one in the .zip had missing relation members.

[warrenwyf]

Attached is a piece of data extracted using osmium with --strategy=smart, the bbox of data is 115.4207,40.0886,115.9086,40.3494

test.osm.pbf.zip

[systemed]

That's super helpful - thanks. I'll see what I can work out.

FWIW https://www.openstreetmap.org/relation/5989768 is the issue. It doesn't seem to be simplification-related (same issues with simplification turned off). I'm not convinced it's well-formed. Looking at the multipolygon assembly code may be the first thing to do.

Interestingly Overpass doesn't appear to recognise it either.

[systemed]

Could you try #432? That appears to fix it for me.

[warrenwyf]

I tested it with this branch and it does seem to have solved the problem. 👍

[systemed]

Excellent! Thank you for your help with this.

It turned out to be a bug in boost::geometry (which tilemaker uses internally for geometry operations) failing to clip the multipolygon correctly to tile boundaries.