High zoom tile performance unacceptably slow in Antartica

[pnorman]

While diagnosing a renderd/Mapnik problem, I was rendering vertical stripes of z17 tiles, starting at 0,0 and going to 0,131071. This revealed a problem where every few minutes the time per tile would go from 800ms to about 3100ms, repeatedly. A bit of investigation revealed that this was happening when it rendered tiles in the southwest corner of the map, and when rendering in just the 16th of the map in the corner, it was much worse.

Performance was worse in this tile than in Europe, an area with significantly more data.

The time spent was spent waiting for CPU in Mapnik, not query performance. Knowing the data and layers, I believe this time is being spent in the icesheet-poly layer, as it is the only layer that would cover much of Antarctica while not being present elsewhere.

Looking at the backtraces, most of the threads are in agg::vcgen_stroke or another vcgen function.

#0  0x00007f632622ce85 in ?? () from target:/lib/libmapnik.so.3.1
#1  0x00007f6326230f50 in agg::vcgen_stroke::add_vertex(double, double, unsigned int) () from target:/lib/libmapnik.so.3.1
#2  0x00007f63260c0af3 in ?? () from target:/lib/libmapnik.so.3.1
#3  0x00007f632609c3cd in ?? () from target:/lib/libmapnik.so.3.1
#4  0x00007f63260d5041 in ?? () from target:/lib/libmapnik.so.3.1
#5  0x00007f63260e04ae in mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4>::process(mapnik::line_symbolizer const&, mapnik::feature_impl&, mapnik::proj_transform const&) () from target:/lib/libmapnik.so.3.1
#6  0x00007f6325967715 in mapnik::feature_style_processor<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >::render_style(mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4>&, mapnik::feature_type_style const*, mapnik::rule_cache const&, std::shared_ptr<mapnik::Featureset>, mapnik::proj_transform const&) () from target:/lib/libmapnik.so.3.1
#7  0x00007f63259689d9 in mapnik::feature_style_processor<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >::render_material(mapnik::layer_rendering_material const&, mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4>&) () from target:/lib/libmapnik.so.3.1
#8  0x00007f6325968eee in mapnik::feature_style_processor<mapnik::agg_renderer<mapnik::image<mapnik::rgba8_t>, mapnik::label_collision_detector4> >::apply(double) () from target:/lib/libmapnik.so.3.1
#9  0x000055a5ff41da88 in ?? ()
#10 0x00007f6325094b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#11 0x00007f6325126a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

The styling here is very simple

https://github.com/gravitystorm/openstreetmap-carto/blob/445e553/style/shapefiles.mss#L19-L23

A few thoughts

1. polygon-clip is the only option I can think of that would impact performance, but I'm skeptical it would do anything here. The time spent is during rendering, not during clipping.
2. Some of the polygons are large (multiple MB). Should the upstream source be splitting the polygons, similar to how oceans are split?
3. If upstream is unable or unwilling to split, should we split them on import?

[imagico]

The icesheet polygons are large because they are generated from the coastline polygons - which are dynamically split based on coastline complexity. Hence low complexity coastline polygons with lots of complex ice free area polygons inside (i.e. Antarctic interior) result in large icesheet polygons with many inner rings. It is not unreasonable that this might lead to rendering being slow.

This is solvable - but it requires some work in development and testing - which is difficult without a budget.

Since we meanwhile have the coastline polygons in the database we could also think about rendering the icesheets on the fly based on either ST_Difference() between (tile clipped) coastline polygons and the ice free areas or by simply drawing the ice free areas in land color over the glaciers (you would then still need to do some geometry operations for proper outline rendering though). Geometry operations would slow down rendering at the low zoom levels - but it would probably not be significant at the high zoom levels since geometric complexity per tile is low. Back when we introduced the Antarctic ice sheet rendering you were strictly against special casing the Antarctic in the style but i am meanwhile quite sure that this would be the better option because it would provide real time updates on the icesheet and would avoid the confusing mismatch between the delayed icesheet updates and the ice free area fills (mostly natural=bare_rock).

See also #4827.