Low-zoom ocean shapefile not rendered correctly on some setups

[codebrane]

Using Mapnik 3.0.9 with openstreetmap-carto 2.39.0 produces tiles with a blue tint. The coastlines are not visible. Using the version from master, the land and inland water is fine but the sea isn't rendered. This tile shows the issue with 2.39.0. I'm using polytiles.py with osm.xml generated by carto from project.mml. If I use the default Mapnik osm.xml on the same data it renders fine but of course with the older OSM style.

[codebrane]

This is what I get using openstreetmap-carto master

and this is what it should look like

[matthijsmelissen]

Have you tried running ./get-shapefiles.sh? We switched from land-polygons to ocean-polygons, this might have something to do with this.

I also saw some blue tiles in Europe on the openstreetmap.org servers, so there might also be a data problem.

[codebrane]

yes. The flow was:

./get-shapefiles.sh
osm2pgsql -c -d gis --slim -C 1000 --flat-nodes flatnodes GB.pbf --style openstreetmap-carto.style
carto -l project.mml > osm.xml
polytiles.py -b -5.93 55.68 -2.29 58.79 -z 10 11 -s osm.xml -t tiles

if I use:
polytiles.py -b -5.93 55.68 -2.29 58.79 -z 10 11 -s osm.xml -t tiles
using the original Mapnik osm.xml the tiles are fine

[pnorman]

The shapefiles are not rendering on either 2.39.0 or master for you.

On master go to the same directory as project.mml and osm.xml and run

ogrinfo -so -al data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp
ogrinfo -so -al data/water-polygons-split-3857/water_polygons.shp

The results should be

INFO: Open of `data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp'
      using driver `ESRI Shapefile' successful.

Layer name: simplified_water_polygons
Geometry: Polygon
Feature Count: 6101
Extent: (-20037558.342789, -20037408.342789) - (20037558.342789, 20037558.342789)
Layer SRS WKT:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
FID: Real (11.0)

INFO: Open of `data/water-polygons-split-3857/water_polygons.shp'
      using driver `ESRI Shapefile' successful.

Layer name: water_polygons
Geometry: Polygon
Feature Count: 35214
Extent: (-20037508.342789, -20037508.342789) - (20037508.342789, 20037508.342789)
Layer SRS WKT:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
FID: Real (11.0)

Check that the feature count is approximately the same.

Also, post the result of ls -hl data/water-polygons-split-3857 data/simplified-water-polygons-complete-3857

[codebrane]

simplified_water_polygons.shp output is identical to above.
water_polygons.shp Feature Count = 35211 instead of 35214

ls -hl data/water-polygons-split-3857 data/simplified-water-polygons-complete-3857

data/simplified-water-polygons-complete-3857:
total 29M
-rw-r--r-- 1 vagrant vagrant    6 Mar  1 08:37 simplified_water_polygons.cpg
-rw-r--r-- 1 vagrant vagrant  72K Mar  1 08:37 simplified_water_polygons.dbf
-rw-rw-r-- 1 vagrant vagrant 169K Mar 30 12:15 simplified_water_polygons.index
-rw-r--r-- 1 vagrant vagrant  847 Mar  1 08:37 simplified_water_polygons.prj
-rw-r--r-- 1 vagrant vagrant  29M Mar  1 08:37 simplified_water_polygons.shp
-rw-r--r-- 1 vagrant vagrant  48K Mar  1 08:37 simplified_water_polygons.shx

data/water-polygons-split-3857:
total 565M
-rw-r--r-- 1 vagrant vagrant    6 Mar  1 08:38 water_polygons.cpg
-rw-r--r-- 1 vagrant vagrant 413K Mar  1 08:38 water_polygons.dbf
-rw-rw-r-- 1 vagrant vagrant 497K Mar 30 12:15 water_polygons.index
-rw-r--r-- 1 vagrant vagrant  847 Mar  1 08:38 water_polygons.prj
-rw-r--r-- 1 vagrant vagrant 564M Mar  1 08:38 water_polygons.shp
-rw-r--r-- 1 vagrant vagrant 276K Mar  1 08:38 water_polygons.shx

[codebrane]

I'm using this to import GB.pbf to the db:

osm2pgsql version 0.91.0-dev (64 bit id space)

[tds4u]

Problem is still there, even with ogr2ogr for all shape files to fix possible inconsistencies.

[matthijsmelissen]

I cannot reproduce this.

@codebrane Did you already find the cause of this problem by any chance?

[tds4u]

I can reproduce that:

mapnik-config -v
2.2.0

node -v
v0.10.29

Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessi

[matthijsmelissen]

@springmeyer Do you perhaps have any idea what might cause this rendering glitch? The screenshot above looks quite particular.

[imagico]

The image shown looks like an 180-degree-meridian reprojection error.

Keep in mind what i wrote in

http://blog.imagico.de/on-slicing-the-world-news-from-openstreetmapdata-com/

on the simplified water polygons - that you should not attempt to reproject them, not even a simple round trip conversion Mercator->Geographic->Mercator. This will fail since the polygons slightly extend over the 180-degree-meridian.

I assume if the coordinate system you have set in Mapnik and the coordinate system in the shapefile are formally different you get a coordinate system conversion and therefore a failure. I am not familiar enough with Mapnik to know if and under what conditions this happens.

[tds4u]

Projection is EPSG:3857 / WGS 84 Pseudo Mercator and in project file it is EPSG:900913 - so the google one. Should be no problem at all.

@springmeyer: Did you test it with planet import?

[tds4u]

QuantumGIS can render this correctly:

I think the shape file is broken, the following is working:

#  - id: "ocean-lz"
#    name: "ocean-lz"
#    class: "ocean"
#    geometry: "polygon"
#    <<: *extents
#    extent: *world
#    srs-name: "mercator"
#    srs: "+proj=merc +datum=WGS84 +over"
#    Datasource:
#      file: "data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp"
#      type: "shape"
#    advanced: {}
#    properties:
#      maxzoom: 9
  - id: "ocean"
    name: "ocean"
    class: "ocean"
    geometry: "polygon"
    <<: *extents
    Datasource:
      file: "data/water-polygons-split-3857/water_polygons.shp"
      type: "shape"
    properties:
#      minzoom: 10
    advanced: {}

[xan-der]

Can confirm this problem with Mapnik 3.0.11 and Mapnik 2.2 and the current openstreetmap-carto pulled earlier this week. All of the global level maps (through level 5 or so) are malformed. I see exactly the same image as reported by tds4u earlier. Note that OSMBright from MapBox works correctly using Mapnik 3.0.11, but it doesn't use water-polygons, it only uses simplified-land-polygons-complete-3857 and land-polygons-split-3857.

[matthijsmelissen]

Thanks for the confirmation. Certainly something we need to look at.

@pnorman Any idea why some users suffer from this and other don't?

[pnorman]

@pnorman Any idea why some users suffer from this and other don't?

No. I'll see if I can reproduce.

[xan-der]

Followup. I was able to get clean world level images by changing project.mml to use the ocean color for the background and land-polygons-split-3857/simplified-land-polygons-complete-3857. Have only tried the first few zoom levels so far. This was the only change made to the mml file - replacing background color and the two layers from water to land.

From this experiment, I have to conclude that there is something squirrely about the water polygons as they are processed by Mapnik.

[pnorman]

Having tried recent water polygons on several versions of Mapnik, I'm unable to reproduce, so I'm concluding that the issue is something local or a problem specific to some version of Mapnik, neither of which are fixable in this repo.

[matthijsmelissen]

Note that we have three people reporting the same problem, so I'd still like to keep an eye on this. I wouldn't like to see this problem suddenly showing up on production without us having an idea of what might cause it.

[springmeyer]

Those that are able to replicate: does the problem go away if you delete the .index file that comes along with the shapefiles?

[xan-der]

The problem did not go away after deleting the .index files for the water polygons. Attached are two images that show (1) what you get when using water color as the background and land polygons as the foreground and (2) what you get when using land color as the background with water polygons as the foreground . Besides the water/land inversion, the rest of the yaml/mml/xml files are identical.

The configuration is: Debian 8.3, Mapnik 3.0.11 (using GDAL 2.0.2, boost 1.60), simplified_water_polygons and water-polygons (split-3857) both dated 18 April 2016, Python 2.7.9.

While Mapproxy is being used as the front end, caching for the example layers is turned off, so the images are regenerated every time.

[springmeyer]

I don't know what would cause this.

[tds4u]

Let me think...

• Carto will convert YAML into CSS via NodeJS. So no problems so far, right?
• Mapnik parses XML and render stuff, equal if 2.2 or 3.x, right?
• Using MapProxy or other render tools is not important because all use Mapnik, right?

So the problem maybe is not inside Mapnik, maybe inside conversion or rendering tool chain. Maybe OGR or similar libraries? But how to check this?

[pnorman]

Mapnik doesn't read CartoCSS, it reads Mapnik XML. Carto converts CartoCSS to Mapnik XML. You could rule it out by building the XML on a known good machine, but I can't see any way for it to be a cartocss problem with that last picture of inverted water bands.

[tds4u]

So, I tested further. With my workaround (#2101 (comment)) I had some blocks on the 180° meridian. That happen if rendering was made till level 10 via MapProxy - rendering till level 8 won't generate such blocks!!! After that all is fine for rendering from 8 to maximum level.
So maybe it's a "feature" / bug on higher zoom levels with shape files?

[pnorman]

I don't know what would cause this.

As @imagico commented, could it be related to the reprojection? Is mapnik making any reprojections here? If so, is it using any libraries which might change the behavior depending on version?

---

@codebrane what zoom range did you observe problems on?

[codebrane]

zoom 10 shows the water problems. Below that there isn't any boundary between the missing water and the land (screenshot of level 5 attached)

[pnorman]

zoom 10 shows the water problems

If you're seeing them on zoom 10+, that means you're seeing problems with both the ocean-lz and ocean layers.

What OS are you using, as well as what geos and proj versions is your Mapnik linked against?

[matthijsmelissen]

drop support for applications that perform round trip coordinate conversion.

Which software would that be?

[imagico]

Apparently the setups used by @codebrane and @tds4u. But since the discussion in this issue was not particularly goal oriented overall so far it is not really clear under what conditions exactly Mapnik performs a reprojection. It is possible that this can be prevented reliably (at least by making sure there isn't any other proj4 string involved than the one in this style) although some applications might be hardwired in a way that makes this impossible.

[matthijsmelissen]

I have reverted the change that causes this for now.

And therefore this issue can be closed.

[jeisenbe]

While this issue is currently closed, it's disappointing that we never confirmed the cause, though it is most likely related to round-trip reprojection of the simplified ocean polygon shape file.

Andy Alan seemed to think it was a problem with incorrect setup on the affected servers, since this reprojection should not be allowed:

"It appears to me to be a fairly straightforward problem - the +over in the projection of the map output is 100% required, and any system missing it out will run into difficulties." Full comment: #2101 (comment)

There were some thoughts that the shapefiles were a problem, but changing to shape files to add +over and to remove the empty geometries did not seem to fix the problem. However only one user seems to have tested after this change, and did not respond to a follow-up question a couple of months later: #2101 (comment)

@pnorman what are your thoughts? Do you agree with Andy's comments that this was due to inappropriate setups on the affected servers?

[rrzefox]

It has been a long time since this was closed, so it might be worth invastigating this again.
As one of those that saw this problem in the past, I can say that I have not seen this occour again after the german fork of this style switched to using water polygons (we render that style too).
As to WHY I did not see this again I have no clue, there are too many variables. The german style might use them in an other way that avoids the issue. We have updated to a new Ubuntu LTS release with significantly newer software versions. And last but not least, I never was convinced it wasn't the shapefiles that were to blame (at least partially), so whether you could see the error or not (also) depended on what day you downloaded them on. BTW, to be able to debug that by using a version from a specific date I have since set up https://ftp.fau.de/pub/openstreetmapdata.com-archive/ as openstreetmapdata.com itself only ever keeps the newest version.

[kocio-pl]

Thanks for the comment, I was about to ask you if you could test it. If water polygons are generally working elsewhere, I think we could take the risk to go this way too, because the stakes are high enough. As @Adamant36 said, it's possible to revert this if something will go wrong.

I would start with making test code branch and asking again affected users if they still see the problem. If not, we could merge it soon after release, so there will be plenty of time to do additional testing.

[imagico]

If this is done - and i don't want to prejudice the opinion of other maintainers in any way here - it would be important to come to a consensus how long to wait afterwards before changes that depend on ocean polygons are made (which would prevent a simple revert).

[jeisenbe]

@rrzefox thank you for the update, and for maintaining that archive! I believe if the German fork is rendering properly, we are likely to be able to implement the same.

how long to wait afterwards before changes that depend on ocean polygons are made

I'd suggest merging the changes to Master right after the next release, which will give current contributors and other interested parties several weeks for testing before the changes are released. Then we should wait 1 more month after the water polygons are in production before merging PRs that depend on this change.

Example: if the new release is tomorrow, we should submit a PR using the split and simplified water polygons; and hopefully we can merge this in a week or two. Then we will have 3 or 4 weeks with the water polygons merged to master, but not yet released. Assuming the next release after that is in mid-March, we would then wait till mid or late April before merging any PRs that depend on the change.

Does that sound like enough time for testing?

[jeisenbe]

We are considering changing to ocean polygons again, which risks reintroducing this issue. See PR #3694 - however @rrzefox has reported that he no longer has the problem.

@codebrane , @xan-der and @tds4u - would you be able to test the new branch from the PR to see if this will now work on your current system? The branch name is jeisenbe:ocean-polygons https://github.com/jeisenbe/openstreetmap-carto/tree/ocean-polygons

If anyone else had seen this bug back in 2016, please let us know if the shapefiles are now working for you.

[sommerluk]

On my local machine with a fresh download of all shape files, the ocean polygons do not render. I can fix this by deleting the .index files. I've done this also for the low-zoom boundaries (otherwise the rendering eats all the memory and crashs) and for ice shields.

What is the role of the .index files? Are there side-effects deleting them? If not, could we do this automatically in our shapefile download script?

[pnorman]

What is the role of the .index files? Are there side-effects deleting them? If not, could we do this automatically in our shapefile download script?

We should automatically be building new ones as part of the download.

[sommerluk]

When building new ones (with default options: shapeindex *.shp) rendering is broken again.

[imagico]

@StyXman pointed out in imagico/poly_grid#1 (comment) that this might be specific to mapnik master and does not occur in released versions.

[StyXman]

It's also broken for mapnik-3.1.0, which is the one that kosmtik is using. See kosmtik/kosmtik#295 and #3717

Rendering those shapefiles with python-mapnik works for me.

[imagico]

Does it work fine without shapeindex as @sommerluk described?

[StyXman]

Indeed it does with mapnik-3.1.0. Interesting...

[sommerluk]

This could maybe be a missmatch of the Mapnik package of the Linux distribution (which provides also the program that generates the index files) and the node-mapnik version used by kosmtik?

[sommerluk]

Given that #4092 has been merged in the meantime, which loads the shapefiles in the database. Does this have an effect on this issue. Can it be closed?

[pnorman]

We were never able to solidly isolate what was going on, and since the issue was reported, we've completely redone how shapefiles are handled, loading them into the DB.