Tuning and optimizations

[Phyks]

• https://github.com/cquest/osmfr-cartocss/blob/master/tuning/tuning.py can be used to tweak the database calls.

• https://github.com/cquest/osmfr-cartocss/blob/master/tuning/cleanxml.py could be used to investigate and fix why the resulting mapnik.xml (after running Kosmtik) is so large.

[Phyks]

We should check db queries with https://github.com/cyclosm/cyclosm-cartocss-style/blob/master/scripts/tuning.py for low zoom levels, queries are very long… :/

[Phyks]

Current Mapnik XML file size

• openstreetmap-carto: 3M
• osmfr-cartocss: 4.3M
• cyclosm-cartocss-style: 5.8M

CyclOSM is not very good :/

I pushed a script to investigate it further in 2bb711c.

[Phyks]

This is very bad since CartoCSS will output in the Mapnik XML all possible values for the associated feature.

Therefore, due to this line, the amenities-points/amenities-poly styles go from 166 rules to 1551 rules!

Full output of the script is

+-------------------------------------+------------+
|              Style name             | # of rules |
+-------------------------------------+------------+
|            amenities-poly           |    1551    |
|           amenities-points          |    1550    |
|         tunnel-outline_left         |    1096    |
|       roads_high-outline_left       |    1096    |
|         bridge-outline_left         |    1096    |
|         tunnel-outline_right        |    1095    |
|       roads_high-outline_right      |    1095    |
|         bridge-outline_right        |    1095    |
|              roads_high             |    914     |
|            bridge-inline            |    914     |
|            tunnel-inline            |    850     |
|           roads-text-name           |    638     |
|              area_label             |    478     |
|          highway_area_label         |    462     |
|            tunnel-surface           |    308     |
|          roads_high-surface         |    308     |
|            bridge-surface           |    308     |
|            bicycle_routes           |    210     |
|            tunnel-outline           |    145     |
|          roads_high-outline         |    145     |
|            bridge-outline           |    145     |
|         amenities-poly-text         |     67     |
|        amenities-points-text        |     65     |
|             tunnel-line             |     61     |
|             bridge-line             |     61     |
|            waterway_high            |     30     |
|        tunnel-steps_ramp_left       |     28     |
|               landuse               |     28     |
|        bridge-steps_ramp_left       |     28     |
| placenames-medium-medium-importance |     26     |
|  placenames-medium-high-importance  |     26     |
|             landuse_gen1            |     26     |
|             landuse_gen0            |     26     |
|            capital-names            |     26     |
|            roads-text-ref           |     24     |
|           paths-text-name           |     24     |
|      roads_high-steps_ramp_left     |     20     |
|               aeroway               |     19     |
|              roads_med              |     13     |
|        bicycle_routes_labels        |     12     |
|               barriers              |     12     |
|              addresses              |     12     |
|         cycle-junction-nodes        |     10     |
|            country-names            |     10     |
|        roads_high-rail_line2        |     9      |
|    placenames-small-neighborhood    |     9      |
|          bridge-rail_line2          |     9      |
|             trees-canopy            |     8      |
|              roads_low              |     8      |
|         roads_high-rail_line        |     8      |
|           landuse_overlays          |     8      |
|           bridge-rail_line          |     8      |
|             waterway_med            |     7      |
|            waterway_label           |     7      |
|            train_stations           |     6      |
|              contours50             |     6      |
|             contours100             |     6      |
|             waterway_low            |     5      |
|         turning_circle_fill         |     5      |
|         turning_circle_case         |     5      |
|       placenames-small-village      |     5      |
|              buildings              |     5      |
|       roads-text-ref-low-zoom       |     4      |
|       placenames-small-quarter      |     4      |
|   placenames-medium-low-importance  |     4      |
|              contours10             |     4      |
|             trees-trunk             |     3      |
|             state-names             |     3      |
|      oriented-highway-amenities     |     3      |
|              contours20             |     3      |
|       placenames-small-suburb       |     2      |
|                cliffs               |     2      |
|        amenities-poly-nozzle        |     2      |
|       amenities-points-nozzle       |     2      |
|              water_gen1             |     1      |
|              water_gen0             |     1      |
|                water                |     1      |
|               land-low              |     1      |
|              land-high              |     1      |
|             hillshade90             |     1      |
|            hillshade5000            |     1      |
|             hillshade500            |     1      |
|            hillshade1000            |     1      |
|           cliffs-man_made           |     1      |
|         amenities-poly-basin        |     1      |
|        amenities-points-basin       |     1      |
+-------------------------------------+------------+

EDIT: As a comparison, the result on the openstreetmap-carto style is https://gist.github.com/Phyks/b46fbf49d417a715ffe2c08b319c50b9.

[Phyks]

amenities-{poly,points} layer was handled in 770b00d. Now with about 160 rules.

Generated Mapnik XML file went down from 5.8M to 4.7M.

New numbers of rules are:

+-------------------------------------+------------+
|              Style name             | # of rules |
+-------------------------------------+------------+
|         tunnel-outline_left         |    1096    |
|       roads_high-outline_left       |    1096    |
|         bridge-outline_left         |    1096    |
|         tunnel-outline_right        |    1095    |
|       roads_high-outline_right      |    1095    |
|         bridge-outline_right        |    1095    |
|              roads_high             |    914     |
|            bridge-inline            |    914     |
|            tunnel-inline            |    850     |
|           roads-text-name           |    638     |
|              area_label             |    478     |
|          highway_area_label         |    462     |
|            tunnel-surface           |    308     |
|          roads_high-surface         |    308     |
|            bridge-surface           |    308     |
|            bicycle_routes           |    210     |
|            amenities-poly           |    187     |
|           amenities-points          |    186     |
|            tunnel-outline           |    145     |
|          roads_high-outline         |    145     |
|            bridge-outline           |    145     |
|        amenities-points-text        |     69     |
|         amenities-poly-text         |     65     |
|             tunnel-line             |     61     |
|             bridge-line             |     61     |
|            waterway_high            |     30     |
|        tunnel-steps_ramp_left       |     28     |
|               landuse               |     28     |
|        bridge-steps_ramp_left       |     28     |
| placenames-medium-medium-importance |     26     |
|  placenames-medium-high-importance  |     26     |
|             landuse_gen1            |     26     |
|             landuse_gen0            |     26     |
|            capital-names            |     26     |
|            roads-text-ref           |     24     |
|           paths-text-name           |     24     |
|      roads_high-steps_ramp_left     |     20     |
|               aeroway               |     19     |
|              roads_med              |     13     |
|        bicycle_routes_labels        |     12     |
|               barriers              |     12     |
|              addresses              |     12     |
|         cycle-junction-nodes        |     10     |
|            country-names            |     10     |
|        roads_high-rail_line2        |     9      |
|    placenames-small-neighborhood    |     9      |
|          bridge-rail_line2          |     9      |
|             trees-canopy            |     8      |
|              roads_low              |     8      |
|         roads_high-rail_line        |     8      |
|           landuse_overlays          |     8      |
|           bridge-rail_line          |     8      |
|             waterway_med            |     7      |
|            waterway_label           |     7      |
|            train_stations           |     6      |
|              contours50             |     6      |
|             contours100             |     6      |
|             waterway_low            |     5      |
|         turning_circle_fill         |     5      |
|         turning_circle_case         |     5      |
|       placenames-small-village      |     5      |
|              buildings              |     5      |
|       roads-text-ref-low-zoom       |     4      |
|       placenames-small-quarter      |     4      |
|   placenames-medium-low-importance  |     4      |
|              contours10             |     4      |
|             trees-trunk             |     3      |
|             state-names             |     3      |
|      oriented-highway-amenities     |     3      |
|              contours20             |     3      |
|       placenames-small-suburb       |     2      |
|                cliffs               |     2      |
|        amenities-poly-nozzle        |     2      |
|       amenities-points-nozzle       |     2      |
|              water_gen1             |     1      |
|              water_gen0             |     1      |
|                water                |     1      |
|               land-low              |     1      |
|              land-high              |     1      |
|             hillshade90             |     1      |
|            hillshade5000            |     1      |
|             hillshade500            |     1      |
|            hillshade1000            |     1      |
|           cliffs-man_made           |     1      |
|         amenities-poly-basin        |     1      |
|        amenities-points-basin       |     1      |
+-------------------------------------+------------+

[Phyks]

There were some issues with bicycle routes generating useless rules as well. Now fixed.

The real issues here are:

+-------------------------------------+------------+
|              Style name             | # of rules |
+-------------------------------------+------------+
|         tunnel-outline_left         |    1096    |
|       roads_high-outline_left       |    1096    |
|         bridge-outline_left         |    1096    |
|         tunnel-outline_right        |    1095    |
|       roads_high-outline_right      |    1095    |
|         bridge-outline_right        |    1095    |
|              roads_high             |    914     |
|            bridge-inline            |    914     |
|            tunnel-inline            |    850     |
|           roads-text-name           |    638     |
|              area_label             |    478     |
|          highway_area_label         |    462     |
|            tunnel-surface           |    308     |
|          roads_high-surface         |    308     |
|            bridge-surface           |    308     |
+-------------------------------------+------------+

This is a huge amount of rules, not sure if they match the amount of rules we actually have in the CartoCSS style or if there are useless rules generated in the Mapnik XML file…

[Phyks]

With the latest set of commits, I now have a mapnik.xml file of 3.3M.

Still some work to do on the roads style:

+-------------------------------------+------------+
|              Style name             | # of rules |
+-------------------------------------+------------+
|              roads_high             |    852     |
|            bridge-inline            |    852     |
|            tunnel-inline            |    796     |
|           roads-text-name           |    638     |
|         tunnel-outline_left         |    516     |
|       roads_high-outline_left       |    516     |
|         bridge-outline_left         |    516     |
|         tunnel-outline_right        |    515     |
|       roads_high-outline_right      |    515     |
|         bridge-outline_right        |    515     |
|              area_label             |    475     |
|          highway_area_label         |    465     |
|            tunnel-surface           |    252     |
|          roads_high-surface         |    252     |
|            bridge-surface           |    252     |
+-------------------------------------+------------+