Vector patterns

[pitdicker]

This PR is an attempt to vectorize the remaining patterns as svg. See #2045. See also part 1 in #4458.
It took six weeks, but I now believe this would be save to merge.

I focused on mostly on these concerns:

• The SVG files should faithfully replicate the raster images (al least initially).
• The SVG files should be simple, to help with performance.
• There are issues in Mapnik that require workarounds.

Pattern generation script

The first two commits add a makefile and a pattern generation script. The intention is not to replace jsdotpattern. I believe jsdotpattern to be very useful because of its abilities in generation random but equal-spaced patterns, and because of its interactive, visual feedback. The goal of the script is to generate SVG files that are as simple in structure as possible.

SVG patterns can actually be quite readable text files. And patterns are easy: they are just one or more shapes repeated at many positions. The shape can be redrawn at every position, or defined once and referenced each time. Some of the patterns before this PR took noticeable time to load in a browser of viewer, which are now near instantaneous. I like to imagine simple SVGs will avoid future performance problems somewhere that would make vector patterns again unusable. Still, a large pattern-heavy area is noticably slower to render in Kosmtik than one without patters.

For every pattern I recorded the shape, the style (color, opacity, stroke-width etc.), and positions in a text file. generate_pattern.awk is a somewhat primitive script that converts it to svg. Changing the color of a pattern should be as easy as changing the color in the txt file and running make.

Simplifications to patterns

Most shapes are hand-optimized to be as simple as possible while looking the same. A line in the wetland patterns for example is no longer a shape with coordinates that look like a rectangle, but just a line with a thickness.

Descriptions per pattern:

Dot-like patterns

I like the fine detail in the patterns beach.svg, beach_coarse.svg, reef.svg, scree.svg. But it makes the patterns very heavy, and I expect prone to rasterizer differences. Therefore the fine shapes are replaced with simple dots. At regular and 2× resolutions it is near impossible to spot the change.

The original beach.svg pattern had 6 different grain shapes, each in 13 orientations. To keep some of the variability of the original pattern I used three sizes of dots, that match the area of the original grain. And the dots are still placed at non-integer positions, making every one of them rasterize a little different.

Except for their color beach_coarse.svg (#969696) and reef.svg (#549ccd) are identical to scree.svg (#cbc9c6). The pattern was somewhat simpler than beach.svg, as it used the same 6 grain shapes but all in just one orientation. The simplified pattern uses two sizes of dots and a short line to imitate the grain shapes.

salt_dots.svg got a similar treatment. The shapes are no longer a shape made up of straight lines that somewhat resemble a circle, but just simple dots with a size.

Forest patterns

The various leaftype_* patterns were already vector patterns. I simplified them a bit by converting the shapes from outlines intended to look like 1px strokes, to true strokes. Note that because of rounding the needleleafed symbols had diagonal lines that were slightly thinner than 1px, especially near the top. They are now exactly 1px (still properly pixel-aligned though).

scrub.svg and wetland are also converted to a pattern with simple strokes.

Wetland patterns

The wetland_* patterns have a 2px casing around the symbols, rounded to the nearest pixel boundary. And if there is only 1 pixel left of the covered wetland pattern, that pixel should be removed. This is hard to reproduce as vector images. generate_pattern.awk has some complexity to deal with this, but only in a very basic way. It is hard-coded for the wetland pattern, and every shape has the outline of its casing manually recorded in the text file.

Outline of the various shapes (created by adding a copy of the shape with stroke-width="4", and eyeballing the closest pixel boundary):

Other patterns

salt_pond and quarry.svg were already good vector patterns. I create text files to make it possible to generate them with generate_pattern.awk. salt_pond was simplified a bit. For quarry.svg I restored a clipped corner at the top left of the symbol, and converted some bezier curves that were straight lines to regular lines.

Rock pattern

(copied from the readme) Random pattern of 12 complex shapes, each with 13 possible rotations. The positions are close enough for the shapes to overlap. Every shape has both a fill, and a casing of 0.6px around it to hide part of the underlying shapes.

Overlapping shapes, with a casing as outline in between, don't work with a pattern with opacity. That is why rock.svg is the only pattern that needs to include both a foreground color (#cfcdca) and background color (#eee5dc).

One option to include casing is to repeat every shape twice: once with a stroke of 1.2px with the background color, and then one with only a fill on top of that. An alternative is to offset the path of the shape by 0.3px, and use that single shape with both a fill and a stroke of 0.6px.

The alternative almost halves the complexity for the rasterizer, so I have created new shapes with offset in Inkscape. A preview of the various original and offset shapes is available in rock_outlines.svg.
The pattern is now much lighter, but still very slow to rasterize.

Other changes to patterns

Besides the changes mentioned before, I made a few small but more visible changes.

• The wetland.svg pattern had a different opacity from patterns such as wetland_bog.svg. I changed wetland.svg to be the same, in line with this comment. Instead of 80% opacity I made the line 0.8px wide, which has the same effect when rasterized at normal and 2× resolution, but seemed neater to me as vector image.
• I changed the wetland_bog.svg pattern to rasterize sharper. It has two horizontal lines under the grass which are 0.75px thick, and separated by 1px. I moved the pattern 0.25px down, aligning the 1px separation with tile pixels.
• Uniformize swamp rendering with forest/wood #3051 hit a couple of problems changing the wetland_swamp pattern (the instructions in wetland.md are not entirely correct). I believe the intention was to change the symbols, but not to change the position of the shapes. I restored the pattern to use the old positions. Also the color and opacity didn't match those of the forest patterns, which are now fixed.

Mapnik workarounds

My first plan was to make a patch to Mapnik, hope to get it merged, wait for a new release, and wait for it to be widely deployed before the new vector patterns could be used. But it is quite doable to apply some workarounds and get usable patterns right now.

Workarounds to Mapniks auto-cropping

In all cases were the geometry of the pattern does not fall exactly on the four edges of the pattern, Mapniks auto-cropping causes trouble.

1. A workaround for patterns were the symbols don't cross the tile boundary is adding an invisible rectangle to the SVG.
2. A pattern with symbols crossing the tile boundary should be clipped. Mapnik doesn't support SVG clip paths, the geometry itself has to be altered.
3. An alternative is modifying a pattern so that symbols don't cross the tile boundary.

I don't prefer the second option because clipping the geometry is a difficult operation (requires Inkscape), and because it easily leads to 5× more complex files. The shapes can't just be referenced at coordinates, but have to be drawn at that position. There is some optimization possible by only inserting shapes that are not near a tile boundary though.

The third option turned out to work surprisingly well, using two techniques: moving the base point of the pattern, and for some patterns moving a few symbols out of the way.

Moving the base point of the pattern

The idea is that a pattern may contain horizontal or vertical lines that are clear; where a line would not intersect any symbols. If the patterns was moved a bit horizontally or vertically, it could fit on a tile without any symbols crossing the tile boundary. I see this as a nice solution, as moving the base point of pattern doesn't make it look any less random.

scrub fits good on a tile when translated by -45,-3. Example:

The patterns leaftype_broadleaved, leaftype_leafless, leaftype_mixed and leaftype_needleleaved fit nicely by translating the pattern by -1,-66. For the leaftype_unknown I applied the same translation, but also moved the symbols with an extra -1,1. Now their symbols align with their center with the center of the other patterns.

The wetland-* patterns are translated by -24,47. As it are big patterns with 512×512px, they don't have any horizontal lines and vertical lines clear of symbols. But with this translation, the necessary adjustments are small or not noticeable unless you put the old and new pattern on top of each other. 3 symbols had to move 1px, and another 3px.

The wetland-* patterns had another small issue: ideally the symbols of all patterns should line up with each other. I moved them mangrove, marsh and reed 1px so that all symbols are centered on their insertion points, fit inside the same bounding box, and have the grass leaves in the same position.

The symbols in the wetland_mangrove pattern are moved 1px up, to align exactly with the other patterns.

The beach and reef patterns have a huge amount of dots, ±3100 and ±5900 respectively. With the help of some scripts I found a place to slice the patterns so that they require minimal adjustments. For beach for example 15 dots had to move, but by an average 0.16px. This is really not noticeable.

Clipping

The wetland pattern is easy to clip, as it is just a bunch of horizontal lines. generate_pattern.awk can handle the clipping of simple horizontal and vertical lines.

Invisible geometry

Just about all patterns still need invisible geometry to ensure Mapnik doesn't crop it by a sub-pixel amount.

Rock pattern

This is the one pattern I couldn't apply workarounds to. So for now that pattern remains a raster image. Even Inkscape seems to find the pattern to hard to clip :-D. Besides that, the method of using a border as casing is (almost) incompatible with clipping the geometry. Realistically, the rock pattern is going to need a new design if it ever is to be used as a vector pattern.

Documentation and organization

Like in #4458 I have put all patterns in a patterns\ directory. The generate_pattern.awk script lives in scripts\, and the various text files that serve as source in patterns\src\.

There is a patterns\src\README.txt (rendered) in which I have made an effort to document the patterns and the script. Were there was a jsdotpattern sequence available and correct it is collected in that file. I have also added a couple of preview files: forest.svg, rock_outlines.svg and wetland_casings.svg.

Testing

• The patterns are tested with Inkscape and viewers, and proof-rendered with Mapniks svg2png.
• I have messed up my landcover.mss to apply every pattern over a large area. This was tested with plain Mapnik and as tiles with Kosmtik.

Preview

Patterns on current master

Patterns before applying workarounds

Rendered with Mapniks svg2png so you can see some of the issues.

Full SVG: https://gist.githubusercontent.com/pitdicker/83ee78ac3b5acc0f5717e97f9092675c/raw/2ca307c7e8c162202c4746172e046773c8727805/patterns_index.svg

Patterns with workarounds

Also rendered with svg2png.

https://gist.githubusercontent.com/pitdicker/8def884c1ea56dfddd4d6d46f7f2852f/raw/e33e27f51ba72a7e926459afb4a5eb813c992dbe/patterns_index_workarounds.svg

[pitdicker]

I finally got to learn a little python yesterday and translated the generate_pattern script.

Now I have to rebase on master and add another pattern, because the golf features pr added a new one. I'll add a comment when ready.

[pitdicker]

Updated to include the new golf_rough pattern:

Full SVG: https://gist.githubusercontent.com/pitdicker/468e4c5b125fbf0711268d237164e128/raw/e6cd937b9004d6e62424f5202da183d4656d3d95/patterns_index_workarounds_v2.svg

Instead of the existing 40x40px pattern for golf_rough, I based it on the beach pattern with a little larger dots. This way there is less visual repetition, and the pattern properly tiles.

[pitdicker]

Fixed merge conflicts.

[kocio-pl]

This is a huge work, thanks for this. Unfortunately, even reading this documentation will take me some time, but I plan to do it, then test the generator and how it all renders.

[imagico]

Thanks for looking over the patterns, this is a very good idea. I have not actually tested the changes yet in rendering but i looked at the implementation.

There are a few things in your PR i see critically:

• i would not like to introduce a dependency on make, in particular if it is as poorly motivated as here where 80 percent of the makefile functionality could be implemented much shorter in a script with a simple loop (most of the makefile targets are essentially identical).
• it is good style we follow so far that scripts have some basic documentation of what they do as a header comment, print out usage instructions when called with -h parameter and add information how the file was generated and a warning not to hand edit it to the generated files.
• you are inventing a new file format with unclear motivation (why a new format if there are plenty of established formats suitable for storing information like this). It is also unclear how exactly the 20+ files in that format were generated and how they are to be maintained in the future.
• the purpose of most of the PR in terms of substance in changes remains unclear to me, in particular the changes to the patterns and the move to generating SVGs from the new file type via script. The declared purpose is to move from PNG patterns to SVG patterns for where currently we use PNG patterns. But that is only a small part of what you are proposing to change and for the rest it is largely unclear to me what the benefit of the changes is meant to be. If complex pattern SVGs are a performance issue in rendering that would put into question Convert patterns to SVG #2045 overall - see Convert patterns to SVG #2045 (comment). I always worked under the assumption that Paul is right in Convert patterns to SVG #2045 (comment).
• i would be fairly strongly against manually editing automatically generated relaxed random and periodic patterns to comply with some poorly motivated need for the pattern symbols not to intersect the bounds of the pattern. This is not sustainable in maintainance and not practically possible with patterns in general. The whole point of moving to patterns auto-generated with periodic boundary conditions was to eliminate the need to manually tune patterns to appear uniformly structured but instead have an algorithm take care of that. jsdotpattern handels the case of symbols intersecting the pattern bounds and i have not seen any reports of this causing problems.

As i mentioned in #4453 (comment) what would be highly desirable to make this style easier to maintain, more easy to use for applications with varying rendering resolutions in light of Mapniks arbitrary choice to differently scale PNG and SVG patterns and in particular to make it easier to customize would be to unify all patterns to SVG format, depending on if #2750 is actually resolved (i don't know if it is - i have not seen it in the map more recently but i also am not aware of any change in Mapnik that is supposed to have fixed it) to move to directly using the SVGs by default and to color the patterns via script - in a similar way as @sommerluk does in #3399. Most of that work i have already done at the October 2018 Karlsruhe hack weekend - though i never submitted it here since we did not have consensus on if to use SVGs for patterns in light of #2750. I am completely fine with you taking a different approach here but i so far do not really see the benefit of most of the changes you make beyond simple conversion/generation of patterns in/into SVG format.

[quinncnl]

Nice work! Thank you. I will cherry pick this change for tracesmap.com

[imagico]

Adding a formal review: What i think needs changing here (for more details and reasoning see above):

• not introduce a dependency on make when the same logic could be implemented in a more compact way with a script.
• documenting the python script in the header, making it self explaining and preferably indicate in the files it generates that and how they were script generated.
• avoid introducing a new file format but make use of existing standards (we so far have mostly used yaml for parameter files).
• developers should have a clear and clearly documented path how to make changes and customizations in the patterns we use. Most of the patterns are generated with jsdotpattern and we supply parameters how to do so (or the original SVG) and a path how to get from there to the pattern files used in the style. Any modification of that system would need to similarly provide a documented path, preferably one that does not require duplicating significant manual steps every time you make adjustments.

Personally i would like to see us having a possibility to switch between use of PNG pattern files and SVG pattern files automatically because of the arbitrary differences in how Mapnik uses both at different rendering resolutions. But that is just my personal preferences, not a request i make here as maintainer.

[dch0ph]

Many thanks for working on this. I'm interested in printed maps for walking, mostly exporting boxes at zoom levels 14 and 15 to PDF. The current bitmap symbols don't work well for print applications.

I haven't looked at the code in detail. As I read it, the dependence on make has gone, but there is still a dependence on awk, which could be pythonified? I agree with @imagico that the current PR is introducing a lot of complexity [the rock pattern, in particular, seems extremely complex]. In particular, I think it would be difficult for new contributors to develop new pattern symbols.

As a practical example, I render a cow pattern on landuse=meadow, meadow=pasture (useful to know in advance if there are animals in a field):

This pattern is not very well designed, and it would be great to have a tool which took a cow SVG and created a pattern from it.

The other practical problem is that a pattern that works at Z14/15 doesn't look great at high zoom:

Too Many Cows!

So it would be good to have an automated way to generate "low zoom" and "high zoom" versions.

Overall, it would be great to have a YAML-driven configuration for pattern symbol generation. I also dislike the .txt files - it would be ideal to have a Python script that would read the YAML and expand SVGs into tile pattern SVGs. There are a number of Python packages for parsing / manipulating paths from SVGs, and we already have the YAML-parsing framework in other scripts, so I don't think this is out of reach.