[WIP] Basic schema enhancements for Felt basemap #37

nvkelso · 2023-05-26T05:30:56Z

@bdon this captures most of what we talked about Monday.

TODO:

Places layer: population_rank isn't working as expected
Places layer: we need a better value to use for client side sorting. Normally this would be the Natural Earth min_zoom (which solves problems like San Francisco having a smaller population than Oakland)
Fussy linting nits

…_rank attr

…asic network support

nvkelso · 2023-05-26T07:19:06Z

tiles/src/main/java/com/protomaps/basemap/layers/Places.java

+              200,
+              0};
+
+      for (int i = 0; i < pop_breaks.length; i++) {


This isn't working in Java, but works fine in Python. Help!

I think you want to break once you find a result in the array that matches, right?

it seems like population_rank at the top is assigned to 0 or the OSM value. Since there's not really a case where it's nullable I don't think you need to use the Integer type, just plain int should be OK.

LOL, yes. Fixed via 5b62ac0

base/src/base_layers.ts

nvkelso · 2023-05-26T17:53:40Z

tiles/src/main/java/com/protomaps/basemap/layers/PhysicalLine.java

        .setZoomRange(12, 15);

+      if( sf.hasTag("intermittent", "yes" ) ) {
+        feat.setAttr("intermittent", 1);


For way 316401077 in israel-and-palestine GeoFabrik:

@bdon up to you if you want this to be a boolean in the MVT or an int

Tilezen uses a boolean so let's converge on that later

nvkelso · 2023-05-26T17:56:25Z

tiles/src/main/java/com/protomaps/basemap/layers/Pois.java

+      } else if (sf.hasTag("shop", "grocery", "supermarket" )) {
+        kind = sf.getString("shop");
+      } else if (sf.hasTag("tourism", "attraction", "camp_site",  "hotel" )) {
+        kind = sf.getString("tourism");


nvkelso · 2023-05-26T18:21:06Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

+      if( shield_text != null ) {
+        if( shield_text.contains("US ") ) {
+          shield_text = shield_text.replaceAll("US ", "");
+          network_val = "US:US";


This is the Tilezen way of specifying the network values, which is mostly true to OSM's values.

nvkelso · 2023-05-26T18:21:42Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

+          network_val = "US:US";
+        } else if( shield_text.contains("I ") ) {
+          shield_text = shield_text.replaceAll("I ", "");
+          network_val = "US:I";


nvkelso · 2023-05-26T18:22:34Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

+          shield_text = shield_text.replaceAll("I ", "");
+          network_val = "US:I";
+        } else {
+          network_val = "other";


nvkelso · 2023-05-26T19:00:10Z

base/src/base_layers.ts

@@ -1187,10 +1231,16 @@ export function labels_layers(source: string, c: Theme): any[] {
      "source-layer": "places",
      filter: ["==", "pmap:kind", "city"],
      layout: {
+        "symbol-sort-key": ["number", ["get", "pmap:min_zoom"]],


This allows San Francisco to win over Oakland (which otherwise has larger population):

You can see the Oakland dot (symbolized in a different layer) but the San Francisco text wins.

nvkelso · 2023-05-26T19:01:09Z

base/src/base_layers.ts

+        "text-anchor": {
+          stops: [
+            [7, "left"],
+            [8, "center"],


Otherwise text was still align left align on zoom 8+ when the townspot goes away:

nvkelso · 2023-05-26T19:07:06Z

tiles/src/main/java/com/protomaps/basemap/layers/Pois.java

+          sf.hasTag("railway", "station")))
+    {
+      String kind = "other";
+      if (sf.hasTag("aeroway", "aerodrome" )) {


Almost all aerodromes are ways and relations in OSM

nvkelso · 2023-05-26T19:08:53Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

      String highway = sourceFeature.getString("highway");
+      String shield_text = sourceFeature.getString("ref");
+      String network_val = sourceFeature.getString("network");
+      shield_text = (shield_text == null ? null : shield_text.split(";")[0].replaceAll("\\s", ""));


Shields don't look good on very long ref values (like US 101;CA 1), so we split on the ; and because sprites generally should be smaller not bigger we also strip the internal whitespace.

nvkelso · 2023-05-26T19:09:40Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

+      String network_val = sourceFeature.getString("network");
+      shield_text = (shield_text == null ? null : shield_text.split(";")[0].replaceAll("\\s", ""));
+      if( shield_text != null ) {
+        if( shield_text.contains("US ") ) {


This is cheap way of imputing the network without walking all the relations. The output for tile user is the same, and can be upgraded to rels later without changing the contract.

nvkelso · 2023-05-26T19:12:49Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

+          network_val = "other";
+        }
+      }
+      Integer shield_text_length = (shield_text == null ? null : shield_text.length());


Once we've done all our transforms on the string, we calculate the shield_text_length for later export as ref_length... This makes styling in MapLibre much easier.

nvkelso · 2023-05-26T19:13:42Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

@@ -31,29 +48,81 @@ public void processFeature(SourceFeature sourceFeature, FeatureCollector feature
        .setAttrWithMinzoom("bridge", sourceFeature.getString("bridge"), 12)
        .setAttrWithMinzoom("tunnel", sourceFeature.getString("tunnel"), 12)
        .setAttrWithMinzoom("layer", sourceFeature.getString("layer"), 12)
-        .setAttrWithMinzoom("oneway", sourceFeature.getString("oneway"), 14)
-        .setAttr("ref", sourceFeature.getString("ref"));


We don't always want export the ref for all zooms... instead we do it by highway types (below). This reduces tile size.

nvkelso · 2023-05-26T19:14:16Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

        OsmNames.setOsmNames(feat, sourceFeature, 13);
      } else if (highway.equals("residential") || highway.equals("service") || highway.equals("unclassified") ||
-        highway.equals("road")) {
+        highway.equals("road") || highway.equals("raceway")) {


This is for speedway roads.

nvkelso · 2023-05-26T19:26:55Z

tiles/src/main/java/com/protomaps/basemap/layers/Pois.java

+
+      // try first for polygon -> point representations
+      if (sf.canBePolygon()) {
+        var poly_label_position = features.pointOnSurface(this.name())


If it's a polygon, then we need to get a point on it's surface for export

TODO: This needs QA/QC

tiles/src/main/java/com/protomaps/basemap/layers/Transit.java

nvkelso · 2023-05-26T19:28:26Z

tiles/src/main/java/com/protomaps/basemap/layers/Water.java

@@ -43,9 +43,18 @@ public void processFeature(SourceFeature sf, FeatureCollector features) {
        .setAttr("leisure", sf.getString("leisure"))
        .setAttr("water", sf.getString("water"))
        .setAttr("waterway", sf.getString("waterway"))
+        // Add less common attributes only at higher zooms


These props mostly apply to water lines, but there are exceptions for polygons, too, so are included here for parity.

nvkelso · 2023-05-26T19:28:47Z

tiles/src/main/java/com/protomaps/basemap/names/NeNames.java

@@ -0,0 +1,21 @@
+package com.protomaps.basemap.names;


New class so NE names can be transformed to look like OSM names

nvkelso · 2023-05-26T19:53:41Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

      String highway = sourceFeature.getString("highway");
+      String shield_text = sourceFeature.getString("ref");
+      String network_val = sourceFeature.getString("network");
+      shield_text = (shield_text == null ? null : shield_text.split(";")[0]);


We just want the first ref, not list of refs as is often done in OSM on ways.

Better way is to walk the relations and rank order them, but later...

nvkelso · 2023-05-26T19:54:12Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

+          network_val = "other";
+        }
+      }
+      shield_text = (shield_text == null ? null : shield_text..replaceAll("\\s", ""));


Shorter refs are better for display in map shields vis-a-vis sprite artwork.

nvkelso · 2023-05-26T21:58:48Z

base/src/base_layers.ts

+      "source-layer": "transit",
+      filter: [
+        "any",
+        ["==", "pmap:kind", "pier"],


nvkelso · 2023-05-26T22:49:59Z

tiles/src/main/java/com/protomaps/basemap/layers/Boundaries.java

      Integer adminLevel = Parse.parseIntOrNull(relation.getString("admin_level"));
+      Integer disputed = relation.hasTag("boundary", "disputed") ? 1 : 0;


TODO: This seems to always eval to 0 in error

@nvkelso what relations/area are you testing on? on observation with Overpass Turbo it seems many relations with boundary=disputed lack the type=boundary tag, eg https://www.openstreetmap.org/relation/13562322

The Line of Control in Cyprus (r/13562322)is an odd ball – it's admin_level is 3 for starters.

I was testing around Israel as the build there has a smaller bounding box than India or China and more admin_level = 2 features.

Israel - Egypt boundary way with relevant relations: w/95211614

Israel - Gaza boundary way with relevant relations: w/25057102

It seems to evaluate to 1 on this relation: https://www.openstreetmap.org/relation/1703814

bdon · 2023-05-29T13:53:28Z

Ran mvn spotless:apply which applies the code formatting

nvkelso · 2023-05-31T17:10:17Z

tiles/src/main/java/com/protomaps/basemap/layers/Roads.java

+
+        if (highway.equals("motorway")) {
+          feat.setAttrWithMinzoom("ref", shield_text, 7)
+            .setAttrWithMinzoom("ref_length", shield_text_length, 7)


@bdon up to you what you want to call the exported shield_text_length. In Tilezen it's shield_text_length but other schemas call it ref_length.

+1 to matching tilezen

nvkelso · 2023-05-31T17:57:15Z

README.md

 cd compare
 npm run serve
 ```

+5. Linting to apply code formatting
+
+```shell


When adding the new linting section, IntelliJ IDEA CE recommends shell not sh so I updated the instances above, too.

nvkelso · 2023-05-31T17:59:34Z

tiles/src/main/java/com/protomaps/basemap/layers/Boundaries.java

@@ -30,6 +30,7 @@ public void processFeature(SourceFeature sf, FeatureCollector features) {
      List<OsmReader.RelationMember<AdminRecord>> recs = sf.relationInfo(AdminRecord.class);
      if (recs.size() > 0) {
        OptionalInt minAdminLevel = recs.stream().mapToInt(r -> r.relation().adminLevel).min();


NOTE: This looks like it prevents the City and County of San Francisco from appearing as a locality boundary line, instead it only appears as a county boundary line. An odd case, but this is true for several other "global cities" in the USA and elsewhere, too.

tiles/src/main/java/com/protomaps/basemap/layers/Landuse.java

nvkelso added 10 commits May 25, 2023 16:17

add disputed boundaries handling

0920a06

add exotic water tags and min pix size

3ea5211

add provinces, add fallback populations by place type, add population…

17f8ddd

…_rank attr

add leisure, add aerodrome, add pmap:kind coallese

e4c53c6

add ref_length, split other into path and other, add pmap:link, add b…

a44c44d

…asic network support

add pier lines

0ed7b0c

bug fixes

c643234

move water farther down, add pier lines, add poi labels

1077301

add calcualted network, clean ref (first network only)

53c8ac7

piers are man_made (not highway)

6825103

nvkelso commented May 26, 2023

View reviewed changes

just under industrial

14d0824

nvkelso commented May 26, 2023

View reviewed changes

base/src/base_layers.ts Outdated Show resolved Hide resolved

draw pier poly above water

3b55d6a

nvkelso commented May 26, 2023

View reviewed changes

base/src/base_layers.ts Show resolved Hide resolved

nvkelso commented May 26, 2023

View reviewed changes

base/src/base_layers.ts Show resolved Hide resolved

nvkelso commented May 26, 2023

View reviewed changes

nvkelso added 7 commits May 26, 2023 11:54

fix pier bug, city sort on min_zoom and fix anchor bug

d8258ee

register Natural Earth with Places layer

43eb247

refactor disputed (for bug)

53ce164

add Natural Earth places at low zooms only

74d4578

allow polygon centroid POI labels

347f121

strip whitespace from ref, and only set ref by zoom and kind

0079233

add NeNames processing class

e675da6

nvkelso commented May 26, 2023

View reviewed changes

nvkelso added 2 commits May 26, 2023 12:21

even if an other road type, still export ref at higher zooms

1e2b1c5

even if an other road type, still export ref at higher zooms

71c827a

nvkelso commented May 26, 2023

View reviewed changes

tiles/src/main/java/com/protomaps/basemap/layers/Transit.java Show resolved Hide resolved

nvkelso commented May 26, 2023

View reviewed changes

strip after network calc

4db8422

nvkelso commented May 26, 2023

View reviewed changes

nvkelso added 2 commits May 26, 2023 14:29

typo

b22c1ea

improve pier widths

8252e77

nvkelso commented May 26, 2023

View reviewed changes

base/src/base_layers.ts

"source-layer": "transit",

filter: [

"any",

["==", "pmap:kind", "pier"],

Copy link

Collaborator Author

nvkelso May 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nvkelso commented May 26, 2023

View reviewed changes

formatting

aeef7ac

add linting note, formatting

f1858ae

nvkelso commented May 31, 2023

View reviewed changes

nvkelso added 2 commits May 31, 2023 10:53

int and break, per PR comments

5b62ac0

remove very small featues to declutter the map

5401363

nvkelso commented May 31, 2023

View reviewed changes

tiles/src/main/java/com/protomaps/basemap/layers/Landuse.java Show resolved Hide resolved

bdon merged commit f802d0d into protomaps:main Jun 1, 2023

nvkelso mentioned this pull request Jun 16, 2023

Round 2 enhancements for Felt #47

Merged

56 tasks

		Integer adminLevel = Parse.parseIntOrNull(relation.getString("admin_level"));
		Integer disputed = relation.hasTag("boundary", "disputed") ? 1 : 0;

[WIP] Basic schema enhancements for Felt basemap #37

[WIP] Basic schema enhancements for Felt basemap #37

Conversation

nvkelso commented May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvkelso May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvkelso May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdon commented May 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nvkelso commented May 26, 2023 •

edited

Loading

nvkelso May 26, 2023 •

edited

Loading

nvkelso May 26, 2023 •

edited

Loading