Create a tile server with the ability to render different layers

[BudgieInWA]

We want to make it as easy as possible to experience the result of the work done here, so;

Let's create a tile server that can render tiles right in rust or through some other means. That way, anyone can look at it in their tool of choice.

• find a library/mechanism for rendering shapes to raster nicely
• implement rendering osm2streets data to tiles (raster or vector) on request
• set up a tileserver api (WMS, TMS, etc.) that provides access to

In future, we could glue on a "download small region" step which downloads .osm xml from osm.org, or whatever data source you want to plug in. Rendering raster will be useful in JOSM too, so we don't have to do it in Java

[dabreegster]

find a library for rendering shapes to raster nicely

I've not worked with raster tiles much before, but I did briefly experiment with some vector tile formats + renderers back in https://github.com/dabreegster/leaflet-experiments/blob/main/NOTES.md. I was trying to shove the raw geometry from A/B Street -- which has one polygon for every single dashed white line on every street! -- into mbtiles. I remember even small areas were very slow and buggy. I never dug into the issues, or figured out if the vector tile renderers are batching draw calls, using quadtrees, etc.

But anyway, maybe using raster tiles is more compatible with the existing ecosystem and could avoid some of these problems entirely.

[BudgieInWA]

I really like raster tiles as a common denominator output, but that's not to say vector tiles aren't easier to work with sometimes. Maybe vector svg tiles get rasterised by an existing svg rasteriser.

[jakecoppinger]

Sorry, sent from wrong account previously.

If I understand correctly, is this issue for creating raster (or vector) tiles from what the interface of AB Street looks like? I'd find that incredibly useful as an OSM contributor - I'm not aware of any OSM tiles that show the amount of detail.

Any tips on where I could help contribute as someone who only has a little bit of Rust experience?

I was even thinking of running something like Webdriver/Selenium, navigating by adjusting the url (eg: http://play.abstreet.org/0.3.35/osm_viewer.html?system/us/seattle/maps/montlake.bin&--cam={zoom}/{lat}/{lon}) and "screenshotting" tiles.

I suppose I could also run some Typescript running a tileserver "proxy", taking in zoom/lat/long and on the backend navigating the browser to the correct spot and sending a screenshot of the right size back as the response.

I'm sure there must be better ways though!

[dabreegster]

I'm not exactly sure what Ben has in mind for this, but I think the overall goal is to get A/B Street-style detailed rendering everywhere in the world, with low effort for the user. Furthermore, let the result be displayed in the iD editor, on web maplibre maps, in JOSM, QGIS, etc -- anything that understands standard raster or vector tiles.

One approach, tiling the world somehow and pre-importing A/B Street maps for each of the tiles, is not looking easy so far. I'm playing around in acteng/atip-data-prep#1. It also would only solve the first part of the problem, not expressing things as raster or vector tiles.

Another approach is to make the client-side do it. You can go to https://a-b-street.github.io/osm2streets/#18.5/51.50687/-0.13376, import current view, and do everything lazily. This gets osm.xml extract from Overpass, calls a JS function in osm2streets (that ends up calling Rust code in this repo through WASM), and renders the result in Leaflet.

I don't have much familiarity with OSM tileservers. I think some of them can lazily generate raster or vector tiles and cache the results? If somebody requests a missing tile, they could do the same thing (query Overpass, pipe through osm2streets, take the resulting GeoJSON polygons and encode as raster or vector tiles).

Hopefully Ben has a better direction to point you. I don't think unfamiliarity with Rust should be a barrier here; the process of tiling OSM data, or dynamically fetching Overpass and calling osm2streets, or turning GeoJSON into raster or vector tiles, could all happen in any language

[BudgieInWA]

Hi @jakecoppinger, thanks for reaching out!

The idea here is to set up a Tile Server that is capable of rendering osm2streets data. I have updated the initial comment to be a bit clearer.

First, we need an actual Tile Server, TMS, or WMS or something. I don't know much about them, I've only configured things to use them, like this Leaflet tutorial. Then we need to generate the tiles somehow, and we have lots of options.

Using A/B Street

Using A/B Street as the renderer is a neat idea. That way we could have an "A/B Street" layer in other tools that leverages that existing work.

This is the least accessible option I think, for updating the rendering. A lower-overhead and more flexible mechanism for rendering is also desired.

Render in rust

Maybe the existing A/B Street rendering code is a good starting point for an in-rust render? This is not that accessible, because it would require writing rust to improve the rendering.

Use osm2streets output in Another renderer

This is the most tempting option for me. It is also how osm2streets is intended to be used in the long term.

We can implement - in the rust side - whichever geom output you need. osm2streets is already outputting a number of geojson "layers" that the Street Explorer is using.

You could start a new project in your favourite language/environment that renders just the road features on transparent tiles.

Or just straight into integrating the osm2streets road geometry into https://github.com/enzet/map-machine or something

[jakecoppinger]

I just had a good play around with Street Explorer and I think I'm getting a better understanding of it works now.

I can see it'd be possible to pre-generate (slowly!) a vector tileset by:

• Getting an OSM dump from Overpass turbo (maybe a local overpass instance so as not to overload the public server)
• Running this though A/B street to generate GeoJson (JS bindings from Rust on show at street-explorer/www/js/main.js:186 and the lines below)
• Generate vector tileset from this huge GeoJson file using https://github.com/mapbox/tippecanoe (or Mapbox Tiling Service for a quick demo on Mapbox infrastructure)

As far as lazy loading, I don't understanding vector tile schemas enough yet to understand if requests have bounds (like {zoom}/{x}/{y}) in the same way that raster tileservers do. (Edit: Yep, turns out they do work the same: https://docs.mapbox.com/api/maps/vector-tiles/)

In that case I could write a Typescript server that takes vector tile requests and either hits a cache or runs the above steps for the requested bounding box - which will be quicker than an entire city/suburb and scale nicely.

Does this sound like the right track? I'm looking forward to seeing A/B street basemaps everywhere 😀

I'll keep looking into how vector tileservers work and if I start a separate repo I'll make sure to share it back here.

[dabreegster]

Getting an OSM dump from Overpass turbo

The osm.xml can come from anywhere. If you're doing a huge area, I'd download from http://download.geofabrik.de/europe.html and maybe clip to a smaller area with osmium. If your typescript server can access a filesystem and shell out, this could be cheaper than hitting Overpass.

Does this sound like the right track?

It does to me! (Keeping in mind I don't understand tileserver workflow)

[dabreegster]

If you do go with geofabrik + osmium for clipping, I'd recommend something like this: osmium extract -b 2.25,48.81,2.42,48.91 large_area.pbf -o paris.xml -f osm,add_metadata=false
add_metadata strips out author, timestamp, and version from the XML, reducing the file size and making it a bit faster to parse.

We will hit issues at tile boundaries, because osm2streets clips roads crossing the boundary polygon. But as a first pass, we can get started with a tile server and see how things look, then figure out a good fix for this. (Maybe just clipping with a buffer?)

[jakecoppinger]

I've got a proof of concept vector tile server working built in Typescript & Koa! https://github.com/jakecoppinger/osm2streets-vector-tileserver

It:

• takes tileserver requests (eg http://localhost:3000/tile/16/60293/39332)
• Finds the bounding box of that tile
• Downloads the OSM XML from a local overpass turbo instance (instructions included) for that bounding box
• Calls osm2streets via the NodeJS bindings
• Generates the geojson (only the main geometry at the moment, very easy to improve though)
• Caches the output in a JS variable (needs improvement)

PRs are very welcome! I'm going to continue working on it and test it with a frontend soon (I haven't verified the GeoJSON output but it looks right).

[dabreegster]

Woo, this is so awesome to see! I just published https://www.npmjs.com/package/osm2streets-js and got it working from Svelte+Vite today (https://github.com/dabreegster/svelte_playground/blob/main/src/App.svelte). I've started https://github.com/dabreegster/osm2streets-vector-tileserver/tree/use_npm to start using the NPM package and simplify the build process. Hit a few snags, but I'll hopefully have time tomorrow to try again.

[jakecoppinger]

Brilliant! A snag I hit was wasm-pack build --dev --target web ./osm2streets-js caused all sorts of issues (from memory around fetch failing) which I eventually resolved with wasm-pack build --dev --target nodejs ./osm2streets-js.

I assume the npm build would be a frontend (web) build?

[dabreegster]

Yes, it's --target web, because I couldn't get the other options to work. I'm a little lost around all these options. I noticed there's no await init or similar in the controller right now -- is that somehow unnecessary with --target nodejs? That may be the current snag I was hitting

[jakecoppinger]

I noticed there's no await init or similar in the controller right now -- is that somehow unnecessary with --target nodejs

Yep exactly, I only found this out when I changed the target and init didn't work any more!

I'm currently trying to figure out how to add a vector layer to MapboxGL JS and actually test out this output of this. Looks like I'll need to convert the GeoJSON into a pbf - my understanding is vector tiles are Protobuf representations of GeoJson, possibly with some extra formatting.

It seems like there's lots of utilities around chopping up a big GeoJSON file to lots of tiles (https://github.com/mapbox/geojson-vt) but not much for a single tile. Maybe I just need to do a straight GeoJSON -> Protobuf conversion. If anyone knows more about this please let me know! (or I'll update when I find a solution).

If I can't convert a single GeoJSON file (or it ends up being too slow even with caching), I could modify the tileserver to do one big XML import centred on the requested tile, then chop up the GeoJSON into vector tiles and cache them. Boundaries might be tricky though!

[dabreegster]

Yep exactly, I only found this out when I changed the target and init didn't work any more!

Ohhh I am just now realizing the tile server is not running in a browser, it's using Node, which is why --target nodejs makes sense. I need to figure out what other projects do to publish to NPM -- maybe just have separate osm2streets-node and osm2streets-web?

I'm currently trying to figure out how to add a vector layer to MapboxGL JS and actually test out this output of this.

I think you can just directly add a new GeoJSON source, like acteng/atip@7e85649. Internally this gets turned into the MVT format or similar, because everything Map{box,libre} renders is split into tiles. My understanding is that all of the vector tile processing tools do this ahead-of-time for speed (so the client doesn't have to do this work, and so clients can later request just the piece of data they need). And also for cartographic generalization.

So ultimately we maybe want this tile server to hand out vector tiles, but as a very first cut, if it just returns GeoJSON, that should be displayable

Boundaries might be tricky though!

Ben and I have been working on that recently! This tile server could be a great test platform to figure it out. The osm2streets API changed recently to explicitly take a clipping boundary. For the tile server, I guess we'll just generate a square covering the tile. I'm not sure yet how to glue together adjacent tiles. We could tackle it at the rendering level and figure out a way to ignore some buffer around the boundary or render overlapping tiles and merge somehow. It'd also be useful to solve this at the StreetNetwork level -- if you have two objects covering adjacent (or partly overlapping) areas, is it possible to glue together into a single object and render that normally?

[BudgieInWA]

This progress is sounding exciting!

Boundaries might be tricky though!

I'm not sure yet how to glue together adjacent tiles.

I don't think we need to glue together adjacent tiles: that's what the client that is rendering the tiles needs to do. We do need to figure out how to clip our output geometry appropriately for each tile though. https://docs.mapbox.com/data/tilesets/guides/vector-tiles-standards/#clipping talks about tiles having a buffer outside of the "tile size". It says Mapbox renders the tile data into a canvas exactly the size of the tile. The data extending into the buffer is how artefacts are avoided at the boundaries. Here is an article with pictures of the process in action: https://blog.cyclemap.link/2020-01-25-tilebuffer/

Now it turns out that is exactly the same kind of problem that me and Dustin were talking about recently: The OSM road centerlines are "1-dimensional" (don't have any width) but the streets we generate are "2-dimensional" areas. We need to start with a buffer of OSM data outside of the area we want to render.

So, in order to produce a tile of size T with a buffer of size Bt we need to output Protobuf/GeoJSON covering T + Bt. In order to have sensible street geometry that covers T + Bt, we need to choose a SteetNetwork buffer Bs and start with OSM data covering T + Bt + Bs.

Does that make sense? Dustin, the buffer idea I was thinking about when we chatted was exactly the kind described in the linked article: osm2streets is essentially "rendering" the width-less OSM ways when it generates geometry, so we need to eventually incorporate the Bs buffer somewhere into the API or docs.

[dabreegster]

The buffer idea makes sense; that second link is great. So from the tileserver, we need to ask osm2streets for a slightly larger square to begin with -- meaning we pass a larger boundary to Overpass and for the GeoJSON boundary input. Then MapLibre / other renderers will just take care of visually clipping the results for us.

It'd also be helpful to clean up how we produce geometry near the edges (#136 (comment)), but not necessary to make progress here. A sufficiently large buffer would avoid issues there anyway

[jakecoppinger]

I've made some more progress on https://github.com/jakecoppinger/osm2streets-vector-tileserver:

• Now using the osm2streets-node npm package thanks to a PR by @dabreegster
• Naively building vector tiles from the GeoJSON for each request
  • not yet caching "deeper" generated tiles as the GeoJSON is not simplified as far as I know
• Managed to add the tileserver to QGIS
  • No colours though, I haven't learned how they work yet 😄
• Haven't figured out how to add a vector layer to Mapbox GL JS
• Really needs some caching & performance work

As far as buffers - I think the Overpass API returns the entire way if a single part of it is in the bounding box,
so I assume tons of overlap is "built in"? Happy to be corrected though.

I found this guide which seems to have some great explanations for why buffers are needed in some cases: https://blog.cyclemap.link/2020-01-25-tilebuffer/

[dabreegster]

Awesome progress! About styling/colors... I think it depends on the frontend. The job of the vector tiles is just to plumb along enough info to let a frontend make decisions. This is a Leaflet example, using the properties from each GeoJSON feature:

osm2streets/street-explorer/www/js/layers.js

Line 53 in 5b40c7a

export const lanePolygonStyle = (feature) => {

[jakecoppinger]

Does anyone know what sort of format a style URL might look like for a QGIS Vector Tile connection, and if there's an easy way of generating one from osm2streets/street-explorer/www/js/layers.js?

@dabreegster (and anyone else interested) I also created a Matrix room at https://matrix.to/#/#osm2streets-vector-tileserver:matrix.org to chat about this project if you're interested- I don't want to clutter this thread too much!

If you've never set up a Matrix client before I'd recommend following the prompts at https://element.io/get-started to download and sign into Element.

[dabreegster]

I don't know anything about QGIS, unfortunately. Don't worry about too many messages here -- this or a thread in the tileserver repo seems like a perfect place to figure this out. I've joined the Matrix. There's also an A/B Street Discord (https://discord.gg/nCvMD4xj4K) and Slack (https://join.slack.com/t/abstreet/shared_invite/zt-1mjdw5ez7-aDiCOpFxFniFyP4NaFk2yw) that people sometimes use

[jakecoppinger]

Got a front end renderer (Mapbox GL JS) using my tile server working, complete with styling and layers!

https://github.com/jakecoppinger/safe-cycling-map

[dabreegster]

This is fantastic! Out of curiosity, what would it take to host this or safe-cycling-map online somewhere? I guess the problem is hosting for the Overpass instance and RAM for the tileserver mainly.

I saw a few comments like parking/footpaths on the wrong side of the road. I'm sure you'll spot a bunch of bugs, or cases where the lane markings aren't regionalized. Please feel encouraged to file issues here so we can make more progress on things like that, now that they're even more visible!

[jakecoppinger]

Out of curiosity, what would it take to host this or safe-cycling-map online somewhere?

Good question! I'm currently working on optimising the tile generation. The current method I have in mind is:

• For a given zoom/x/y request:
• Zoom out to a fixed zoom value, and calculate x & y values for that (dividing by 2 each time you reduce zoom by 1 - I found that out by asking ChatGPT 🤣 )
• Get the tileIndex for that zoom value (ie: get the Overpass XML and call the osm2streets bindings)
• Store that tileIndex (in a cache, indexed in a 2d array by x & y (assuming a single fixed zoom value - could have a 3d array for multiple cached zoom stops)
• When a request comes in, zoom out to the fixed zoom as above, and if it already exists grab the tileIndex, then use that to calculate the exact zoomed in tile. For cache miss follow method above.

As for hosting the Overpass server - I'm not sure how much storage the entire world takes! It seems feasible spinning up a $5 VPS to run the tileserver and overpass docker image for one country, but unsure about the world.

[jakecoppinger]

I've got a "production" deployment working! https://safecyclingmap.com/

• Works in all of Australia at this stage. I needed 8GB of swap space (+1GB physical I think) to set up the docker image without a failure
• Frontend on Github pages (behind Cloudflare) because I couldn't be bothered fighting SSL config on AWS S3 -> Cloudflare yet
• Backend on Vultr droplet, using Cloudflare Tunnel to send requests like https://api.safecyclingmap.com/tile/18/241180/157318 to localhost:3000 😄
• I implemented the cache algorithm I detailed in the above comment! Using an LRU cache with a 1 hour expiry (so you can fix OSM data)
• I suppose I could point all non-Australian Overpass requests to one of the public servers at https://wiki.openstreetmap.org/wiki/Overpass_API#Public_Overpass_API_instances - with the new caching there isn't too much traffic

Let me know what you think! Lots I could improve on but it's a start.

[dabreegster]

Super awesome! A few ideas:

• Is the intersection markings layer included? Where sidewalks meet, they should generally have corners drawn, unless there's a bug here
• It'd be cool to hover/click a road or intersection and jump to that OSM object, for easy editing/querying. Not sure how that works with the vector tile format
• If you set hash: true when creating the map, the URL will track location, for easier URL sharing. You might need to do something like https://github.com/acteng/atip/blob/7e3ea66fc3d93e2c159db84f60436ba0fa4ef54d/components/Map.svelte#L9 to jump to the initial location if there's no hash
• I didn't look into how your caching works, but would it be easy to have two cache policies and toggle between them on the map? If someone isn't updating data frequently, it'd be neat to use any previously cached tile just for speed. But I guess that also starts to blow up your storage requirements over time

Works in all of Australia at this stage. I needed 8GB of swap space (+1GB physical I think) to set up the docker image without a failure

Wait, did you pregenerate anything, like run osm2streets over the whole country? I thought everything was built lazily