Mapzen shutdown #747
Comments
|
WTF, why is the next OSM service shutting down? I remember there was already a problem with one, which Gnome e.g. used. They then switched to mapbox.com. |
I assume that answer is standard - money (funding run out before company become profitable). |
westnordost
added
the
help wanted
|
One month, this is a pretty short period. I am pretty sure that I/we will not find and set up something until then. But this has a high(est) priority now. Without a map view, the app is barely usable. At least it is time enough to add an info message before their server is shutdown that the map view does not display the map and why. I guess it is not feasible to set up an own tileserver for StreetComplete. Are there any other options than MapBox? The MapBox tileserver is only allowed to be used through the MapBox SDK. I initially decided against MapBox because they have some telemetry built-in and their terms and conditions require to have this by default on (only letting the user switch it off in the settings). It is really a pity that Mapzen is shutting down, I considered the company a healthy competition to Mapbox. By the way, if Mapzen as a company is shutting down, then LOST will probably also not longer be maintained and an alternative needs to be searched sooner or later. |
It is a hackish band-aid but it seems that satellite images may be used for transitory period (code is at https://github.com/ENT8R/streetcomplete-mapstyle/blob/master/satellite-style.yaml ).
tangram-es is also likely to die. https://github.com/tangrams/tangram-es is under https://github.com/tangrams with
with website set to https://mapzen.com/tangram |
|
I can have a look into self hosting the tileserver for Streetcomplete. I already have found out how to do most of it, it seems not overly complicated. I currently see two Problems:
I think that it should be possible to reduce the traffic via caching and maybe more than one server serving the cached tiles, but the Database has to live somewhere. If it is wanted and is not to much traffic (or we can distribute the caches) I can look into it if hosting a tileserver possible |
Maybe it can be reduced by finishing https://github.com/ENT8R/streetcomplete-mapstyle and storing only limited data used by SC? Dropping some less used regions (Africa?) is probably not going to help much as I expect that regions with large amount of data and active mappers are the same.
StreetComplete is already caching map data |
|
Tangram(-ES) is a really neat library and it would be a big loss (to the openstreetmap ecosystem) if it would die like this. But without a tileserver it can use, it makes no sense. MapBox prohibits the use of their vector tileserver using another SDK than their own and I do not know any other. I imagine that tangram-es as an active project could be saved if someone from within the community of openstreetmap or maybe the foundation itself would slide in and set up an own vector tile server analogous to the Mapnik bitmap tiles server. If traffic is a problem, it probably is, limit usage to openstreetmap related projects. Given that the announcement about the company shutdown came with such short notice, I would say that it was also a surprise to all the developers who are now being laid off and probably both quite pissed now and have other worries than what may happen to their old project from work, so I don't expect any of the current maintainers to stick to the project. This will make it hard to retain the project. Has this news already reached the (talk) mailing list(s)? |
|
I see no posts on dev and talk. dev current state: https://lists.openstreetmap.org/pipermail/dev/2018-January/thread.html talk: https://lists.openstreetmap.org/pipermail/talk/ (no posts in 2018) |
Hmm… strange. GNOME does not have this. Maybe it is only in their Android SDK? |
Also thanks from me - tangram-es and vector tiles were very pleasant to use. |
|
Hi There A few things right now, will follow up on more later.
Will follow up more probably on the readme pages of the project(s). |
You really have no reason whatsoever to be sorry for continuing to work on project in your free time. It is amazing and thanks for that! If any support/development continue it would be great, but obviously nobody should feel obligated to do that. |
|
What about using tiles directly from OSM ? That's we were doing with WikiJourney and it works pretty well. |
|
Tangram-es supports raster tiles really well, so if you use raster tiles directly from osm or any other service it should work. Plus all those markers (we were working on a better interface here :() stuff will work out of the box since that's all added by the client code itself. With respect to using other vector tiles, you will have to work on stylesheet supporting the specific properties in that tile format. Once anyone of us gets some free cycles, we can support some property mapping functionality directly on tangram-es which will help you guys use amazing mapzen styles. |
|
if wanted, you can use the OSM WMS basemap-services, that my company provide. They are free to use. @sylvainar, @tallytalwar : Using tiles directly from OSM in your application is normally "tolerated", but not really what the OSM community likes you to do (officially it is even forbidden without the permission of the administrators). If usage exceeds a certain limit, people are asked to setup their own OSM server. See a brief discussion on this here: (I have some direct insight here, because I am board member of FOSSGIS e.V. (www.fossgis.de), which is also the local german chapter of OSMF and that pays for some of the OSM servers ;-)) |
|
Thank you all for your input. So, here is my battle plan: I would like to stick with Tangram-ES, if possible. Since the future of Tangram-ES remains largely unclear, I will put as least effort into the map display as possible. Any features concerning the map display will be delayed indefinitely. Thank you, @matkoniecz, for the remark about the satellite images. These can be used for transitory period to either another vector tile server or, if none can be found, to MapBox. @Akasch thank you for your offer, it is absolutely wanted! I can see from the Mapzen statistics that StreetComplete users are pretty consistently making about 20k tile requests per day. The size of one tile varies from mean about 2-20KB in towns, depending on how well-mapped a place is (i.e. if buildings are present etc). If you take ~10KB as a global mean, then we are at ~200MB a day, or 6GB a month. That sounds OK, doesn't it? @tilladams also thank you for your offer with the OSM WMS basemap-service! I have to say that I know very little about all those map services, this is why I very much appreciate help there. (What's WMS?) Is this a vector tile service or something Mapnik-based? |
AFAIK it refers to https://en.wikipedia.org/wiki/Web_Map_Service "standard protocol for serving (over the Internet) georeferenced map images". EDIT: wikipedia is misleading, it may be text (including vector tiles) or an image:
|
|
It does not list MapBox vector tile service, is MapBox not a WMS? |
Is it about https://ows.terrestris.de/dienste.html#openstreetmap-wms or https://en.wikipedia.org/wiki/Web_Map_Service or something else?
From reading https://www.mapbox.com/help/mapbox-arcgis-qgis/ and looking at https://www.mapbox.com/api-documentation/#retrieve-raster-tiles-from-styles it seems that mapbox is using WMTS protocol ( https://en.wikipedia.org/wiki/Web_Map_Tile_Service ) for delivering its raster maps. |
|
Have you considered OpenMapTiles to replace Mapzen Tile services ? |
|
Have you considered OpenMapTiles to replace Mapzen Tile services ? note
though with "making about 20k tile requests per day" 40$/month is not terrible (https://www.tilehosting.com/#plans) |
|
I will test over the next days what I exactly need. I think the traffic is manageable. At first I will setup a test server with only the data from Germany, to see how everything is working. From the what I have seen until yet I think I will change the URL format a little bit for a more efficient access control on my side. After that I will try to import the whole world. If this is working I will look into reducing the data size and doing more efficient caching. Especially in the beginning I will cache rendered tiles for relative long times, I think of around 12 hours at the moment. After I have seen the load it generates a reduction is hopefully possible. The first step should be finished in two or three days, the whole world in the middle of the next week I think. |
|
That sounds odd, are you saying that the mapzen vector tile service has inefficent caching and/or puts unnecessary data into the reply? If this is the case, perhaps you should put your enhancements upstream into their repository. |
|
Hi, "WMS" is Web Map Service", which is a standarized protocol, that allows to request map-tiles from any client (see ->http://metaspatial.net/w/index.php/OGC_WMS for a simple explanation . But you tell it, a WMS only serves tiles, no vectors. In general the advantage of using tiles (*.png- or *.jpg-images) is, that images are normally smaller (less traffic!) than using the vectors. I am not sure, because I do not know your code, but from my usage of the App "StreetComplete" I see no interaction with the real vectors. I think that just as background service tiles might be fine. But if vectors work fine, of course it is an opportunity. Just another idea (without really thinking about all the consequences): We at FOSSGIs donate money to open source projects and very often this is done by paying for server capacity. Might be an idea to think about setting up a project own vector tile server. For the technical stuff, I (or guys from my company) may help (for free of course). |
|
@westnordost I thought about putting the api key in the URL path instant of a GET parameter, but after some more consideration I think both forms are more or less equal. Once the index generation is finished I have all things ready for a test on the data from Germany. As the import on the hardware I use is relatively slow (no big SSDs), and it would be a secondary use of other resources. I think it would be grate to have a (more) dedicated machine for it if it is possible @tilladams . |
|
For Testing the vector tile service is available at http://test.osm.rokita.it/{tilesize}/{layers}/{z}/{x}/{y}.{format} . As the Polygon indices are not finished now it is really slow and Polygons are missing. I hope it will finish soon. Caching is not enabled, I will do it when the Polygons are ready. Only the Data of Germany is imported. |
|
Oh my… yes the URL is encrypted, only subdomain is visible. And it is exactly the location information, whcih is the problematic metadata here. |
|
The import is finally finished. After some tests, the tiles are generated mostly fast enough for StreetComplete in the meaning it is not really slower than before. Only some middle zoom tiles in citys are a slow. I render them this afternoon, and it should get better with caching over time. After the rendering of higher zoom levels I will switch over tiles.map-data.de from a Mapzen Proxy to the self generated tiles. |
|
To reduce Data costs you maybe could add support for the offline tils from Osmand. (Or create an StreetComplete-Addon for Osmand) My offline cache for Osmand is located on external sdcard. |
|
BTW before you actually switch the proxy of, can you already share some stats on the load you observe? |
|
Cool! Definitely, we will want to look at the stats, so that we can see if that server will be enough or if we need to look for a dedicated solution - and in any case we can estimate the running costs it will generate and look for how it should be financed. I am living close to the IKUM by the way. |
|
Yesterday I had 25551 requests, not counting cached responses (Cache duration is at 6 hours at the moment). The Database is over 1 TB at the moment I use two 512 GB SSDs in RAID 0 and one 512 HDD. The import is realistically only possible with SSD backed storage as it is really slow over wise. The pre rendering of the tiles running at the moment is slowing down the live rendering for requests, as the Database is IO waiting for the HDD as far as I can see. Generally most tasks are more bound by IO capacity than CPU Power from that I have seen so far. |
|
StreetComplete also caches, but as the map server URL changed, I guess the cache is rebuilt slowly for all the different users now. |
|
Hi Tobias - sorry for joining late.. I've just found that mapfit.com provides tiles with the tilezen schema. So you could ask your users to get an api key there for a fast tiles-server and to save you some server-load during the transition - though I would not recommend mapfit further - as they have zero OSM attribution... Great work with the tile-server, btw! |
|
I have finally switched the tile Server from proxy to locally generated tiles. If there are Problems please report them. The delay was mostly to the need to shuffle the database tables around on the Storage so that the tables with most reads are on the SSDs as the whole database is not fitting on the SSDs I currently have. I will optimize the Operations in the next hours/days and post some statistics. |
I don't think so... It was already discussed earlier in this issue (#747 (comment)) that this service won't be an option... |
|
one could use the openmaptiles tile-server, not necessarily use their service. The implementation is likely way more lightweight on the server resources due to using imposm3 vs osm2pgsql and rust instead of python for the tile generation. This would be my technical recommendation after having managed our universities vector tile server with osm2pgsql for almost five years. (imposm3 was ready to use when I left university and I only kept it running the last two years) |
|
oh it's been a while that I was into this... t-rex is actually the rust vector tile server http://t-rex.tileserver.ch/ - not sure what klokantech/openmaptiles is using. mixed up the two groups from switzerland :) |
|
If we are using any map tile provider, from my point of view, it is irrelevant on what soft- and hardware it runs. Thank you @hjanetzek for the Mapfit tip, I did not know them yet, will put this into consideration. |
|
Ah I just saw your edit regarding mapfit. Yeah, I wouldn't want to support a service like that, then. So it looks like openmaptiles would be one option, but it requires comittment in the sense that the map style(s) need to be rewritten. |
|
Update on Mapzen tiles! https://mapzen.com/blog/long-term-support-mapzen-maps/ Wish we could've talked about this sooner, but arrangements weren't certain until very recently :) |
|
This is awesome! It's a good idea to have a tilezen server around to keep all those demos and styles alive :-) That was a big concern. @Akasch and me were thinking about whether a full-blown tilezen server could be set up to serve as exactly that, for StreetComplete and also for other OSM related projects. I mean, it is already online, but currently it is only an intermediate solution. Is the nextzen server hosted with that same goal in mind? How is it financed? Will the tiles always be from December 2017? Scrolling down on your blog post (at Vector Tiles), I am getting a bit lost in the detail: Is the stuff about Tapalcatl and requester pays S3 an instruction on how to host an own tileserver? Or is the requester pays S3 your solution to how to finance the server? Where/how would I get a nextzen API key? Through explanations from @Akasch, I understand that a tileserver (at least) consists of two parts: One part that renders OSM data into vector tiles, the other part that caches the rendered vector tiles and serves them. So, a cheap vector tile server could consist of only the second part with a giant HDD which whenever the vector tile requested is not in the cache, fetches it from the "real" server. Something like a proxy. Is this what Tapalcatl does? |
|
Hi all! I happen to be spending the most time on putting together the Nextzen Tiles and am happy to answer questions. tiles.nextzen.org is an AWS Lambda that runs tapalcatl-py, which pulls tiles out of a requester-pays bucket on AWS S3. This bucket contains tiles from the last run of tiles we did before Mapzen was shut down. These tiles don't automatically update at this point, but we do have plans to update them regularly in the near future (once more of the Mapzen alumni don't have to spend time looking for jobs). I don't want to promise any update cycle, but it'll probably be on the order of monthly or weekly, not hourly or minutely. tiles.nextzen.org will primarily be a demonstration service that I try very hard to keep up, but probably won't be willing to put an uptime guarantee on. My hope is that people run their own tapalcatl-py's, but I also understand that that's a lot to ask, so I'm going to let others use it with an API key. The API key will be free to start with and will probably remain free for several months. Lambda is very cheap, especially for these very small tasks. I am in the process of working on the API key web app as we speak and am about to start testing it in production (on tile.nextzen.org) tonight or tomorrow. If you're interested in using it with Street Complete, I can post back here when it's ready for you to try. |
|
@Akasch Thank you very much for handling hosting of tiles! BTW, it is great that it was possible thanks to extensive open sourcing by Mapzen (I will try to consider such practice as even more important where I will be looking for any kind of service in future). |
|
@Akasch, thanks for the great work in rescuing StreetComplete from the Mapzen shutdown. Can I ask what your plan is in terms of keeping the data up-to-date as changes are made to the underlying OSM data? I've noticed that new buildings I'm adding are not showing up in StreetComplete now. |
|
So, I think this is solved now. Either, there is funding for a permanent proper vector tile server, or, if not, I can always switch to nextzen vector tiles (=archived tiles). |
|
If the app does no longer use Mapzen, shouldn't the corresponding paragraph in the About section should be removed? In version 4.0-beta2 I still see it. (I hope I didn't read over something like this in this thread) |
|
@westnordost wanted to change this later... See the comments on this comment: 25baa93 |
Today, Mapzen sadly announced to shutdown their services at the end of January (see more here: https://mapzen.com/blog/shutdown/)
They suggest to migrate to other services here: https://mapzen.com/blog/migration/
This will also affect StreetComplete as the map tiles are received from the Vector Tiles Service of Mapzen (https://github.com/westnordost/StreetComplete/blob/master/app/src/main/assets/cinnabar-style-8.0.0.yaml#L242) which means that on February, 1st, there will be no map displayed anymore in StreetComplete... I think we should search for an alternative as soon as possible...
The text was updated successfully, but these errors were encountered: