Adding vector tile service from ArcGIS / Esri not working #170
Comments
|
|
The one with the "int() argument must be a string or a number, not 'NoneType'" was clearly an error and has been fixed. The fix will be included in the next release, which will be in the next few days. |
|
:: vector_layers field - Could you make this an optional parameter? :: TileJSON - There exists NO TileJSON Url for the Esri vector tiles, as far as I know. The mapbox spec https://www.mapbox.com/mapbox-gl-js/style-spec/#sources allows 3 different implementations. Esri choose supplying TileJSON properties such as "tiles", "minzoom", and"maxzoom" directly in the source. Could the plugin support this one? :: "int() argument..." - We will try this with the new release. Thanks for your answers. |
Technically, this would be possible.
We use the StlyeJSON URL only for styling, nothing else. Due to this, I don't think there will be a change soon in this matter in the implementation of the plugin. @sfkeller Would you mind to comment on our StyleJSON usage, especially, that we use it only for styling, even though there are TileJSON URLs in the StyleJSON? |
|
The "int() Argument error..." is fixed, thanks. Summarizing, Esri vector tiles can't be used with this plugin in QGIS as the plugin supports a different implementation than Esri. Both implementations are valid. @mnboos Thanks for your help. |
|
Happy to help. |
|
@mboos wrote
For a GIS usability point-of-view it's obvious: Data/layer/source are loaded, styled, then save as a GIS project. So, a project acts as "grouping concept". A TileJSON represents a Mapbox VT (metadata) data/layer/source without style (like a vector GeoPackage). A MB GL JS style (style.json), as it is handled in Mapbox tools (like Studio) in fact can contain several sources and maps them to a style descriptor. So here, the style acts as style config. and as grouping container, aka "project" file (=> "project" style.json with many data-source refs plus many style-descriptors). Currently the plugin manages one VT source and allows to assign optionally one style (ignoring other sources there) and persists this in the QGIS project file (=> one data-source plus many style-descriptors). Now given TileJSON/mbtiles is not even standardized yet, it makes sense to wait for a maturation of the specs. |
|
@StefanJaq wrote
I'd rather say that I still hope, that it's possible to generate a minimal TileJSON metadata file. This is needed by Maputnik as source and by Mapbox Studio too. Given this TileJSON it would be possible at least to load it as data source/layer in this plugin too. @mboos: Any ideas to preprocess any VT source (may it come from ArcGIS Pro or tilemaker or so) and generate this TileJSON metadata file? |
|
@sfkeller I think the generation/creation of a TileJSON is not even the problem. The reader is implemented this way, that it reads from a single source (=TileJSON) at a time. A StyleJSON on the other hand, consists of multiple TileJSON sources and that's the problem. I'm thinking about the following change: The StyleJSON input is removed and the only remaining input field then accepts both type (i.e. StyleJSON and TileJSON). The support of a StyleJSON as source, would be a nice enhancement of the plugin. |
|
Sorry to bump this very old thread, but I am currently trying to get this resource showing in QGIS: The conclusion I got from the various bug reports is that it is still currently not possible, even with a local metadata.json. Has there been any developments on this, or have I misunderstood the conclusion? |
|
@Moult No worries, I'm happy about feedback. The resource you linked here, won't work out of the box, due to several missing properties. What you can do is create a metadata.json like described here: https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/wiki/Help#metadatajson The file can be on your local disk. Instead of using a URL in the connections dialog, just enter the full filepath. If you need further assistance, please let me know! |
|
Thanks @mnboos - I've already tried to create a Please see the attached file (called .txt because that's what github will allow me to upload, but of course in real life it is a json file). With this file, it can connect and load layers, but I only see black and no outcome in QGIS. Do you mind investigating my metadata.json for me? |
|
Sorry for the delay, I'm currently on vacation. As far as I can see on my mobile, you copied the style json into the vector layers property, what either won't work or at least won't do anything. I'll have an in depth look at this scenario, as soon as I'm back. |
|
No worries - take your time, no rush :) The style json is there, but doesn't seem to do anything. It certainly succeeds to connect and lists the layers though. |
|
Ok, I'm currently looking at it. The good news is, I can see something:
The bad news: An error occurs, due to invalid filenames, which I create in the temp folders (see #240). I first have to fix this bug and hopefulle, it then works. |
|
I release version v3.0.5 which should work with this TileJSON: https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/files/3269379/arcgis_tilejson.json.txt |
|
Thanks @mnboos - can you give step by step instructions on how to get output? Here's what I have tried:
Result is that I have a whole bunch of layers, but can't see anything. |
Not quite; Select the server tab and just insert the whole filepath to the metadata.json
That's because they're read from the metadata, not from the actual server. So the whole process would be:
|
|
Sorry, @mnboos I still cannot get your result. I've added it as a server as you've specified, with that json file having the contents of what you uploaded. It detects the layers of "foobar", which I can add.
However here is my map with OSM tiles as another layer to show how I am hovering around Shenzhen. As you can see apart from the OSM tiles basemap no vector tiles are loaded.
I have also tried setting the project CRS to EPSG 4214. In the log messages I see this, suggesting that the X Y Z is incorrect: What am I missing? |
|
Can you do the following:
And then copy logs and a screenshot. At this moment, for me it looks like this:
See the little purple area in the middle? That's your actual data. And the logs: |
|
Results: I also notice that after pressing add, my EPSG changes to 4326 automatically. Log: Our logs seem quite different? Perhaps the JSON isn't the same? |
|
It tries to load zoom level 7 for you. And all the network requests fails. Therefore, I assume your QGIS map location is outside of the bounds of the available data. Could you try to scroll out stepwise. Maybe, at one point you'll reach zoom level and it should load something. |
|
Thanks @mnboos I can now see vector tiles. However, they don't seem to be geographically located in the right place? I can see both vector tiles, as well as an OSM base map somewhere between Italy and Spain in EPSG 3857. |
|
Yes, I realize that now too. @sfkeller : Do you think it might be related to something visible here: https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer {
"currentVersion": 10.61,
"name": "BaseMap_withoutDEM",
"capabilities": "TilesOnly,Tilemap",
"type": "indexedVector",
"tileMap": "tilemap",
"defaultStyles": "resources/styles",
"tiles": ["tile/{z}/{y}/{x}.pbf"],
"exportTilesAllowed": false,
"initialExtent": {
"xmin": 53766.609476385944,
"ymin": -1804.5673782131707,
"xmax": 199102.52514821733,
"ymax": 63838.688908299424,
"spatialReference": {
"wkt": "PROJCS[\"Shenzhen_Local\",GEOGCS[\"GCS_Beijing_1954\",DATUM[\"D_Beijing_1954\",SPHEROID[\"Krasovsky_1940\",6378245.0,298.3]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],PROJECTION[\"Gauss_Kruger\"],PARAMETER[\"False_Easting\",-199250.645],PARAMETER[\"False_Northing\",-2477994.076],PARAMETER[\"Central_Meridian\",111.0],PARAMETER[\"Scale_Factor\",1.0],PARAMETER[\"Latitude_Of_Origin\",0.0],UNIT[\"Meter\",1.0]]"
}
},
"fullExtent": {
"xmin": 53766.609476385944,
"ymin": -1804.5673782131707,
"xmax": 199102.52514821733,
"ymax": 63838.688908299424,
"spatialReference": {
"wkt": "PROJCS[\"Shenzhen_Local\",GEOGCS[\"GCS_Beijing_1954\",DATUM[\"D_Beijing_1954\",SPHEROID[\"Krasovsky_1940\",6378245.0,298.3]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],PROJECTION[\"Gauss_Kruger\"],PARAMETER[\"False_Easting\",-199250.645],PARAMETER[\"False_Northing\",-2477994.076],PARAMETER[\"Central_Meridian\",111.0],PARAMETER[\"Scale_Factor\",1.0],PARAMETER[\"Latitude_Of_Origin\",0.0],UNIT[\"Meter\",1.0]]"
}
},
"minScale": 1.476693380542385E8,
"maxScale": 563.3138200921574,
"tileInfo": {
"rows": 512,
"cols": 512,
"dpi": 96,
"format": "pbf",
"origin": {
"x": -1.0201387142540421E7,
"y": 7524142.421540422
},
"spatialReference": {
"wkt": "PROJCS[\"Shenzhen_Local\",GEOGCS[\"GCS_Beijing_1954\",DATUM[\"D_Beijing_1954\",SPHEROID[\"Krasovsky_1940\",6378245.0,298.3]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],PROJECTION[\"Gauss_Kruger\"],PARAMETER[\"False_Easting\",-199250.645],PARAMETER[\"False_Northing\",-2477994.076],PARAMETER[\"Central_Meridian\",111.0],PARAMETER[\"Scale_Factor\",1.0],PARAMETER[\"Latitude_Of_Origin\",0.0],UNIT[\"Meter\",1.0]]"
},
"lods": [{
"level": 0,
"resolution": 39070.84569351727,
"scale": 1.476693380542385E8
}, {
"level": 1,
"resolution": 19535.422846758636,
"scale": 7.383466902711925E7
}, {
"level": 2,
"resolution": 9767.711423379318,
"scale": 3.6917334513559625E7
}, {
"level": 3,
"resolution": 4883.855711689659,
"scale": 1.8458667256779812E7
}, {
"level": 4,
"resolution": 2441.9278558448295,
"scale": 9229333.628389906
}, {
"level": 5,
"resolution": 1220.9639279224148,
"scale": 4614666.814194953
}, {
"level": 6,
"resolution": 610.4819639612074,
"scale": 2307333.4070974765
}, {
"level": 7,
"resolution": 305.2409819806037,
"scale": 1153666.7035487383
}, {
"level": 8,
"resolution": 152.62049099030185,
"scale": 576833.3517743691
}, {
"level": 9,
"resolution": 76.31024549515092,
"scale": 288416.67588718457
}, {
"level": 10,
"resolution": 38.15512274757546,
"scale": 144208.33794359228
}, {
"level": 11,
"resolution": 19.07756137378773,
"scale": 72104.16897179614
}, {
"level": 12,
"resolution": 9.538780686893865,
"scale": 36052.08448589807
}, {
"level": 13,
"resolution": 4.769390343446933,
"scale": 18026.042242949035
}, {
"level": 14,
"resolution": 2.3846951717234663,
"scale": 9013.021121474518
}, {
"level": 15,
"resolution": 1.1923475858617332,
"scale": 4506.510560737259
}, {
"level": 16,
"resolution": 0.5961737929308666,
"scale": 2253.2552803686294
}, {
"level": 17,
"resolution": 0.2980868964654333,
"scale": 1126.6276401843147
}, {
"level": 18,
"resolution": 0.14904344823271665,
"scale": 563.3138200921574
}
]
},
"maxzoom": 18,
"resourceInfo": {
"styleVersion": 8,
"tileCompression": "gzip",
"cacheInfo": {
"storageInfo": {
"packetSize": 128,
"storageFormat": "compactV2"
}
}
},
"serviceItemId": "527e0a10a0dd4c9a9cecdc430a35e220",
"maxExportTilesCount": 100000,
"minLOD": 0,
"maxLOD": 18
} |
|
@StefanJaq: Mapbox Vector Tiles (MVT) "MAY be used to represent data with any projection and tile extent scheme." See https://github.com/mapbox/vector-tile-spec/tree/master/2.1#3-projection-and-bounds: "Web Mercator is the projection of reference and the Google tile scheme is the tile extent convention of reference". Even forthcoming MVT version 3 sticks to Web Mercator and leaves open any standardization on CRS and tile extent scheme. Esri has built into the vector tile generator (e.g. of ArcGIS Pro) many extensions (like CRS) and optimizations (like a different scheme and like object generalization). So AFAIK there are some options to override in ArcGIS Pro in order to generate MVT 2 compatible tiles. So, before continuing to resolve this issue, I strongly recommend to make sure that the tiles created have Web Mercator (EPSG:3857) and stick to the Goog tile scheme. |
|
@sfkeller @mnboos We’re already testing for a while the integration from vector tiles in QGIS.
This is the url to vector tile server in ArcGIS Online: (This link will be only temporarly published) First of all, we have modified the style file, as described above. (see attached root.json) Afterwards we added it with Vector Tiles Reader to QGIS:
The tiles are created with ArcGIS Online / Bing Maps / Google Maps Tiling scheme and are Web Mercator with EPSG: 3857. Do you have any idea why the vector tiles look different and with less features? |
|
Hi @mnboos, would it be possible that you have a look at the issue above? Or have you already done that? |
|
Ah sorry. I did not debug it yet.
So according to your information, this requirements should be met. I'll have to debug it. I'll try to look at it on Sunday. |
|
@ninitartaruga: Just wondering: Do you think that Esri or someone could sponsor this feature? In any case, I assume nobody refuses inkind payments :-). |
|
@ninitartaruga From what I can tell, it seems to work as expected. There are no errors in the plugins log and the features map OSM.
Why there are less features I don't know. But since the plugin is showing some content, I would assume there might be a misconfiguration on the server side. |
|
I’ve used the information from this issue to get maps.wien.gv.at vector basemap, an ESRI Vector TileCache, loaded in QGIS. I’ve written it up here: https://nextjournal.com/kommen/use-wien-vector-basemap-with-qgis Still needs some work, but it’s at least something. |
|
@mnboos thanks for this awesome plugin. I've created a proxy that generates (hopefully proper) Details here: https://bergwerkgis.github.io/vtbasemapat/ Unfortunately I'm seeing inconsistent behavior using the plugin, especially with regards to styling. Also from time to time I get here at line Vector-Tiles-Reader-QGIS-Plugin/plugin/vtr_plugin.py Lines 522 to 532 in d8666a2 Not sure where these problems come from (my style/tile.json, plugin, data, ...). Any way I can help debug/solve this? Btw: - EDIT - |
|
Wait a moment: Does this mean that ArcGIS (Pro) generates MVT compatible data - against all assumptions - and the core compatibility issues lie 1. in the metadata and 2. in the styling (style.json)? |
|
@sfkeller That is how I—bearing my very limited understanding and GIS experience—understood it, yes. At least when the export from ArcGIS (Pro) is done using the |
|
I don't own or have access to any ESRI products so these are all my assumptions or things I've read somewhere on the internet (don't know if they are still true, or were ever true):
The orginal tile.json of the Austrian data does have the Yes, if the cache is exploded I think it's just a metadata issue. |
|
@kommen 😄 you beat me by seconds 😂 |
|
@BergWerkGIS Thanks for your efforts! The problem is that the style converter is pretty incomplete and a bit error prone. The MB GL style specification is to complex in order for me to maintain the style converter properly. Of course, errors like the above one shouldn't happen. edit: Oh wait, sorry. I still seem to be a bit tired, the bug you reported comes from the plugin and not the style converter, which is indeed bad. |
|
@mnboos I totally understand that a full blown style converter would be a huge undertaking. Let me know if you need anything for debugging the other issue - might be an error on my side as well when mangling the ESRI JSONs. |
|
Got pulled into this via twitter. A few comments / questions: |
Exactly, that's the problem. Where a tilejson should be linked there's some other json that throws off other tools. EDIT --->
<--- EDIT
No issues that I am aware of.
No, not referring to overzoom. As I said above
So, no way for me to say with certainty. But I was thinking about a presentation I've seen years ago, similar to this which clearly suggests (at least to me) that tiles have a smaller extent, where there is more data
That's perfect 👍 And that's also why this plugin, my Mapbox GL JS demo and maputnik are working with the ArcGIS tile cache - after creating a proper tilejson. |
|
Hi Craig. Thanks for joining this issue. My intention here is to find out how ArcGIS (Pro, or other Esri SW) can be configured so that it generates vector tiles that are compatible to MVT and can be consumed by a client like this QGIS plugin. That's what I'm aware of what ArcGIS Pro users must configure:
Something missing? |
|
a) This TileJSON specific addition to source is new to me. Versioning of the style spec has been an ongoing problem with Mapbox as they don't explicitly manage versions since they merged the spec repo into their implementation repo. That said, minzoom / maxzoom has defaults if not specified. As stated above, TileJSON is insufficient for many needs and full spec is not required for most mapbox-gl compatible clients. Mapbox themselves has created their mbtiles metadata spec to makeup for many such inefficiencies. For maximum interchangeability produced from ArcGIS Pro, it's:
We have R&D work going on WRT supporting more clients even with Indexed tiles. |
|
thanks for you speedy answer @williamscraigm
As I understand it, the first explicit mention of TileJSON appeared in style spec
Oh yes, you are right. I had forgotten that just the major version is epxressed in the layer.
Thanks for the clarification.
👌 |
|
"indexed overzoom" is interesting, but I assume you understand, that adding yet another complexitiy to the VT spec. has lower priority than interoperability and compatibility. Regarding TileJSON spec. A (vector tiles) client in a loosely coupled, open environment, needs some metadata. minzoom / maxzoom may have defaults, but 'bounds' and the (source) 'json' are required. IMHO TileJSON (metadata.json, https://github.com/mapbox/tilejson-spec ) and MBTiles (metadata table, https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md ) must be consistent. Citing MBTiles spec.: "In a future revision of this specification, the bounds, minzoom, and maxzoom rows of the metadata table will be mandatory." => Assuming you/ArcGIS are'nt going to support MBTiles soon, I'd like to focus on interoperability with TileJSON/MBTiles compatible metadata json (and vector tile handler servers returning tiles in the usual tiling schema). Just for testing I'd suggest to look at how t-rex (https://t-rex.tileserver.ch/ ) and especially how GDAL (https://gdal.org/drivers/vector/mvt.html ) are handling TileJSON metadata.json . P.S. I wish there could be a meeting about VT with you, Blake (@flippmoke) and Jeff (@slarjy) at FOSS4G end of August... |
That's nothing the client would have to worry about. |
|
As member of the team realising the basemap.at-VTC and as I started the discussion here, I'm really happy for your inputs and discussion. Sorry for not responding earlier, but at the moment I'm "out-of-office" and will be back during next week. Nevertheless some short comments: a) TileJSON b) c) @BergWerkGIS at the moment there are no fields (attributes) included. This might be included in a further release. d) @BergWerkGIS There are other tools except ArcGIS Pro that can allready use the basemap.at Vektor. ;-) https://woportal.mysynergis.com/BasemapAT/synserver?project=BasemapAT&client=flexjs&language=en f) Again, thanks a lot for your work and input. I want to add that I can't speak for the basemap.at team. |
|
@BergWerkGIS : Happy to hear this that a client doesn't need the overzoom index(?). There was a similar but simpler trick I've seen in MVT (Mapbox vector tile) sources, that some single tiles aren't existing/served (between default minzoom z8 and maxzoom z14), being NULL in MBTiles and/or giving error 404 over http. Clients then instead take a tile at z8 by default. This saves much data/space e.g. in monotone sea, forest or snowy areas. But what we should focus here, is
|
|
👋 @StefanJaq thanks for chiming in.
Has been since 2014 #170 (comment)
Not urgent, my tool takes care of that on the fly.
I think project team is more than happy to finally share their great effort with a broader audience, not just At least that's what I get form my little tool being linked to on the https://basemap.at/ main page already. Really looking forward to more interoperability in the near(?) future 🚀
Yes, at least that's how I understand it should work. |
|
Thanks to your hints we found errors in the basemap.at VTC implementation regarding the mapbox specification. We have adjusted those settings at our development area. I don't know by now when those new files will be public available. Hopefully soon. Thanks again. |
|
@StefanJaq On basemap.at in the News section it says the root.json of the vector basemap.at now is adheres the Mapbox 2.0 spec. So I assume this is live now? |
Servus @kommen . There are other json-files on the server: I know that the basemap.at team plans to provide a full mapbox compatible vector tile cache till approx june 2020. (Here I'm doing supporting work). Then the "beta" status should move to final! Until then please be aware that everything is subject to change. |
|
In case anyone is interested in having an mbtiles version of https://github.com/BergWerkGIS/vtpk2mbtiles Note: |
|
Nice work! |
Trying to add a vector tile service from ArcGIS to QGIS results in the following Error.
We are using QGIS Las Palmas with Extenstion Vector Tiles Reader QGIS Plugin.
Using the style URL https://tiles.arcgis.com/tiles/Ok7pjj1jT9rEneC7/arcgis/rest/services/Vector_tiles/VectorTileServer/resources/styles/root.json
Our setting are as in following screenshot (default):
This operation results in the following error:
We adopted the style-json:
http://austria.maps.arcgis.com/sharing/rest/content/items/e01226ca419642f08a929974adcff3a1/resources/styles/root.json
After doing this and the same default setting in QGIS the following error occurs:
It looks like the plug-in is requiring a tilejson-spec extension named vector_layers, despite it isn't part of the current documentation (I didn't find it). Why is this needed?
Document vector_layers extension mapbox/tilejson-spec#14
Why is it neccessary to enter two URLS? Why especially the TileJSON-URL? Usually the Tile-Url is linked in the styles - JSON.
Any ideas?
The text was updated successfully, but these errors were encountered: