Make .mbtiles compatible with Mapbox upload

[Maltretieren]

I converted a *.osm.pbf file (http://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf) to mbtiles with tilemaker, default config+lua (tilemaker liechtenstein-latest.osm.pbf --output=lichtenstein.mbtiles). When I try to upload to mapbox.com I get an error "unknown format" by mapbox. Does it need a special config to be compatible with mapbox?

[pnorman]

I wonder, might Mapbox be expecting compressed vector tiles instead of plain ones with their mbtiles uploader? Mapbox GL JS wants uncompressed, but Mapbox Studio needs compressed. You might want to contact their support.

[Maltretieren]

Thanks for clarification. There is an issue on mapbox/mapbox-studio-classic#1076 . It would be nice to have an option to activate/deactivate compression in tilemaker. I think the issue can be closed.

[systemed]

We actually have compression/decompression code in the tilemaker codebase, so it should be pretty easy to add. I'll change the title to make it clear that an enhancement is required.

[pnorman]

Although optional compression is a good enhancement, it might not be the cause of the mapbox.com incompatibility. It's just one likely cause, and you want compressed for uploading anyways.

[systemed]

@Maltretieren Do you have an example file that works with the Mapbox upload, so I can disassemble and compare?

[Maltretieren]

@systemed You can download the file from geofabrik (http://download.geofabrik.de/europe/liechtenstein-latest.osm.pbf) and produce the mbtiles file with following command: "tilemaker liechtenstein-latest.osm.pbf --output=liechtenstein.mbtiles"

[systemed]

Yes - sorry, I wasn't clear: I was wondering if you had an example of a file not produced by tilemaker and which does work with the Mapbox upload.

[Maltretieren]

Didn't read carefully ;) https://github.com/b-g/tilehut/blob/master/data/tiles-world-vector.mbtiles works...

[systemed]

Thanks!

Ok, there appear to be a couple of issues with Mapbox upload.

• Expects compressed .pbf contents (see https://github.com/mapbox/tiletype/blob/master/index.js#L35)
• Expects a special json entry in the metadata table (see https://github.com/mapbox/mapbox-upload-validate/blob/master/lib/validators/mbtiles.js#L43). This appears to be vaguely related to TileJSON but mostly contains a mysterious undocumented vector_layers key. For reverse engineering reference, the entry in your sample file contains:

name=json
value={"vector_layers":[{"id":"admin-countries","description":"admin-countries","fields":{"ne_10m_adm":"String","ScaleRank":"Number","LabelRank":"Number","FeatureCla":"String","OID_":"Number","SOVEREIGNT":"String","SOV_A3":"String","ADM0_DIF":"Number","LEVEL":"Number","TYPE":"String","ADMIN":"String","ADM0_A3":"String","GEOU_DIF":"Number","GEOUNIT":"String","GU_A3":"String","SU_DIF":"Number","SUBUNIT":"String","SU_A3":"String","NAME":"String","ABBREV":"String","POSTAL":"String","NAME_FORMA":"String","TERR_":"String","NAME_SORT":"String","MAP_COLOR":"Number","POP_EST":"Number","GDP_MD_EST":"Number","FIPS_10_":"Number","ISO_A2":"String","ISO_A3":"String","ISO_N3":"Number"}}]}`

• mbtiles apparently uses TMS, not OSM, tile numbering (this is in the spec, mea culpa for not spotting it)

[pnorman]

I suggest focusing on matching the published specs, then raise any incompatibility issues on the Mapbox end. Trying to reverse engineer their code in preference to the specs is an exercise in frustration.

[systemed]

I've pushed an update which should permit Mapbox-compatible .mbtiles to be created. Three changes:

• Tiles can now be compressed (specify "compress": true in the config file)
• TMS numbering
• You can now add arbitrary metadata to the .mbtiles by specifying it in the config file - see the instructions, and the supplied config file, for examples. Most relevantly in this case, as well as string key/value pairs, you can have a hash (or other JSON object) as the value, and Tilemaker will encode it as JSON before writing it into the .mbtiles - which appears to be what Mapbox expects.

I considered adding direct support for the vector_tiles value, but given that this appears to be specific to Mapbox's implementation rather than part of any standard, decided that a "bring your own metadata" approach was better.

I've tested it by uploading it to Mapbox.com, which seems to accept it happily. I've not proceeded any further because I couldn't instantly see how to get at it via Mapbox Studio's UI, but any further comments/suggestions are very welcome.