GeoJSON datasets

[jsanz]

Fixes #191

This PR includes the following new GeoJSON data sources, with the same precision that their already existing TopoJSON versions.

• admin_regions_lvl2_v2.geo.json 25MB
• usa_counties_v2.geo.json 2MB
• usa_zip_codes_v7.geo.json 11MB
• world_countries_v7.geo.json 11MB

Changes in existing admin regions and world countries are related to a small fix in the data pipeline to include proper IDs that I did as part of improving the Makefile.

For the World Countries, the new datasets are only available from 7.10, offering to the older versions the current published low precision dataset.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[nickpeihl]

Can you double-check that you have the latest shapefiles from the Google drive? I'm seeing some missing regions such as AZ-LAC, MU-PU, and others in the admin_regions dataset.

There remains some inconsistencies when dissolving the regions into countries. For example, the overseas territories of France (such as French Guiana) are completely missing from world countries. Also in world countries, Taiwan has been merged into China.

Compare your PR to what's currently in staging.

vimdiff <(curl --compressed https://storage.googleapis.com/elastic-bekitzur-emsfiles-vector-dev/files/world_countries_v1.geo.json | jq '.features[].properties.iso2' | sort | uniq) <(jq '.features[].properties.iso2' < data/world_countries_v1.geo.json | sort | uniq)

[jsanz]

Can you double-check that you have the latest shapefiles from the Google drive? I'm seeing some missing regions such as AZ-LAC, MU-PU, and others in the admin_regions dataset.

Sorry about that, I will double check the data, thanks!!

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[jsanz]

@nickpeihl I've marked again this for review, hopefully data issues are addressed. I also changed the source for the world countries to only offer TopoJSON for >= 7.10.

[nickpeihl]

World countries look good to me.

Administrative regions are missing a few regions in Azerbaijan and Slovenia, but that's my fault because I did not rsync my local changes to the Google Drive. Sorry about that.

vimdiff <(curl --compressed https://vector.maps.elastic.co/files/admin_regions_lvl2_v1.geo.json | jq '.features[].properties.region_iso_code' | sort | uniq) <(jq '.features[].properties.region_iso_code' < data/admin_regions_lvl2_v1.geo.json | sort | uniq)

Getting rid of the ~99- regions is a good idea though. Thanks for doing that.

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[jsanz]

@nickpeihl thanks for updating the datasets, I've tested again the regions and now we don't have any missing records compared with current production datasets and the new ones are all under control either as current missing data, or new data as per processing to generate all countries.

Also, there are no diffs with the current production world countries dataset

diff \
 <(curl -s --compressed https://vector.maps.elastic.co/files/world_countries_v1.geo.json | jq '.features[].properties.iso2' | sort | uniq) \
 <(jq '.features[].properties.iso2' < data/world_countries_v1.geo.json | sort | uniq)

[nickpeihl]

lgtm

Code review and compared datasets in this PR to production