Reorganizes and cleans up all geo scripts (#3331)

* Moves third-party-maps to new "geo" folder

* Merge

* Merges updated url

* Moves generate-zone-bounding-boxes script

* Updates paths

* Updates paths

* Updates more paths

* Rearranges things and adds comments from README to file

* Fixes merge-conflict change

* Moves util functions to util file

* Fixes paths

* Fixes paths

* Fixes more paths

* Fixes imports

docker-compose.yml

@@ -23,15 +23,13 @@ services:
     volumes:
       - './config:/home/config'
       - './web/.eslintrc:/home/src/electricitymap/contrib/web/.eslintrc'
-      - './web/generate-geometries.js:/home/src/electricitymap/contrib/web/generate-geometries.js'
+      - './web/geo:/home/src/electricitymap/contrib/web/geo'
       - './web/locales:/home/src/electricitymap/contrib/web/locales'
       - './web/locales-config.json:/home/src/electricitymap/contrib/web/locales-config.json'
       - './web/package.json:/home/src/electricitymap/contrib/web/package.json'
       - './web/public:/home/src/electricitymap/contrib/web/public'
       - './web/server.js:/home/src/electricitymap/contrib/web/server.js'
       - './web/src:/home/src/electricitymap/contrib/web/src'
-      - './web/third_party_maps:/home/src/electricitymap/contrib/web/third_party_maps'
-      - './web/topogen.sh:/home/src/electricitymap/contrib/web/topogen.sh'
       - './web/views:/home/src/electricitymap/contrib/web/views'
       - './web/webpack.config.js:/home/src/electricitymap/contrib/web/webpack.config.js'
   web-watch:
@@ -49,14 +47,12 @@ services:
     volumes:
       - './config:/home/config'
       - './web/.eslintrc:/home/src/electricitymap/contrib/web/.eslintrc'
-      - './web/generate-geometries.js:/home/src/electricitymap/contrib/web/generate-geometries.js'
+      - './web/geo:/home/src/electricitymap/contrib/web/geo'
       - './web/locales:/home/src/electricitymap/contrib/web/locales'
       - './web/locales-config.json:/home/src/electricitymap/contrib/web/locales-config.json'
       - './web/package.json:/home/src/electricitymap/contrib/web/package.json'
       - './web/public:/home/src/electricitymap/contrib/web/public'
       - './web/server.js:/home/src/electricitymap/contrib/web/server.js'
       - './web/src:/home/src/electricitymap/contrib/web/src'
-      - './web/third_party_maps:/home/src/electricitymap/contrib/web/third_party_maps'
-      - './web/topogen.sh:/home/src/electricitymap/contrib/web/topogen.sh'
       - './web/views:/home/src/electricitymap/contrib/web/views'
       - './web/webpack.config.js:/home/src/electricitymap/contrib/web/webpack.config.js'

web/BUILD.yaml

@@ -13,9 +13,7 @@ steps:
       ELECTRICITYMAP_PUBLIC_TOKEN: ${BRICK_ELECTRICITYMAP_PUBLIC_TOKEN:-unknown}
     inputs:
       # geometries
-      - generate-geometries.js
-      - topogen.sh
-      - third_party_maps
+      - geo
       # source code
       - ../config/{exchanges,zones}.json
       - locales
@@ -26,7 +24,7 @@ steps:
       - ./{.babelrc,.eslintrc.js,server.js,webpack.config.js}
     commands:
       - mkdir -p public/dist
-      - bash topogen.sh
+      - bash geo/topogen.sh
       - yarn build-release
     outputs:
       - src/world.json

web/Dockerfile

@@ -9,10 +9,10 @@ RUN yarn install --frozen-lockfile
 
 # Generate map
 RUN mkdir -p public/dist/
-ADD web/third_party_maps ./third_party_maps
-ADD web/generate-geometries.js web/topogen.sh ./
+ADD web/geo/third_party_maps ./geo/third_party_maps
+ADD web/geo/ ./geo/
 ADD web/src/world.json ./src/world.json
-RUN bash topogen.sh
+RUN bash geo/topogen.sh
 
 ARG ELECTRICITYMAP_PUBLIC_TOKEN
 

web/README.md

@@ -7,7 +7,7 @@ The world map data is open source and provided by [NACIS](http://nacis.org/initi
 To generate a new world.json run the following command from the web directory after making changes:
 
 ```sh
-docker-compose run --rm web ./topogen.sh
+docker-compose run --rm web ./geo/topogen.sh
 ```
 
 ### `topogen.sh`
@@ -18,30 +18,20 @@ The file `topogen.sh` does the following:
 and added to the `web/build` folder
 - other data (at 12.01.2018, only Canaries Islands) is added to the `build` folder
 - the module `shp2json` is used to convert the shp data to the GeoJSON format
-- `generate-geometries.js` is then called
+- `geo/generate-geometries.js` is then called which simplifies and compresses the geojson data
+- The output is `world.json` which is sent to the client
 
-### `generate-geometries.js`
+### `geometries-config.js`
 
 The variables `zoneDefinitions` should be updated manually to reflect intended changes in mapping between electricityMap zones and NACIS geometries. It relates each zone from the electricityMap to how it is described by data from NACIS (or third party). A zone can correspond to a country, a group of countries, a state, a group of states...
 
-The function `geomerge` merges a list of GeoJSON Polygons or MultiPolygons into a single
-multi-polygon. This allows to merge a group of geometries into a single one.
-
-According to `zoneDefinition`, a single GeoJSON MultiPolygon is created for each zone, by
-grouping all geometries corresponding to that zone.
-
-Finally, the module `topojson` converts the GeoJSON into into the
-TopoJSON format, which is a more compressed format than geoJSON. It only stores arcs that are used multiple times once. We also perform some simplifications and project points
-on a grid. All together, this allows to convert a ~`24MB` file to a ~`1MB` one.
-
-The final file is named `world.json` and is the one sent to the client.
 
 ## `generate-zone-bounding-boxes.js`
 
 You can create bounding boxes for new or existing zones in `config/zones.json`:
-1) Run: `docker-compose run --rm web ./topogen.sh`
+1) Run: `docker-compose run --rm web ./geo/topogen.sh`
 2) Update the zone you want to update in `config/zones.json` with `"bounding_box": null`
-3) Run: `node generate-zone-bounding-boxes.js`
+3) Run: `node geo/generate-zone-bounding-boxes.js`
 
 ## Useful tips