[1-1-2023]: Feature/COR-1226 municipality reorganization (#4538)

* feat: update schema and documentation * feat: combine municipalities and add redirects * chore: cleaned up and updated documentation * chore: adjust docs and update to current year * feature/deal-with-misssing-GM-archived-data_V2 (#4598) * feat(municipality-restructuring): change the schema to remove minItems; * feat(municipality-restructuring): create empty graph version * feat(municipality-restructuring): simplified empty data objects approach; fixed TS/compilation issues; Co-authored-by: VWSCoronaDashboard26 <[email protected]> Co-authored-by: J <[email protected]> Co-authored-by: VWSCoronaDashboard26 <[email protected]>
minvws · Jan 6, 2023 · 2709e37 · 2709e37
1 parent bc7f275
commit 2709e37
Show file tree

Hide file tree

Showing 10 changed files with 132 additions and 193 deletions.
diff --git a/docs/choropleth-topojson.md b/docs/choropleth-topojson.md
@@ -16,21 +16,19 @@ This section describes how to generate this data with the correct projection app
 
 ### Importing the source data
 
-To create the data files we need, we will be using `cbsgebiedsindelingen_2022_v1.gpkg` as the
-data source. This package can be downloaded from: [www.pdok.nl/downloads/-/article/cbs-gebiedsindelingen](https://www.pdok.nl/downloads/-/article/cbs-gebiedsindelingen). Download the XML file and find the link to the latest `*.gpkg` file in it. The URL looks like this: https://geodata.nationaalgeoregister.nl/cbsgebiedsindelingen/extract/cbsgebiedsindelingen_2022_v1.gpkg.
+To create the data files we need, we will be using `cbsgebiedsindelingen_2023.gpkg` as the
+data source. This package can be downloaded from: [www.pdok.nl/downloads/-/article/cbs-gebiedsindelingen](https://www.pdok.nl/downloads/-/article/cbs-gebiedsindelingen). Download the XML file and find the link to the latest `*.gpkg` file in it. The URL looks like this: https://geodata.nationaalgeoregister.nl/cbsgebiedsindelingen/extract/cbsgebiedsindelingen_2023_v1.gpkg.
 
-ATTENTION: At the time of writing, the `cbsgebiedsindelingen_2022_v1.gpkg` file is the latest version.
+> ⚠️ **ATTENTION**: At the time of writing, the `cbsgebiedsindelingen_2023.gpkg` file is the latest version.
 It is of course very likely that by the time new data needs to be generated that a newer file is available.
-Pay attention to download the very latest version.
+Pay attention to download the very latest version. If there may be a case that the latest CBS data is still not available and some manual work needs to be done, use the current TopoJson file and proceed to [Generate TopoJson](#generate-topojson).
 
 After downloading, import the package into QGIS (the easiest way of doing this is by simply dragging the package into
 the main QGIS window) and select the following layers to be added:
 
-- `cbs_gemeente_2022_gegeneraliseerd`
-- `cbs_veiligheidsregio_2021_gegeneraliseerd` (2022 not available at the moment of this writing)
-- `cbs_landsdeel_2022_gegeneraliseerd`
-
-NOTE: The year indicator will differ when dealing with newer data. Find the layer with the most recent year.
+- `gemeente_gegeneraliseerd_2023`
+- `veiligheidsregio_gegeneraliseerd_2023`
+- `landsdeel_gegeneraliseerd_2023`
 
 Import with the CRS:`EPSG:28992 - Amersfoort / RD New - Projected` projection which should the default while importing.
 
@@ -40,7 +38,7 @@ Import with the CRS:`EPSG:28992 - Amersfoort / RD New - Projected` projection wh
 
 To clean up the data we have to perform the following steps:
 
-Create the municipalities data file (**cbs_gemeente_2022_gegeneraliseerd**):
+Create the municipalities data file (**gemeente_gegeneraliseerd_2023**):
 
 1. Right-click on the **layer > Properties > Fields**
 2. Rename (click the pencil to active edit mode):
@@ -49,72 +47,70 @@ Create the municipalities data file (**cbs_gemeente_2022_gegeneraliseerd**):
 4. Right-click on the **layer > Export > Save Features As..**
 5. Use the following settings:
    - Format: `GeoJSON`
-   - File Name: `cbs_gemeente_2022_gegeneraliseerd.geojson`
-   - CRS: `EPSG:28992 - Amersfoort / RD New - Projected`
+   - File Name: Click on the three dots next to the file name on Step 5 of creating data files and choose a directory and save it as following: `gemeente_gegeneraliseerd_2023.geojson`
+   - CRS: `Project CRS: EPSG:28992 - Amersfoort / RD New`
    - Open **"Select fields to export..."**
-     - Select: `code`
+     - Deselect all and only select: `code`
 6. Export by clicking **"Ok"**
 
-Create the safety regions data file (**cbs_veiligheidsregio_2021_gegeneraliseerd**):
+Create the safety regions data file (**veiligheidsregio_gegeneraliseerd_2023**):
 
-1. Right-click on the **layer > Properties > Source fields**
+1. Right-click on the **layer > Properties > fields**
 2. Rename (click the pencil to active edit mode):
    - `statcode` to `code`
 3. **Apply > Ok**
 4. Right-click on the **layer > Export > Save Features As..**
 5. Use the following settings:
    - Format: `GeoJSON`
-   - File Name: `cbs_veiligheidsregio_2021_gegeneraliseerd.geojson`
-   - CRS: `EPSG:28992 - Amersfoort / RD New - Projected`
+   - File Name: Click on the three dots next to the file name on Step 5 of creating data files and choose a directory and save it as following: `veiligheidsregio_gegeneraliseerd_2023.geojson`
+   - CRS: `Project CRS: EPSG:28992 - Amersfoort / RD New`
    - Open **"Select fields to export..."**
-     - Select: `code`
+     - Deselect all and only select: `code`
 6. Export by clicking **"Ok"**
 
-Create the Netherlands data file (**cbs_landsdeel_2022_gegeneraliseerd**):
+Create the Netherlands data file (**landsdeel_gegeneraliseerd_2023**):
 
 1. Select the layer and go to **Vector > Geoprocessing Tools > Dissolve > Run**, this will merge the different areas
 2. Select the new layer and right-click on the **layer > Export > Save Features As..**
 3. Use the following settings:
    - Format: `GeoJSON`
-   - File Name: `cbs_landsdeel_2022_gegeneraliseerd.geojson`
-   - CRS: `EPSG:28992 - Amersfoort / RD New - Projected`
+   - File Name: Click on the three dots next to the file name on Step 5 of creating data files and choose a directory and save it as following: `landsdeel_gegeneraliseerd_2023.geojson`
+   - CRS: `Project CRS: EPSG:28992 - Amersfoort / RD New`
    - Open **"Select fields to export..."**
      - Deselect all
 4. Export by clicking **"Ok"**
 
 To make sure the coordinate system is correct we have to convert the exported files to lat and lon coordinates:
 
 1. Create a new project and add the three files:
-   - `cbs_landsdeel_2022_gegeneraliseerd.geojson`
-   - `cbs_veiligheidsregio_2021_gegeneraliseerd.geojson`
-   - `cbs_gemeente_2022_gegeneraliseerd.geojson`
+   - `landsdeel_gegeneraliseerd_2023.geojson`
+   - `veiligheidsregio_gegeneraliseerd_2023.geojson`
+   - `gemeente_gegeneraliseerd_2023.geojson`
 2. For each layer:
    - Select the new layer and right-click on the **layer > Export > Save Features As..**
    - Format: `GeoJSON`
-   - File Name: `cbs_veiligheidsregio_2021_gegeneraliseerd_WGS84.geojson`
-   - CRS: `EPSG:4326 - WGS 84 - Geographic`
+   - File Name: `veiligheidsregio_gegeneraliseerd_2023_WGS84.geojson`
+   - CRS: `(Default CRS:) EPSG:4326 - WGS 84`
    - Export by clicking **"Ok"**
-   * Make sure the output files contain longitude and latitude coordinates by opening the file and manually checking this, otherwise the next step will not work.
+   * Make sure the output files contain longitude and latitude coordinates by opening the file and manually checking this by hovering over the map and looking for `coordinate` on the bottom toolbar, otherwise the next step will not work.
 
 ### Generate TopoJson:
 
-1. Upload the three different (WGS84) files to: [mapshaper.org](https://mapshaper.org)
-2. Rename the layers:
-   - Change: `cbs_gemeente_2022_gegeneraliseerd` to `gm_features`
-   - Change: `cbs_landsdeel_2022_gegeneraliseerd` to `nl_features`
-   - Change: `cbs_veiligheidsregio_2021_gegeneraliseerd` to `vr_features`
-3. Export to TopoJSON > `nl-vr-gm-high-detail.topo.json`;
-
-> Note: It's possible that the map looks a little streched on MapShaper. This is most likely not an issue and it will display correctly once it's integrated in the Dashboard again.
+> ⚠️ **ATTENTION**: To make custom changes on the TopoJson using mapshaper, please take a look at [this link](https://handsondataviz.org/mapshaper.html) for tips and tricks.
+> It's possible that the map looks a little streched on MapShaper. This is most likely not an issue and it will display correctly once it's integrated in the Dashboard again.
 
-Simplifying:
-(NB: this needs to be separate step; when doing this in the step above the output will not be correct**!!**):
+1. Upload the three different `*_WGS84.geojson` files to: [mapshaper.org](https://mapshaper.org)
+2. Rename the layers by clicking on the name in the dropdown:
+   - Change: `gemeente_gegeneraliseerd_2023_WGS84` to `gm_features`
+   - Change: `landsdeel_gegeneraliseerd_2023_WGS84` to `nl_features`
+   - Change: `veiligheidsregio_gegeneraliseerd_2023_WGS84` to `vr_features`
+3. Export to TopoJSON > `nl-vr-gm-high-detail.topo.json`;
 
 1. Upload the output `nl-vr-gm-high-detail.topo.json` to a new instance of [mapshaper.org](https://mapshaper.org/)
 2. Do for each layer:
    - Select the layer and open the console; simplify using the following command: $ -simplify 27.5%
 3. When all the layers are simplified export to TopoJSON > `nl-vr-gm.topo.json`
-4. Add the new data file to the project at `packages/app/pages/api/topo-json`.
+4. Add the new data file to the project at `packages/app/src/pages/api/topo-json`.
 5. Update the file `topology.ts`, if necessary. (located at `packages/app/src/pages/api/choropleth`)
 
 ### Example of the topology.ts file
@@ -128,25 +124,13 @@ export type CodedGeoProperties = {
   code: string;
 };
 
-export type CodedGeoJSON = FeatureCollection<
-  MultiPolygon | Polygon,
-  CodedGeoProperties
->;
-
-export const nlGeo = topojson.feature(
-  nlTopology,
-  nlTopology.objects.nl_features
-) as CodedGeoJSON;
-
-export const vrGeo = topojson.feature(
-  nlTopology,
-  nlTopology.objects.vr_features
-) as CodedGeoJSON;
-
-export const gmGeo = topojson.feature(
-  nlTopology,
-  nlTopology.objects.gm_features
-) as CodedGeoJSON;
+export type CodedGeoJSON = FeatureCollection<MultiPolygon | Polygon, CodedGeoProperties>;
+
+export const nlGeo = topojson.feature(nlTopology, nlTopology.objects.nl_features) as CodedGeoJSON;
+
+export const vrGeo = topojson.feature(nlTopology, nlTopology.objects.vr_features) as CodedGeoJSON;
+
+export const gmGeo = topojson.feature(nlTopology, nlTopology.objects.gm_features) as CodedGeoJSON;
 ```
 
 Note that the map data is stored as TopoJson. This is because of the size optimizations that TopoJson provides,

diff --git a/packages/app/schema/gm/booster_coverage_archived_20220904.json b/packages/app/schema/gm/booster_coverage_archived_20220904.json
@@ -5,7 +5,6 @@
   "properties": {
     "values": {
       "type": "array",
-      "minItems": 1,
       "maxItems": 2,
       "items": {
         "$ref": "#/definitions/value"

diff --git a/packages/app/schema/gm_collection/__index.json b/packages/app/schema/gm_collection/__index.json
@@ -29,36 +29,40 @@
     },
     "hospital_nice": {
       "type": "array",
-      "maxItems": 344,
+      "minItems": 342,
+      "maxItems": 342,
       "items": {
         "$ref": "hospital_nice.json"
       }
     },
     "hospital_nice_choropleth": {
       "type": "array",
-      "maxItems": 344,
+      "minItems": 342,
+      "maxItems": 342,
       "items": {
         "$ref": "hospital_nice.json"
       }
     },
     "tested_overall": {
       "type": "array",
-      "maxItems": 344,
+      "minItems": 342,
+      "maxItems": 342,
       "items": {
         "$ref": "tested_overall.json"
       }
     },
     "sewer": {
       "type": "array",
-      "maxItems": 344,
+      "minItems": 342,
+      "maxItems": 342,
       "items": {
         "$ref": "sewer.json"
       }
     },
     "vaccine_coverage_per_age_group": {
       "type": "array",
-      "minItems": 688,
-      "maxItems": 688,
+      "minItems": 684,
+      "maxItems": 684,
       "items": {
         "$ref": "vaccine_coverage_per_age_group.json"
       }

diff --git a/packages/app/src/components/choropleth/logic/vr-bounding-box-gm-codes.ts b/packages/app/src/components/choropleth/logic/vr-bounding-box-gm-codes.ts
@@ -122,24 +122,7 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM1896',
     'GM0193',
   ],
-  VR05: [
-    'GM0141',
-    'GM1859',
-    'GM0147',
-    'GM1774',
-    'GM0153',
-    'GM0158',
-    'GM0163',
-    'GM0164',
-    'GM1735',
-    'GM0168',
-    'GM0173',
-    'GM0175',
-    'GM1742',
-    'GM0183',
-    'GM1700',
-    'GM0189',
-  ],
+  VR05: ['GM0141', 'GM1859', 'GM0147', 'GM1774', 'GM0153', 'GM0158', 'GM0163', 'GM0164', 'GM1735', 'GM0168', 'GM0173', 'GM0175', 'GM1742', 'GM0183', 'GM1700', 'GM0189'],
   VR06: [
     'GM0197',
     'GM0141',
@@ -541,10 +524,9 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM0482',
     'GM0613',
     'GM0489',
-    'GM0501',
+    'GM1992',
     'GM0502',
     'GM0503',
-    'GM0530',
     'GM0531',
     'GM0534',
     'GM0537',
@@ -569,7 +551,6 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM0627',
     'GM0629',
     'GM1783',
-    'GM0614',
     'GM0637',
     'GM0638',
     'GM1892',
@@ -639,14 +620,13 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM0613',
     'GM0484',
     'GM0489',
-    'GM0501',
     'GM0502',
     'GM0503',
     'GM0777',
     'GM1924',
     'GM0513',
     'GM1655',
-    'GM0530',
+    'GM1992',
     'GM0531',
     'GM1963',
     'GM0542',
@@ -672,7 +652,6 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM0627',
     'GM0629',
     'GM1783',
-    'GM0614',
     'GM0637',
     'GM0638',
     'GM1892',
@@ -683,7 +662,7 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM0613',
     'GM1959',
     'GM0489',
-    'GM0501',
+    'GM1992',
     'GM0502',
     'GM0503',
     'GM0766',
@@ -725,25 +704,7 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM1892',
     'GM0642',
   ],
-  VR19: [
-    'GM0748',
-    'GM0654',
-    'GM1924',
-    'GM0664',
-    'GM0530',
-    'GM1963',
-    'GM0677',
-    'GM0678',
-    'GM0687',
-    'GM1695',
-    'GM0703',
-    'GM1676',
-    'GM1714',
-    'GM0715',
-    'GM0716',
-    'GM0717',
-    'GM0718',
-  ],
+  VR19: ['GM0748', 'GM0654', 'GM1924', 'GM0664', 'GM1992', 'GM1963', 'GM0677', 'GM0678', 'GM0687', 'GM1695', 'GM0703', 'GM1676', 'GM1714', 'GM0715', 'GM0716', 'GM0717', 'GM0718'],
   VR20: [
     'GM0482',
     'GM0613',
@@ -755,7 +716,6 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM1728',
     'GM0757',
     'GM0758',
-    'GM0501',
     'GM0502',
     'GM0766',
     'GM0505',
@@ -768,7 +728,7 @@ export const vrBoundingBoxGmCodes: Record<string, string[]> = {
     'GM0512',
     'GM1655',
     'GM0523',
-    'GM0530',
+    'GM1992',
     'GM0531',
     'GM0797',
     'GM0798',

diff --git a/packages/app/src/data/gm/vaccinations/empty-coverage-data.ts b/packages/app/src/data/gm/vaccinations/empty-coverage-data.ts
@@ -0,0 +1,42 @@
+// This file was introduced as part of COR-1226 which covers the municipality reorganisation of 2023.
+// During this reorganisation, three municipalities (GMs) have been merged into a single GM, specifically
+// GM0501, GM0530 and GM0614 became GM1992. This reorganisation resulted in missing coverage data
+// for both booster vaccinations as well as base vaccinations. This file combats the missing data by
+// generating 'empty' data objects with no values.
+
+const emptyBoosterCoverageValue = {
+  percentage: null,
+  percentage_label: null,
+  date_of_insertion_unix: null,
+  date_unix: undefined,
+};
+
+const emptyVaccineCoveragePerAgeGroupValue = {
+  has_one_shot_percentage: null,
+  has_one_shot_percentage_label: null,
+  fully_vaccinated_percentage: null,
+  fully_vaccinated_percentage_label: null,
+  date_of_insertion_unix: null,
+  date_unix: null,
+};
+
+export const emptyCoverageData = {
+  booster_coverage_archived_20220904: {
+    values: [
+      { age_group: '12+', ...emptyBoosterCoverageValue },
+      { age_group: '18+', ...emptyBoosterCoverageValue },
+    ],
+  },
+  vaccine_coverage_per_age_group_archived: {
+    values: [
+      { age_group_range: '18+', birthyear_range: '-2003', ...emptyVaccineCoveragePerAgeGroupValue },
+      { age_group_range: '12+', birthyear_range: '-2009', ...emptyVaccineCoveragePerAgeGroupValue },
+    ],
+  },
+  vaccine_coverage_per_age_group_archived_20220908: {
+    values: [
+      { age_group_range: '18+', birthyear_range: '-2004', ...emptyVaccineCoveragePerAgeGroupValue },
+      { age_group_range: '12+', birthyear_range: '-2010', ...emptyVaccineCoveragePerAgeGroupValue },
+    ],
+  },
+};