GSoC: Support sub-national maps in grapher

[larsyencken]

DIFFICULTY: MEDIUM, SIZE: 175 HOURS

Project

We currently only support plotting world maps, for example: a map of life expectancy around the world. We have encountered cases where we have sub-national data (e.g. US states) and would like to use a different map that further sub-divides countries.

Your challenge would be to add a framework to grapher for handling sub-national data, and the mapping support to be able to visualise that data well. Ideally, this framework would work for a huge number of countries with subnational divisions.

Required skills

• Javascript/typescript

Technical background

The current status quo (as of March 2022) is that there is a hardcoded set of countries in our codebase which we reference sometimes by (harmonized) name, sometimes by entity id (the id in our Entities Mysql database table for that country) and sometimes by code (the entity code in the Entities Mysql table). Also hardcoded in our codebase is a specific set of SVG paths that have been ingested from geoJson into a hardcoded representation in our source. All of these have the upside that all necessary information is immediately available to Grapher when it starts running and there is nothing else to fetch to show a map; but we are currently moving away from this setup. (See #1233 for a related issue which links further into the history of this project).

As part of another project to rely less on our MySQL database and provide datasets in more re-usable form, we are currently also reworking the understanding Grapher has of countries and geographic regions. One central piece of this new setup is a countries_regions.csv dataset and an effort to always use entity codes as the primary identifiers. This countries_regions.csv file is designed to allow different groupings. Currently we only have continents and the catch all entity World, but soon we want to have e.g. the World Bank Income Regions defined there as well.

The content of this file would be available to grapher always and as such would be a good point to also store subnational entities (e.g. the States of the US). The same hierarchy mechanism that is used there to group countries into continent entities could be used to group such subnational entities into the corresponding national entity.

GeoJson files will probably live in a separate place for the world map with a mapping of our entity codes to the outline of the matching country. Additional GeoJson files could then be added to show subnational entities when data for such is available.

This project would then be a demonstration of coming up with subnational codes, providing their shape, adding the mapping and adding the logic to fetch and show the relevant outlines if subnational entities are present in a dataset that is shown as a map.

Useful resources

• https://www.geoboundaries.org/index.html#getdata is a project that collects geoJson outlines for subnational entities for a lot of countries

Expected outcomes

• Grapher has a coding scheme for subnational geo regions merged in, covering many countries
• Grapher has a source of geo shapes for subdivisions for many countries
• Grapher has a documented API for charting subnational data on maps, with examples
• (Bonus) ETL project has support for subnational region coding

Potential mentors

• @danyx23
• @danielgavrilov

[mpark63]

Hello OWID community,

My name is Minseo Park and I am a Computer Science undergraduate student at Johns Hopkins University. If I’m given the chance, I would love to contribute to OWID’s mission to succinctly convey existing data regarding the world’s problems we face today.

Regarding the idea to enable sub-national maps in grapher, would we treat these maps as a subset of the current map charts (i.e. could it be a Child chart of a World Root Table)? Furthermore, just to get an idea of the codebase, would we have a <countryName>RegionsToProjection.ts kind of file for every country like ./grapher/mapCharts/WorldRegionsToProjection.ts?

If a prospective mentor could please review my proposal in draft, it would be much appreciated.

Thanks in advance!
Minseo Park

[danyx23]

Hi @mpark63!

Thanks for you interest in participating with us! I filled in a bit of technical background in the project proposal above. As I tried to explain there, we are currently in a transition phase to do country mappings a bit differently. We hope that this will be done by the time GSoC implemenation work starts but we'll have to see. In the current setup something like what you describe would be indeed the way forward but I think that this would scale poorly. Ideally we can put in place a system that can work to fill in subnational data for most countries. So even if this project only demonstrates this for a few large countries, it would be great if the implementation would be done is such a way that there are clear upstream data sources for the svg outlines of the regions and their geo coding so that this could easily be extended to most/all countries in the future.

Please let me know if this clears things up or if it is still confusing. I'll also try to reply to your email later today.

[mpark63]

Thanks for filling us in those technical details! I agree with you that the countries_regions.csv dataset would make the sub-national map implementation more straightforward. A few more questions came to my mind as I was studying the Grapher codebase.

How are the users to access subnational maps? When they click on a country on the world map?
Regarding the API for charting subnational data on maps, would this involve adding features to the Grapher config which fetches data from the MySQL database?
Also, line 699 of grapher/mapCharts/MapChart.tsx defines viewPorts for each possible continental projection. Are subnational maps to be thought similarly as projections to parts of the world map or are they their own type of map charts e.g. one country only without its neighboring countries?

Please let me know if you need me to clarify anything about my questions. Thank you and have a good day!

[danyx23]

These are all good questions. I'll consult with my colleagues but here are my initial thoughts:

How to access subnational maps?

I think that would be a good option. If so I think it is important to visually indicate that this country has sub-national data. One option is that for countries for which sub-national data are available, thin lines are drawn to indicate those regions - e.g. the States in the USA. In the country level map on hover on the US you would still see the national data, but the thin outlines of the states would indicate that this country has finer resolution available on clicking.

An alternative would be to have a toggle checkbox visible when there is subnational data available that toggles in the world map whether the national or subnational values (where available) are shown.

API

I think API is a bit too big a term in the proposal text on reading it again. This mostly means that if there are any author defined options to be added that these would be added to our grapher config json and the schema documentation. As an example of what such an option could be: if the implementation settles on the alternative above with the user toggle checkbox then authors could have the option to hide the checkbox for specific charts and force either the national or subnational view.

Similarity with existing projections

I think it would make sense to make this work similar to projections. Especially if subnational data is only available for a few countries initially, then I think extending the existing dropdown with those countries would be the best way forward. If many countries have subnational data then this gets too crowded but then we could think of switching to normal zooming for maps or adding a search in the projections drop down.

[mpark63]

Thank you for the detailed explanations!

As for how to access the subnational maps, I was leaning towards the alternative option, only because in grapher pages currently, clicking the country already has some functionality such as switching tabs to a line chart over time as in a map of life expectancy around the world. It's definitely worth thinking whether the subregional map should 'override' the current behavior. I also really like the idea of extending the existing dropdown and showing thin outlines on hover for countries that are big enough.

One question I had about the geoJSON is, where do we want to fetch it from? I was thinking in grapher/mapCharts like MapTopology.ts but in separate .ts files.

And on a side note, I mentioned in my proposal that we should 'define url paths that make API calls to fetch sub-national datasets' but I now realized that's not quite correct. When we're looking at a subnational map we might add a query string like ?region=Germany. However, since we already have the necessary datasets in memory, we should not have to make an API call. So as you said, I will describe how the JSON schema for Grapher config will need to be updated. Thank you for clarifying that.