Examples of data/code access failures

[RaoOfPhysics]

This is a living issue documenting failures to get our hands on (meaningful/useful) data and code that are supposedly open.

---

Data-access issues

Several problems related to #20

• First, I looked at the data for WG2’s Fig. SPM.4 (2022), because – unlike WG1 – they haven’t released any source code openly.
  • The data (for Fig. SPM.4 (a) and for Fig. SPM.4 (b)) are available as .xlsx files. But the Excel files don’t contain raw data; they contain data that can be mapped 1:1 with the final visualisation, as seen below:
     
    
• Then, abandoning the idea of “reproducing” those plots, I moved on to WG1’s Fig. SPM.3 (2021). We couldn’t locate the source code in the @IPCC-WG1 organisation, so I decided to poke around the data to see how we could re-build the hex plots. Turns out, it’s not much better, despite WG1 making a lot of their code available and providing somewhat better-documented data.
  • The data in question are available in three CSV files, but they aren’t really CSV, because the top part of the file contains metadata, making it impossible to parse completely with things like R’s read.csv(). So, I opened the files in LibreOffice, only to find that – much like the previous case – the data are merely mapped 1:1 with the visualisation:
     
    

Code-access issues

Minor issues relating to alternative visualisations for #20

• Abandoning the idea to reproduce the above graphs (we can reproduce them, but there are no meaningful data underlying the visualisation), I looked at other visualisations from the IPCC, focusing on Working Group 1’s work, since they make a lot of their code available openly: https://github.com/IPCC-WG1
  • After discussing with @rolyp, we settled on SPM.5 and SPM.6, whose code is available under https://github.com/IPCC-WG1/Chapter-11/blob/main/code/Figure_SPM.5_SM_map.ipynb and https://github.com/IPCC-WG1/Chapter-11/blob/main/code/SMDroughtIndex.ipynb respectively.
  • However, running the second notebook proved problematic. Initially, mplotutils could not be installed, but it turns out that’s because the package isn’t available on PyPI or conda-forge: https://github.com/mathause/mplotutils/blob/main/docs/installation.md
  • After getting mplotutils to run, the next issues were with the utils and data_tables packages. It doesn’t appear that the former is https://pypi.org/project/utils/ and the latter cannot be found. I have contacted the author of the Jupyter Notebook (Mathias Hauser at ETH Zurich) to ask for help.
  • Above issues were because I was working with the notebook in isolation from the repo. The repo has an environment.yml file for building a conda environment. Will explore later and report back.

[rolyp]

Assigned to ICCS Summer School milestone.

[rolyp]

There may be a Climate Informatics 2024 position/commentary paper on reproducibility towards the end of the year, for which some of these examples might be useful.

[rolyp]

Migrating to discussion.