CLI: downloaded pseudo archive can be corrupt for install sssp

[sphuber]

Originally reported in #59 which was closed with just an addition to the docs how to circumvent the problem, but apparently it is possible that the download of the pseudo archive in aiida-pseudo install sssp finishes without an error but then the next unpacking step fails with the exception:

EOFError: compressed file ended before the end-of-stream marker was reached

This suggests that the archive was incomplete and in turn suggests that the download was never completed. However, this apparently didn't return an error as the download step was marked successful. This bug was encountered on a QM instance and I have not been able to reproduce it.

[zooks97]

For this and for #65, I would be tempted to keep a small database of MD5 hashes of the files we use in the automatic installers. That way, we could check if the downloaded file is corrupted, has been changed on the server-side, or if the user has a conflicting (and incorrect) file for the --download-only flag.

edit: I see you already made this recommendation in #62, sorry for the duplication!

[sphuber]

Implementing should be easy, the only problem that I see is that aiida-pseudo can become outdated. At least for the SSSP we sometimes release updated version of the archive that fix minor things that do not change the actual important parts (so pseudos and metadata stay the same) but the md5 of the archive can change nonetheless. In this case, the installer would break and we would be forced to release a new version. Together with dependency requirements of packages using aiida-pseudo this may become a mess. I thought of maybe adding an override flag for this case, which would skip the md5 check, but the question is if we then don't nullify the whole concept. So in short, I am not sure what is best

[ltalirz]

I would say the canonical answer is that the MD5 hashes should not be stored by aiida-pseudo, but they should advertised together with an update of the SSSP archive.

I.e. wherever aiida-pseudo looks for downloading the archive, there should also be a place to look for the expected MD5 hash.

If previous versions of the SSSP records didn't have that - no problem; just include a list of MD5 hashes with the next update.