Implement download #1
Conversation
| @@ -1,11 +1,59 @@ | |||
| """Download tests.""" | |||
| import glob | |||
There was a problem hiding this comment.
I've been using a file storage package called pystow - it handles compression/uncompression, storing everything in a consistent folder location that is programmatically retrievable, retrieving from URLs/FTP, refreshing the file when it changes - or not - etc. e.g. https://github.com/geneontology/gopreprocess/blob/d258d475dfe65886d39cc795e1fae78c2bab8d36/src/utils/download.py#L44
There was a problem hiding this comment.
That looked interesting, so I just tried it out. It looks like it uses urllib underneath and on a Mac, I get an SSL certificate verification failed error. And then I was looking at their documentation and wasn't seeing an option for ensure_zip, without knowing the specific files we want to download (ensure_open_zip opens a specific file in the zip). We want to download the whole thing so we can parse out the tests we want to run. Do you know how we might use pystow for our specific use-case?
There was a problem hiding this comment.
yeah no worries, its awesome that you tried it out! :)
https://stackoverflow.com/questions/52805115/certificate-verify-failed-unable-to-get-local-issuer-certificate -- that's the Mac fix SO that has worked for me.
Definitely would still need some software to iterate through the downloaded files (or even to iterate through all files in a source URL - e.g. a folder on an FTP site), but instead of keeping track of the path amongst methods, you can always retrieve the path that pystow stores the files by querying its metadata (and then managing the download URLs/etc in a conf file).
pystow does manage zip/unzip and refresh nicely. here's a example of the file structure it creates for me:
SMoxon@SMoxon-M82 .data % ls
ALLIANCE MGI_GPI SEMMEDDB monarch
GAF_OUTPUT MOUSE SEMMEDDB_DOMAIN oaklib
GO ORTHO SEMMEDDB_RANGE pub
HUMAN README.md bioregistry pykeen
HUMAN_ISO README_testing.md gilda
MGI RGD goa_files
SMoxon@SMoxon-M82 .data % cd RGD
SMoxon@SMoxon-M82 RGD % ls
rgd-src.gaf rgd-src.gaf.gz rgd.gaf
SMoxon@SMoxon-M82 RGD %
There was a problem hiding this comment.
I did a little more playing around with this. I got the SSL certificate error fixed from that SO above, but there doesn't seem to be a method in pystow to download and unzip an entire zip file. The only options are to download the zip but leave it unzipped, or download it and unzip but only save a specific file inside, which doesn't quite meet our needs.
No description provided.