add_data.sh - curl commands (should retry for transient errors during transfer)

[drotheram]

Verify downloads have completed without errors, otherwise retry downloads

[anthonyfok]

Ideas:

1. According to https://unix.stackexchange.com/questions/287431/continue-interrupted-curl-download, we can use something like:

     curl -L -O --retry 999 --retry-max-time 0 -C - http://url
   

2. Having some kind of checksum mechanism, e.g. sha256sum, or even with GPG signature, would be nice.

Pending on completion of #68, i.e. after we have moved the multiple curl calls to its own function.

[anthonyfok]

Notes to self:

• Check for type: file vs dir
• --no-act or --dry-run or something similar for add_data.sh to "predict" how much files would be downloaded before download.
• Get sha256 checksum (Git LFS only)

$ jq -r '.content' response.json | base64 -d
version https://git-lfs.github.com/spec/v1
oid sha256:7d2487268dad05f83a0c5be6dcf90419e0bb53cf34706362baae21ac7841c2b5
size 234453739

• Use a repo for compressed cached copies?
  • Encrypted, compressed cached private data on a (public or private?) repo?
  • See if the compressed files are small enough (100MB or less) to avoid Git LFS altogether.
  • See: Uploading Large Files to GitHub. 3 ways to avoid getting error messages… | by Eden Au | Towards Data Science
    https://towardsdatascience.com/uploading-large-files-to-github-dbef518fa1a

[jvanulde]

@anthonyfok are we moving to Sprint 32?

[anthonyfok]

PR #82 partly solves this issue, but not quite. Please see follow-up issues #83 and #90.