Adding speedy, big uploads

[vsoch]

This will add a web interface view and command line (sregistry-cli) way to upload images using chunks. This will close #123 #133 #109.

Overview

The client to test the new upload (from the command line) is provided as a PR here, and you will need to install this version first:

singularityhub/sregistry-cli#127

# Do this until it tells you not installed
pip uninstall sregistry

# Install the PR branch
git clone -b add/chunked-upload https://www.github.com/vsoch/sregistry-cli
cd sregistry-cli
python setup.py install

• It's recommended to test this on a fresh docker-compose up -d (e.g., never update your previous sregistry server for testing!) and you will notice that the nginx image needs to be built. It will do this automatically.
• Don't forget you need to create accounts (user / superuser) before you can be granted a token (in your user menu).
• The token json needs to be downloaded to $HOME/.sregistry

Then when you use sregistry client, it's helpful to set the client to be registry by exporting

export SREGISTRY_CLIENT=registry

and then a typical push looks like this:

sregistry push manbat.simg --name superhero/batman:mobile

This is all in the docs (and I'm repeating) but please don't hesitate to ask if you have questions.

Nginx Upload

I first had intended to install chunked_upload, and spent quite some time getting this to work in the web interface and command line. It wasn't super speedy but worked generally ok. After frustration with figuring out how to "trick" my server from the command line I took a 180 and implemented the nginx upload module --> https://www.nginx.com/resources/wiki/modules/upload which is a bit weird to get your head around, but largely it works like this:

client submits multipart form data --> 
nginx block has location for upload --> 
nginx processes upload, sends to temporary location --> 
server is forwarded the request (authentication and permissions checked) and the image kept or discarded

It is SO fast and I'm really happy with it :) I had originally intended to keep both the upload module (chunked upload version) and this one, but the latter is so much better I removed the chunked upload dependency.

[vsoch]

ping @victorsndvg !

[vsoch]

hey @victorsndvg I'm going to address the changes needed here so they are easy to find! You ran into the issue of getting an integrity error when trying to push a frozen container, and the bug was triggered under the condition that the container was frozen (meaning that create_new remained true) but then we didn't check that the version was already existing (I'm guessing you tried to push a container with the exact same name / collection / version that was frozen?

I made some changes that should address, but I didn't test in full, so let me know if there are any issues!

[victorsndvg]

Hi @vsoch ,

I checked it and now it's working for me :) This is the reponse trying to overwrite the frozen container:

[1. Collection return status 200 OK]
[================================] 88/88 MB - 00:00:00
[Return status 200 test/test:test@c6c536e5a32959fbfdbd49fb15e11ddf already exists.]

Here is a piece of your first comment inthis PR:

It's recommended to test this on a fresh docker-compose up -d (e.g., never update your previous sregistry server for testing!) and you will notice that the nginx image needs to be built. It will do this automatically.

Do you think this is ready for updating my production version? It's dangerous?

Thanks!

[vsoch]

If all reviewers are happy then that is good condition for merging! As for updating, I would say this is of course up to you - you would get support for larger images. My recommendation would be to back up the database and your containers before doing the update. It looks like there are many ways, and django-archive is one idea.

If you find a backup routine that seems ok, and write down the steps, we can add to the docs.

When/if you are satisfied with this PR (and the matching for the client) please let me know so that I can merge!

[vsoch]

@victorsndvg ready for merge?

[victorsndvg]

For me yes. I did not deploy it in production yet, but this is not going to happen soon.

Thanks!,
Víctor

[vsoch]

Awesome! Merging and will release then!