[WIP] Remote Build Plugin

Dockerfile

@@ -7,6 +7,7 @@ ARG ENABLE_LDAP=false
 ARG ENABLE_PAM=false
 ARG ENABLE_PGP=false
 ARG ENABLE_GOOGLEBUILD=false
+ARG ENABLE_REMOTEBUILD=true
 ARG ENABLE_GLOBUS=false
 ARG ENABLE_SAML=false
 
@@ -65,6 +66,26 @@ RUN if $ENABLE_PGP; then pip install pgpdump>=1.4; fi;
 RUN if $ENABLE_GOOGLEBUILD; then pip install sregistry[google-build] ; fi;
 ENV SREGISTRY_GOOGLE_STORAGE_PRIVATE=true
 
+# Install websocket requisite for remote build
+RUN if $ENABLE_REMOTEBUILD; then \
+apt-get update && apt-get install -y \
+    cryptsetup \
+    debootstrap \
+    yum ; \
+pip install channels channels_redis ; \
+export VERSION=1.13.4 OS=linux ARCH=amd64 && \
+  wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
+  tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
+  rm go$VERSION.$OS-$ARCH.tar.gz; \
+export VERSION=3.5.0 && \
+    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-${VERSION}.tar.gz && \
+    tar -xzf singularity-${VERSION}.tar.gz -C /tmp && \
+        rm singularity-${VERSION}.tar.gz; \
+cd /tmp/singularity && export  PATH=/usr/local/go/bin:$PATH && ./mconfig && \
+    make -C builddir && \
+    make -C builddir install; \
+fi;
+
 # Install Globus (uncomment if wanted)
 RUN if $ENABLE_GLOBUS; then /bin/bash /code/scripts/globus/globus-install.sh ; fi;
 

docs/_docs/plugins/README.md

@@ -25,6 +25,7 @@ your registries' local `shub/settings/secrets.py` file.
  - [SAML](saml): Authentication with SAML
  - [Google Build](google-build) provides build and storage on Google Cloud.
  - [Keystore](pgp) provides a standard keystore for signing containers
+ - [Remote Build](google-build) provides remote build library as per Sylabs API
 
 The Dockerfile has some build arguments to build the Docker image according to the plugins software requirements. These variables are set to false by default:
 
@@ -34,11 +35,12 @@ ARG ENABLE_PAM=false
 ARG ENABLE_GOOGLEBUILD=false
 ARG ENABLE_GLOBUS=false
 ARG ENABLE_SAML=false
+ARG ENABLE_REMOTEBUILD=false
 ```
 
 Therefore, if you want to install the requirements of all current supported plugins, you can build the image as follows: 
 ```bash
-docker build --build-arg ENABLE_LDAP=true --build-arg ENABLE_PAM=true  --build-arg ENABLE_GOOGLEBUILD=true --build-arg ENABLE_GLOBUS=true --build-arg ENABLE_SAML=true -t quay.io/vanessa/sregistry .
+docker build --build-arg ENABLE_LDAP=true --build-arg ENABLE_PAM=true  --build-arg ENABLE_GOOGLEBUILD=true --build-arg ENABLE_GLOBUS=true --build-arg ENABLE_SAML=true --build-arg ENABLE_REMOTEBUILD=true -t quay.io/vanessa/sregistry .
 ```
 
 
@@ -67,6 +69,6 @@ RUN if $ENABLE_{PLUGIN_NAME}; then {INSTALLATION_COMMAND}; fi;
 ```
 ## Writing Documentation
 Documentation for your plugin is just as important as the plugin itself! You should create a subfolder under
-`docs/pages/plugins/<your-plugin>` with an appropriate README.md that is linked to in this file.
+`docs/_docs/plugins/<your-plugin>` with an appropriate README.md that is linked to in this file.
 Use the others as examples to guide you.
 

docs/_docs/plugins/remote_build/README.md

@@ -0,0 +1,101 @@
+---
+title: "Plugin: Custom Builder and Storage"
+pdf: true
+toc: true
+permalink: docs/plugins/remote-build
+---
+
+# Plugin: Remote Build mimic sylabs API
+
+## Configure sregistry
+
+By default, remote build is disabled. To configure sregistry to 
+use Google Cloud build and Storage, in settings/config.py you can enable the plugin by 
+uncommenting it from the list here:
+
+```bash
+PLUGINS_ENABLED = [
+#    'ldap_auth',
+#    'saml_auth',
+#    'globus',
+#     'google_build',
+     'remote_build'
+]
+```
+You will need to build the image locally with, at least, the build argument ENABLE_REMOTEBUILD set to true:
+
+```bash
+$ docker build --build-arg ENABLE_REMOTEBUILD=true -t quay.io/quay.io/vanessa/sregistry .
+```
+
+## Secrets
+
+Next, set the following variables in `shub/settings/secrets.py`, 
+that you can create from `dummy_secrets.py` in the shub/settings folder.
+The first two speak for themselves, your project name and path to your
+Google Application Credentials.
+
+## Singularity Remote Build
+
+This is a first effort to provide support to `remote build`.
+Freshly build image on application server (aka `worker`) is then pushed on library...
+So we need [singularity client](https://sylabs.io) installed on application server.
+
+### Motivation
+
+Remote build provide user without local compute resource (for instance), 
+to build remotely and retrieved locally container image on their desktop.
+
+It's also a way to share quickly conitainer image.
+
+You can proceed through [googlebuild](https://singularityhub.github.io/sregistry/docs/plugins/google-build) plugin,
+but it's not everyone that have the opportunity to access google cloud, for security reason for instance...
+
+### In the nutshell
+
+This basic implementation of the Sylabs Library API use django
+[channels](https://channels.readthedocs.io/en/latest/) Websocket Server
+[Daphne](https://github.com/django/daphne/) and [ASGI](https://channels.readthedocs.io/en/latest/asgi.html)
+
+### Requisite
+
+This is the same than for [Singularity Push](#singularity-push)
+
+### Install
+
+You need to build new locally image, with new argument ENABLE_REMOTEBUILD set to true:
+
+```
+docker build --build-arg ENABLE_REMOTEBUILD=true -t quay.io/quay.io/vanessa/sregistry .
+```
+
+### Utilisation
+
+To build remotely image on [sregistry](https://singularityhub.github.io/sregistry):
+
+```
+singularity build --builder https://127.0.0.1 --remote <image name> <spec file>
+```
+
+Container image `<image name>` will then be generate locally and on remote library.
+
+To generate image only remotely, use:
+
+```
+singularity build --builder https://127.0.0.1 --detached <spec file>
+```
+
+### Features
+
+- [X] build on remote library
+- [X] retrieve locally build
+- [ ] implement `WYSIWYG` via web interface through popular [django-ace](https://github.com/django-ace/django-ace)
+
+### TODO :boom:
+
+- [ ] Automatically create collection `remote-builds`
+- [ ] Re-use Django Push View in Build View
+- [ ] Optimize channels consumer `BuildConsumer`
+- [ ] Extend collection spacename to username
+- [ ] Dedicated worker for build
+

https/nginx.conf.wss

@@ -0,0 +1,163 @@
+upstream websocket {
+    ip_hash;
+    server 172.17.0.6:3032 fail_timeout=0;
+}
+
+server {
+  listen                *:80;
+  server_name           localhost;
+
+  client_max_body_size 10024M;
+  client_body_buffer_size 10024M;
+  client_body_timeout 120;
+
+  add_header X-Clacks-Overhead "GNU Terry Pratchett";
+  add_header X-Clacks-Overhead "GNU Terry Pratchet";
+  add_header Access-Control-Allow-Origin *;
+  add_header 'Access-Control-Allow-Credentials' 'true';
+  add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
+  add_header 'Access-Control-Allow-Headers' 'Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
+
+  location /images {
+    alias /var/www/images;
+  }
+
+  location ~* \.(php|aspx|myadmin|asp)$ {
+    deny all;
+  }
+
+  location / {
+    include /etc/nginx/uwsgi_params.par;
+    uwsgi_pass uwsgi:3031;
+    uwsgi_max_temp_file_size 10024m;
+  }
+
+  location /static {
+    alias /var/www/static;
+  }
+
+  # Upload form should be submitted to this location
+  location /upload {
+
+        # Pass altered request body to this location
+        upload_pass   /api/uploads/complete/;
+
+        # Store files to this directory
+        # The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
+        upload_store /var/www/images/_upload 1;        
+        upload_store_access user:rw group:rw all:rw;
+
+        # Set specified fields in request body
+        upload_set_form_field $upload_field_name.name "$upload_file_name";
+        upload_set_form_field $upload_field_name.content_type "$upload_content_type";
+        upload_set_form_field $upload_field_name.path "$upload_tmp_path";
+
+        # Inform backend about hash and size of a file
+        upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
+        upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";
+
+        upload_pass_form_field "^submit$|^description$";
+        upload_pass_form_field "^SREGISTRY_EVENT$";
+        upload_pass_form_field "^collection$";
+        upload_pass_form_field "^name$";
+        upload_pass_form_field "^tag$";
+        upload_cleanup 400-599;
+
+    }
+
+    location /v1/build-ws/ {
+        proxy_pass http://websocket; # daphne (ASGI) listening on port 3032
+        proxy_http_version 1.1;
+        proxy_read_timeout 86400;
+        proxy_redirect     off;
+
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Host $server_name;
+   }
+}
+
+server {
+
+    listen 443;
+    server_name localhost;
+
+    root html;
+    client_max_body_size 10024M;
+    client_body_buffer_size 10024M;
+
+    ssl on;
+    ssl_certificate /etc/ssl/certs/chained.pem;
+    ssl_certificate_key /etc/ssl/private/domain.key;
+    ssl_session_timeout 5m;
+    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
+    ssl_ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA;
+    ssl_session_cache shared:SSL:50m;
+    ssl_dhparam /etc/ssl/certs/dhparam.pem;
+    ssl_prefer_server_ciphers on;
+
+    location /images {
+        alias /var/www/images;
+    }
+
+    location /static {
+        alias /var/www/static;
+    }
+
+    location ~* \.(php|aspx|myadmin|asp)$ {
+      deny all;
+    }
+
+    # Upload form should be submitted to this location
+    location /upload {
+
+        # Pass altered request body to this location
+        upload_pass   /api/uploads/complete/;
+
+        # Store files to this directory
+        # The directory is hashed, subdirectories 0 1 2 3 4 5 6 7 8 9 should exist
+        upload_store /var/www/images/_upload 1;        
+        upload_store_access user:rw group:rw all:rw;
+
+        # Set specified fields in request body
+        upload_set_form_field $upload_field_name.name "$upload_file_name";
+        upload_set_form_field $upload_field_name.content_type "$upload_content_type";
+        upload_set_form_field $upload_field_name.path "$upload_tmp_path";
+
+        # Inform backend about hash and size of a file
+        upload_aggregate_form_field "$upload_field_name.md5" "$upload_file_md5";
+        upload_aggregate_form_field "$upload_field_name.size" "$upload_file_size";
+
+        upload_pass_form_field "^submit$|^description$";
+        upload_pass_form_field "^SREGISTRY_EVENT$";
+        upload_pass_form_field "^collection$";
+        upload_pass_form_field "^name$";
+        upload_pass_form_field "^tag$";
+        upload_cleanup 400-599;
+
+    }
+
+    location / {
+            include /etc/nginx/uwsgi_params.par;
+            uwsgi_pass uwsgi:3031;
+            uwsgi_max_temp_file_size 10024m;
+        }
+
+    location /v1/build-ws/ {
+        proxy_pass http://websocket; # daphne (ASGI) listening on port 3032
+        proxy_http_version 1.1;
+        proxy_read_timeout 86400;
+        proxy_redirect     off;
+
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Host $server_name;
+   }
+}
+