Virtualbox requirements

[satra]

@mjtravers @yarikoptic - add as necessary

• desktop interface/x11
• docker
• singularity 2.5.1

wonder if the following should be installed or done through containers as well

• git
• vi, emacs, pico, nano, gedit

[mjtravers]

Here is the image: https://training.repronim.org/repronim-training.ova (it's ~2GB in size)

The password is same as the username: vagrant

Installed:

• Ubuntu 16.04 desktop
• Docker 18.03.1-ce
• Singularity 2.5.1-dist
• git 2.7.4
• vi 7.4
• emacs 25.3.2
• pico 2.5.3
• nano 2.5.3
• gedit 3.18.3

[satra]

@mjtravers - a few observations:

• we should add neurodebian as a source
• cannot build an image with the installed singularity

[satra]

@yarikoptic - can you try the vm? i can't build with singularity installed from neurodebian either. i get a "implausibly old timestamp" error.

[satra]

actually the more pertinent error is:
"ERROR: 'Bootstrap' type not supported: docker"

[djarecka]

@satra - I tested quickly docker within the vm, and it works fine. I also had no problem with saving files to a mounted directory

[satra]

@djarecka - the error comes from singularity - not docker.

[satra]

does docker save the files with the right permissions?

[djarecka]

yes, i was just referring to our discussion earlier today

[djarecka]

@satra - ok, i see what you mean... no, the permission is not correct, can work on docker groups tomorrow

[mjtravers]

I am unable to reproduce the singularity error. I am running these build commands taken from the singularity docs page:

singularity build hello-world.simg shub://vsoch/hello-world
singularity build lolcow.simg docker://godlovedc/lolcow

What's the singularity build command being run that is failing with errors?

However, I am getting errors when building a Docker image, though. I cleaned up the packer script I am using for the build and have an updated the image that runs Docker build cleanly for me:

https://training.repronim.org/repronim-training.ova

[satra]

@mjtravers - building from a pre-built image works but not when bootstrapping from docker:

try a Singularity bootstrap file:

Bootstrap: docker
From: busybox

and do sudo singularity build test.img Singularity

[mjtravers]

I'll take a look.
BTW: The new ova file I uploaded this morning has a NeuroDebian apt source setup.

[satra]

thanks - downloading now

[mjtravers]

Could it be simply an issue of bootstrapping being depreciated?

https://singularity.lbl.gov/docs-bootstrap

[satra]

yes, but you can bootstrap using build now. they just converged a single image building command.

[satra]

worst case, i will compile from source and put some debug statements before the error is generated.

[mjtravers]

Ah, I see... digging through docs now

[mjtravers]

The new ova file may work for you. I was able to build using your instructions above.

[satra]

importing now.

[mjtravers]

(crossing fingers)

[satra]

no go:

[mjtravers]

Weirdly, I am not getting the error

[djarecka]

@satra - it works for me, have a similar output as @mjtravers

[mjtravers]

I am running the VM in VirtualBox 5.2.12 on an Ubuntu 16.04 machine.

[djarecka]

I updated my VB yesterday: 5.2.12 r122591 (Qt5.6.3) on OSX (10.12.1)

[yarikoptic]

casting my vote: works for me (proof: http://www.onerussian.com/tmp/gkrellShoot_05-30-18_120709.png)

[satra]

did anyone mount an outside folder using shared folders?

[djarecka]

@satra i didn't

[mjtravers]

@satra did not

[satra]

so it didn't have to do with shared folders, but changing the number of cores for the machine.

can someone verify if you can change the number of cores and still build?

[djarecka]

it works when I changed CPU

[dnkennedy]

For what it's worth, I have @satra 's error from above...

[satra]

@dnkennedy:

• i restarted my computer
• set my docker preferences to not use all my cores and memory
• imported @mjtravers ova (without changing any settings in my virtualbox)
• created this Singularity file

Bootstrap: docker
From: busybox

• ran the build
  sudo singularity build test.simg Singularity

that worked for now.

i did try to debug the error i was having and the pointers were all the direction of some form of memory corruption. examples of weird string substitution happening in the shell:
https://www.dropbox.com/s/7p2ygcyg77xy96t/Screenshot%202018-05-30%2017.53.11.png?dl=0
https://www.dropbox.com/s/2b7jp236k6xfak1/Screenshot%202018-05-30%2017.47.21.png?dl=0
https://www.dropbox.com/s/eoyjlywaw3q7011/Screenshot%202018-05-30%2017.39.22.png?dl=0

[satra]

@mjtravers, @dnkennedy - i cannot seem to copy and paste from my host into virtualbox and vice versa. i haven't used virtualbox in a while, any ideas?

[mjtravers]

@satra Normally, selecting "Devices -> Shared clipboard -> bidirectional" from the VirtualBox menu does the trick but I think the VBoxGuestAdditions is not installed properly. Let me tinker and get back to you.

[dnkennedy]

I was just going to ask the same thing about cut and paste. I know it did work on the Savoy Course VM...

[mjtravers]

@satra @dnkennedy
There is a new VM image up on the training website:

https://training.repronim.org/repronim-training.ova

To get the VirtualBox Guest Additions working, I needed to make some significant changes to how the baseline Ubuntu 16.04 desktop image is built. Here are the resulting changes to the VM:

• Copy-and-paste is working for me. I will hold off declaring victory until others verify.
• The VM file size has increased to ~3GB
• The vagrant account is automatically logged in to the desktop at startup. ("vagrant" is still the account password when requested)

I set the default memory size to 2048MB. Is that enough? I was pondering setting it to 4096MB but then thought it best to keep it low as possible and let users increase if desired.

[dnkennedy]

Yay: the Singularity example and the copy & paste work for me!!! thanks, @mjtravers

[djarecka]

@mjtravers - Singularity and copy&paste works for me too, thank you! It's a bit funny to have 2 types of copy&paste (OSX and Linux) on one laptop, but will get used to :)

[satra]

@mjtravers - unfortunately i am still getting the darn error!

[mjtravers]

@satra The copy&paste error or the Singularity error?

[satra]

singularity error - my old tricks didn't work - i'll try one more thing.

[satra]

@mjtravers - my latest solution.

1. stop docker
2. import ova
3. sleep
4. test singularity build first before doing anything else

[satra]

well that was short lived!! i again cannot create the image! quite fed up right now!

[mjtravers]

@satra For web-based, I will look into creating an AWS image that trainees can use. It will mean the trainee will need access to an AWS subscription that will pay for the EC2 time used. Did you have another web-based solution in mind?

Also, so as not to duplicate work, I am installing Miniconda in the VM to set up the requirements.txt package list JB sent on Friday.

[satra]

@mjtravers - if you install python-virtualenv and python-pip at the system level i think we can use that to create JB's environment - that will reduce the size a little bit.

miniconda will pull in too many things.

[satra]

@mjtravers - regarding the web-based - ec2 is fine. we would support it via repronim. (@dnkennedy).

[djarecka]

I've just realized that when I'm running VM, my local Docker is crashing, could be related to the issue reported by @satra

[mjtravers]

@satra @jbpoline JB's requirements.txt file has versions attached to the packages which are putting some constraints on the install, which is why I started working with Miniconda. For example, ipython 6.1.0 requires python >= 3.3. Virtualenv (from what I can tell so far, not very familiar with it) will need the additional python version installed in the VM system outside venv and alongside the existing python 2.7. Miniconda handles additional python versions within the conda environments themselves which is a lot easier to manage and does not require the system installation of new python versions. The Miniconda baseline footprint doesn't appear to be very large. Are you thinking of the full Anaconda environment? Here are the baseline packages installed by Miniconda for a new environment:

• certifi:    2016.2.28-py36_0
  

• openssl:    1.0.2l-0        
  

• pip:        9.0.1-py36_1    
  

• python:     3.6.2-0         (I installed python 3.6 in my test env, so this is additional)
  

• readline:   6.2-2           
  

• setuptools: 36.4.0-py36_1   
  

• sqlite:     3.13.0-0        
  

• tk:         8.5.18-0        
  

• wheel:      0.29.0-py36_0   
  

• xz:         5.2.3-0         
  

• zlib:       1.2.11-0  
  

Another way to approach the problem is: JB, how attached are you to the versions specified in your requirements.txt file? Could we simply pull all the versions that work with Python 2.7?

[satra]

@mjtravers - in that case just use miniconda.

[mjtravers]

@satra - Whew. Thanks! :)

@satra @djarecka Both of you are on Mac? Did you install Docker via "Docker for Mac" or "Docker Toolbox"? I've been digging into Docker's use of its own VirtualBox as a possible cause to the Docker/Singularity problem and was wondering how Docker was installed on your systems.

https://docs.docker.com/docker-for-mac/docker-toolbox/

One tidbit that caught my eye was: "Docker for Mac uses HyperKit instead of Virtual Box."

[djarecka]

@mjtravers - we both have Mac and we both have Docker for Mac

[mjtravers]

@djarecka Ah, okay. Does not seem relevant then.

[djarecka]

@mjtravers - i restarted, removed all data from docker, and it's been working for the last 15min. So maybe it was just a coincidence, but don't remember having this problem with Docker...

[satra]

@mjtravers - finally tracked this down:

ReproNim/neurodocker#180

we will fix this in the container. but let's install dos2unix (just in case :) )

[mjtravers]

@satra Will make sure dos2unix is in the VM. Wow, that took some serious digging to locate

[jbpoline]

sorry I'm seeing this only now - I think any version above these would be fine - I just need to test my notebook but very confident there will be no issues .. cheers JB

…

On Mon, Jun 4, 2018 at 1:43 PM, Matt Travers ***@***.***> wrote: @satra <https://github.com/satra> @jbpoline <https://github.com/jbpoline> JB's requirements.txt file has versions attached to the packages which are putting some constraints on the install, which is why I started working with Miniconda. For example, ipython 6.1.0 requires python >= 3.3. Virtualenv (from what I can tell so far, not very familiar with it) will need the additional python version installed in the VM system outside venv and alongside the existing python 2.7. Miniconda handles additional python versions within the conda environments themselves which is a lot easier to manage and does not require the system installation of new python versions. The Miniconda baseline footprint doesn't appear to be very large. Are you thinking of the full Anaconda environment? Here are the baseline packages installed by Miniconda for a new environment: - certifi: 2016.2.28-py36_0 - openssl: 1.0.2l-0 - pip: 9.0.1-py36_1 - python: 3.6.2-0 - readline: 6.2-2 - setuptools: 36.4.0-py36_1 - sqlite: 3.13.0-0 - tk: 8.5.18-0 - wheel: 0.29.0-py36_0 - xz: 5.2.3-0 - zlib: 1.2.11-0 Another way to approach the problem is: JB, how attached are you to the versions specified in your requirements.txt file? Could we simply pull all the versions that work with Python 2.7? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQyaMwSo6QBKuPlqZ1kPnqv2r0dN3qcks5t5XHbgaJpZM4UK5DX> .

[satra]

closing in favor of #15