From 47d4deea70d462766563824206d204321d8761d3 Mon Sep 17 00:00:00 2001 From: Richard Gowers Date: Tue, 28 Mar 2017 11:51:16 +0100 Subject: [PATCH] Update 2017-11-11-environments.md --- _posts/2017-11-11-environments.md | 64 +++++++++++++++++-------------- 1 file changed, 36 insertions(+), 28 deletions(-) diff --git a/_posts/2017-11-11-environments.md b/_posts/2017-11-11-environments.md index 7610a0be..c3bf5307 100644 --- a/_posts/2017-11-11-environments.md +++ b/_posts/2017-11-11-environments.md @@ -1,62 +1,75 @@ --- layout: post -title: Working with different package versions +title: Managing software versioning using Conda environments --- -In science projects often take years until they are complete. During that time -your the libraries you are using will be updated and most have new features you -like to use. But sometimes the upgrade of a library means that old programs and -scripts will stop working. It can also be that you need a new library version -for another project you are working on. The solution to this problem is to -install the same programs multiple times. +Research projects can often take months to years to complete, however the +precise version of software they use will often have a shorter lifetime than +this. These new versions of software will often include new features which +might be of great use, but they might also introduce changes which break +your existing work and introduce compatibility issues with other pieces +of software. So whilst for existing projects we might wish to freeze +all pieces of software installed, for newer projects we instead want to +use the most up-to-date versions, leaving us needing to install multiple +versions of multiple different pieces of software. + +In this post we will try to explain how conda and Python virtual environments +can be used to precisely manage the software used across many independent +research projects. -In this post I will explain how conda and python virtual envs can be used to -manage different package versions. # Conda Environments -[conda]() is a general package manager for scientific applications. It is mostly -used for python packages but the system can be used with any programs. The -[conda-forge] community also provides a large collection of scientific software -for python, R and perl. Conda should be your first choice to manage different +[Conda](https://conda.io/docs/index.html) is a general package manager for scientific +applications. It is mostly used for Python packages but the system can be used with +any programs. The [conda-forge] community also provides a large collection of scientific software +for Python, R and perl. Conda should be your first choice to manage different software versions. -In this guide we will only concentrate on creating and managing environments +In this guide we will concentrate only on creating and managing environments with conda. For more information on general installation of package please refer to the [official documentation](). -As a preparation add our conda-channel to your configuration +Software is made available through different conda channels, which each act as a +source for different software. When attempting to install packages into a conda +environment, these channels are searched. In this post we will be using the +MDAnalysis channel which you can add to your configuration like so: {% highlight bash %} conda config --add channels MDAnalysis {% endhighlight %} +For each research project, it is advised that you create a new environment so that +the software used in each project does interfere across different projects. To create a new environment for your next project that uses MDAnalysis in version 0.15.0 run: {% highlight bash %} -conda create -n mda-15 MDAnalysis=0.15.0 -y +conda create -n myproject MDAnalysis=0.15.0 -y {% endhighlight %} -This only create a new environment. You still have to activate it to be able to -use it. +This has created a new software environment called `myproject` but has not affected +anything currently! To have access to all the software installed within it we +must first activate it {% highlight bash %} -source activate mda-15 +source activate myproject {% endhighlight %} -To list your environments +To list your available environments {% highlight bash %} conda env list {% endhighlight %} A nice feature of using conda-environments is that they are easy to share with -colleagues or transferred to other computers. To store the state of the -environment we created in a file called `mda-15-environment` +colleagues or transferred to other computers. This allows all team members on +a project to use an identical set of software and makes your research projects +to be reproducible. To store the state of the environment we created in a file +called `myproject-environment` {% highlight bash %} -conda list --explicit --md5 -n mda-15 > mda-15-environment +conda list --explicit --md5 -n myproject > myproject-environment {% endhighlight %} You can now copy this file to a colleague or onto another computer. The first 3 @@ -68,11 +81,6 @@ lines also contain instructions how this file can be used with conda. # platform: linux-64 {% endhighlight %} -Instead of just creating environments for specific versions of software packages -you can of course also create a new environment for each project you are working -on. - - More information about conda environments can be found in the [official documentation]().