Skip to content

installationAdvanced

Jump to bottom
Joshua J. Cogliati edited this page Jun 29, 2022 · 10 revisions

This page is for when the recommended methods of installation do not work. Note that these are not regularly tested, so they probably will require some modification for the current version of RAVEN. Miniconda or PIP can be used to install the raven libraries and hints for using them are below. Easybuild has also been used, but is only recommended if you are already using Easybuild for other software.

#standard pip install

This is the standard changes needed for using a pip only install.

Changed the establish_conda_env.sh install command

./scripts/establish_conda_env.sh --install --installation-manager PIP

Build raven as usual:

./build_raven

Add --skip-conda when running tests and with raven_framework

/run_tests --skip-conda

#Finding commands for installation

RAVEN includes a script (that is used by establish_conda_env.sh) to find the commands needed for installati.n

For example, to find the commands for conda, the following can be used:

python3 ./scripts/library_handler.py conda --action install
python3 ./scripts/library_handler.py conda --action install  --subset pip

example output:

conda install --name raven_libraries -y -c conda-forge h5py=2.10 numpy=1.18 scipy=1.2 scikit-learn=0.24 pandas=1.1 xarray=0.16 netcdf4=1.5 matplotlib=3.2 statsmodels cloudpickle=1.6 tensorflow=2.0 python=3 hdf5 swig pylint coverage lxml psutil pip importlib_metadata pyside2 nomkl numexpr imageio=2.9 setuptools numpy-financial dill pyomo pyutilib glpk ipopt coincbc
pip install  ray[default]==1.9.* xmlschema

Then, the conda and the pip install commands can be run manually.

For pip, the command can be found with:

python3 ./scripts/library_handler.py pip --action install

with example output:

pip3 install  h5py==2.10.* numpy==1.18.* scipy==1.2.* scikit-learn==0.24.* pandas==1.1.* xarray==0.16.* netcdf4==1.5.* matplotlib==3.2.* statsmodels cloudpickle==1.6.* tensorflow==2.0.* pylint coverage lxml psutil importlib_metadata pyside2 ray[default]==1.9.* imageio==2.9.* xmlschema setuptools numpy-financial dill pyomo pyutilib

A setup.cfg file can be created by running:

python3 ./scripts/library_handler.py pip --action=setup.cfg > setup.cfg

#Weakening or changing dependencies

The dependencies are stored in the file dependencies.xml This can be modified to change the dependencies used (however, this may result in failures in RAVEN.

For example, to remove the requirement for a specific scipy version, the line:

<scipy>1.2</scipy>

could be changed to:

<scipy/>

#installing without access to the internet

The pip commands can be gotten from scripts/library_handler.py, and changed to just download on an internet connected computer, and then on the non-internet connected computer, the downloaded pip packages can be installed.

Installing with pip in a virtual environment

The following install instructions can be used to install with PIP in a virtual environment

Setting up directories

BASE_DIR="$HOME/raven_libs"
INSTALL_DIR="$BASE_DIR/install"
VE_DIR="$BASE_DIR/ve"
mkdir -p $BASE_DIR
cd $BASE_DIR

Install swig

wget https://downloads.sourceforge.net/project/swig/swig/swig-3.0.12/swig-3.0.12.tar.gz
tar -xvzf swig-3.0.12.tar.gz
cd swig-3.0.12/
./configure --prefix="$INSTALL_DIR"
make
make install
export PATH="$INSTALL_DIR/bin:$PATH"

Installing virtualenv

cd
pip install --upgrade --target="$INSTALL_DIR" virtualenv

Installing other libraries in a virtual environment using pip

python "$INSTALL_DIR"/virtualenv.py "$VE_DIR"
source "$VE_DIR/bin/activate"
pip install numpy==1.11.0 h5py==2.6.0 scipy==0.17.1 scikit-learn==0.17.1 matplotlib==1.5.1

Run with the Virtual Environment

BASE_DIR="$HOME/raven_libs"
INSTALL_DIR="$BASE_DIR/install"
VE_DIR="$BASE_DIR/ve"
export PATH="$INSTALL_DIR/bin:$PATH"
source "$VE_DIR/bin/activate"

Download remotely with PIP

If the machine raven is being installed on can't get pip to install from the internet, pip can download the files on one machine and then install them on another. The below example is to install lazy-import.

mkdir pip_download
cd pip_download
pip download lazy-import

Then copy the files that pip downloads in to the pip_download directory:

cp -a  orig_pip_download new_pip_download  #or scp or whatever it takes to get them to the correct place
and then go to the raven_libraries environment and run:
conda activate raven_libraries
cd new_pip_download
pip install *

If you need a complete pip install, than run:

python scripts/library_handler.py pip --action install

to find out which libraries are needed, and then run something like (substitute in the libraries from library_handler.py)

mkdir pip_download
cd pip_download
pip download h5py==2.9.0 numpy==1.18.1 scipy==1.2.1 scikit-learn==0.21.2 pandas==0.24.2 xarray==0.12.1 netcdf4==1.4.2 matplotlib==3.1.1 statsmodels==0.9.0 cloudpickle==1.1.1 tensorflow==1.13.1 pylint coverage lxml psutil importlib_metadata pyside2 lazy-impor

Then do copy the files in the pip_download directory to the other computer, and use pip install on them.

Example complete compile

This will probably need to be customized for different cluster, but these were used on one CentOS 7.2 cluster in 2017:

BASE_DIR="/opt/raven_libs"
INSTALL_DIR="$BASE_DIR/install"
VE_DIR="$BASE_DIR/ve"
mkdir -p $BASE_DIR
cd $BASE_DIR
wget https://downloads.sourceforge.net/project/swig/swig/swig-3.0.12/swig-3.0.12.tar.gz
tar -xvzf swig-3.0.12.tar.gz
cd swig-3.0.12/
./configure --prefix="$INSTALL_DIR"
make
make install
export PATH="$INSTALL_DIR/bin:$PATH"

cd $BASE_DIR
wget http://prdownloads.sourceforge.net/tcl/tcl8.6.6-src.tar.gz
tar -xvzf tcl8.6.6-src.tar.gz
cd tcl8.6.6/unix
./configure --prefix="$INSTALL_DIR"
make
make test
make install

cd $BASE_DIR
wget http://prdownloads.sourceforge.net/tcl/tk8.6.6-src.tar.gz
tar -xvzf tk8.6.6-src.tar.gz
cd tk8.6.6/unix/
./configure --prefix="$INSTALL_DIR"
make
make install

export LD_LIBRARY_PATH="$INSTALL_DIR"/lib:"$LD_LIBRARY_PATH"

cd $BASE_DIR
wget https://www.python.org/ftp/python/2.7.13/Python-2.7.13.tgz
tar -xvzf Python-2.7.13.tgz
cd Python-2.7.13/
./configure --prefix="$INSTALL_DIR"
make -j16
make install

cd
pip install --upgrade --target="$INSTALL_DIR" virtualenv
python "$INSTALL_DIR"/virtualenv.py "$VE_DIR"
source "$VE_DIR/bin/activate"
pip install numpy==1.11.0 h5py==2.6.0 scipy==0.17.1 scikit-learn==0.17.1
pip install matplotlib

#Commands to run each time.

BASE_DIR="/opt/raven_libs"
INSTALL_DIR="$BASE_DIR/install"
VE_DIR="$BASE_DIR/ve"
export PATH="$INSTALL_DIR/bin:$PATH"
export LD_LIBRARY_PATH="$INSTALL_DIR"/lib:"$LD_LIBRARY_PATH"
source "$VE_DIR/bin/activate"
Clone this wiki locally