Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the docker #494

Merged
merged 7 commits into from
Mar 8, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#################################################################################################################
# Base Images
#################################################################################################################
FROM ubuntu:14.04

#################################################################################################################
# ENV Setting
#################################################################################################################
ENV CONDA_DIR /opt/conda
ENV PATH $CONDA_DIR/bin:$PATH

#################################################################################################################
# Initial Setting
#################################################################################################################
RUN mkdir -p $CONDA_DIR && \
echo export PATH=$CONDA_DIR/bin:'$PATH' > /etc/profile.d/conda.sh && \
apt-get update && \
apt-get install -y wget git libhdf5-dev g++ graphviz && \
wget --quiet https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
/bin/bash /Miniconda3-latest-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
rm Miniconda3-latest-Linux-x86_64.sh

#################################################################################################################
# User Setting
#################################################################################################################
ENV NB_USER edward
ENV NB_UID 1000

RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
mkdir -p $CONDA_DIR && \
chown edward $CONDA_DIR -R && \
mkdir -p /src && \
chown edward /src

USER edward

#################################################################################################################
# Python Setting
#################################################################################################################
# Python
ARG python_version=3.5.3-0
ARG python_qt_version=4
RUN conda install -y python=${python_version} && \
pip install numpy six tensorflow keras prettytensor pystan pymc3 && \
pip install ipdb pytest pytest-cov python-coveralls coverage==3.7.1 pytest-xdist pep8 pytest-pep8 pydot_ng && \
conda install Pillow scikit-learn notebook pandas matplotlib nose pyyaml h5py && \
conda install -y pyqt=${python_qt_version} && \
pip install edward && \
conda clean -yt

ENV PYTHONPATH='/src/:$PYTHONPATH'

#################################################################################################################
# WORK Jupyter
#################################################################################################################
WORKDIR /src

EXPOSE 8888

CMD jupyter notebook --port=8888 --ip=0.0.0.0

60 changes: 60 additions & 0 deletions docker/Dockerfile-gpu
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
#################################################################################################################
# Reference
# https://github.com/NVIDIA/nvidia-docker
#################################################################################################################
FROM nvidia/cuda:8.0-cudnn5-devel

#################################################################################################################
# ENV Setting
#################################################################################################################
ENV CONDA_DIR /opt/conda
ENV PATH $CONDA_DIR/bin:$PATH

#################################################################################################################
# Initial Setting
#################################################################################################################
RUN mkdir -p $CONDA_DIR && \
echo export PATH=$CONDA_DIR/bin:'$PATH' > /etc/profile.d/conda.sh && \
apt-get update && \
apt-get install -y wget git libhdf5-dev g++ graphviz && \
wget --quiet https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && \
/bin/bash /Miniconda3-latest-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
rm Miniconda3-latest-Linux-x86_64.sh

#################################################################################################################
# User Setting
#################################################################################################################
ENV NB_USER edward
ENV NB_UID 1000

RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
mkdir -p $CONDA_DIR && \
chown edward $CONDA_DIR -R && \
mkdir -p /src && \
chown edward /src

USER edward

#################################################################################################################
# Python Setting
#################################################################################################################
# Python
ARG python_version=3.5.3-0
RUN conda install -y python=${python_version} && \
pip install numpy six tensorflow-gpu keras prettytensor pystan pymc3 && \
pip install ipdb pytest pytest-cov python-coveralls coverage==3.7.1 pytest-xdist pep8 pytest-pep8 pydot_ng && \
conda install Pillow scikit-learn notebook pandas matplotlib nose pyyaml h5py && \
pip install edward && \
conda clean -yt

ENV PYTHONPATH='/src/:$PYTHONPATH'

#################################################################################################################
# WORK Jupyter
#################################################################################################################
WORKDIR /src

EXPOSE 8888

CMD jupyter notebook --port=8888 --ip=0.0.0.0

57 changes: 57 additions & 0 deletions docker/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
help:
@cat Makefile

DATA?="${HOME}/Data"
GPU?=0
DOCKER_FILE=Dockerfile
DOCKER_FILE_GPU=Dockerfile-gpu
DOCKER=docker
DOCKER_GPU=GPU=$(GPU) nvidia-docker
BACKEND=tensorflow
TEST=tests/
SRC=$(shell dirname `pwd`)
CUDA=/usr/local/cuda
LD_LIBRARY=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64

# CPU environment

build:
docker build -t edward --build-arg python_version=3.5 -f $(DOCKER_FILE) .

bash: build
$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data edward bash

ipython: build
$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data edward ipython

notebook: build
$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data --net=host edward

test: build
$(DOCKER) run -it -v $(SRC):/src -v $(DATA):/data edward py.test $(TEST)

# GPU environment

build-gpu:
docker build -t edward-gpu --build-arg python_version=3.5 -f $(DOCKER_FILE_GPU) .

bash-gpu: build-gpu
$(DOCKER_GPU) run -it -v $(SRC):/src -v $(DATA):/data --env CUDA_HOME=${CUDA} \
--env LD_LIBRARY_PATH=${LD_LIBRARY} \
edward-gpu bash

ipython-gpu: build-gpu
$(DOCKER_GPU) run -it -v $(SRC):/src -v $(DATA):/data --env CUDA_HOME=${CUDA} \
--env LD_LIBRARY_PATH=${LD_LIBRARY} \
edward-gpu ipython

notebook-gpu: build-gpu
$(DOCKER_GPU) run -it -v $(SRC):/src -v $(DATA):/data --net=host --env CUDA_HOME=${CUDA} \
--env LD_LIBRARY_PATH=${LD_LIBRARY} \
edward-gpu

test-gpu: build-gpu
$(DOCKER_GPU) run -it -v $(SRC):/src -v $(DATA):/data --env CUDA_HOME=${CUDA} \
--env LD_LIBRARY_PATH=${LD_LIBRARY} \
edward-gpu py.test $(TEST)

80 changes: 80 additions & 0 deletions docker/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Using Edward via Docker

This directory contains `Dockerfile` to make it easy to get up and running with
Edward via [Docker](http://www.docker.com/).

## Installing Docker

General installation instructions are
[on the Docker site](https://docs.docker.com/installation/), but we give some
quick links here:

* [OSX](https://docs.docker.com/installation/mac/): [docker toolbox](https://www.docker.com/toolbox)
* [ubuntu](https://docs.docker.com/installation/ubuntulinux/)

## Installing NVIDIA Docker (GPU Environment)

General installation instructions are
[on the NVIDIA Docker site](https://github.com/NVIDIA/nvidia-docker)

## Running the container

We are using `Makefile` to simplify docker commands within make commands.

### CPU environment

Build the container and start a jupyter notebook

$ make notebook

Build the container and start an iPython shell

$ make ipython

Build the container and start a bash

$ make bash

Build the container and start a test

$ make test

### GPU environment

Build the container and start a jupyter notebook

$ make notebook-gpu

Build the container and start an iPython shell

$ make ipython-gpu

Build the container and start a bash

$ make bash-gpu

Build the container and start a test

$ make test-gpu

For GPU support install NVidia drivers (ideally latest) and
[nvidia-docker](https://github.com/NVIDIA/nvidia-docker). Run using

$ make notebook-gpu GPU=0 # or [ipython, bash]

Mount a volume for external data sets

$ make DATA=~/mydata

Prints all make tasks

$ make help


Note: If you would have a problem running nvidia-docker you may try the old way
we have used. But it is not recommended. If you find a bug in the nvidia-docker report
it there please and try using the nvidia-docker as described above.

$ export CUDA_SO=$(\ls /usr/lib/x86_64-linux-gnu/libcuda.* | xargs -I{} echo '-v {}:{}')
$ export DEVICES=$(\ls /dev/nvidia* | xargs -I{} echo '--device {}:{}')
$ docker run -it -p 8888:8888 $CUDA_SO $DEVICES gcr.io/tensorflow/tensorflow:latest-gpu