Skip to content
This repository has been archived by the owner on Jan 29, 2023. It is now read-only.

Commit

Permalink
Add Deepo2
Browse files Browse the repository at this point in the history 
Add a series of Docker images, and their Dockerfile generator that
allows you to customize your own environment with Lego-like modules, and
automatically resolves the dependencies for you.
  • Loading branch information
@ufoym
ufoym committed Nov 27, 2017
1 parent e132dd2 commit e96f620
Show file tree
Hide file tree
Showing 53 changed files with 3,486 additions and 259 deletions.
62 changes: 62 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/

# Translations
*.mo
*.pot

# Django stuff:
*.log

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# pyenv python configuration file
.python-version
196 changes: 102 additions & 94 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,59 +1,53 @@
![deepo](https://user-images.githubusercontent.com/2270240/32102393-aecf573c-bb4e-11e7-811c-dc673cae7b9c.png)

[![CircleCI](https://img.shields.io/circleci/project/github/ufoym/deepo.svg)](https://circleci.com/gh/ufoym/deepo)
[![docker](https://img.shields.io/docker/pulls/ufoym/deepo.svg)](https://hub.docker.com/r/ufoym/deepo)
![license](https://img.shields.io/github/license/ufoym/deepo.svg)


***Deepo*** is a [*Docker*](http://www.docker.com/) image with a full reproducible deep learning research environment. It contains most popular deep learning frameworks:
[theano](http://deeplearning.net/software/theano),
[tensorflow](http://www.tensorflow.org),
[sonnet](https://github.com/deepmind/sonnet),
[pytorch](http://pytorch.org),
[keras](https://keras.io),
[lasagne](http://lasagne.readthedocs.io),
[mxnet](http://mxnet.incubator.apache.org),
[cntk](https://www.microsoft.com/en-us/cognitive-toolkit),
[chainer](https://chainer.org),
[caffe](http://caffe.berkeleyvision.org),
[torch](http://torch.ch/).
***Deepo*** is a series of [*Docker*](http://www.docker.com/) images that
- allows you to quickly set up your deep learning research environment
- supports almost all commonly used deep learning frameworks: [theano](http://deeplearning.net/software/theano), [tensorflow](http://www.tensorflow.org), [sonnet](https://github.com/deepmind/sonnet), [pytorch](http://pytorch.org), [keras](https://keras.io), [lasagne](http://lasagne.readthedocs.io), [mxnet](http://mxnet.incubator.apache.org), [cntk](https://www.microsoft.com/en-us/cognitive-toolkit), [chainer](https://chainer.org), [caffe](http://caffe.berkeleyvision.org), [torch](http://torch.ch/)

and their Dockerfile generator that
- allows you to customize your own environment with Lego-like modules
- automatically resolves the dependencies for you


# Table of contents
- [Quick Start](#Quick-Start)
- [Installation](#Installation)
- [Usage](#Usage)
- [Customization](#Customization)
- [I hate all-in-one solution](#One)
- [Other python versions](#Python)
- [Build your own customized image](#Build)
- [Comparison to Alternatives](#Comparison)
- [Contributing](#Contributing)
- [Licensing](#Licensing)

---

<a name="Quick-Start"/>

## Quick Start
# Quick Start


<a name="Installation"/>

### Installation
## Installation

#### Step 1. Install [Docker](https://docs.docker.com/engine/installation/) and [nvidia-docker](https://github.com/NVIDIA/nvidia-docker).

#### Step 2. Obtain the Deepo image

You can either directly download the image from Docker Hub, or build the image yourself.
#### Step 2. Obtain the all-in-one image from [Docker Hub](https://hub.docker.com/r/ufoym/deepo)

##### Option 1: Get the image from Docker Hub (recommended)
```bash
docker pull ufoym/deepo
```
##### Option 2: Build the Docker image locally
```bash
git clone https://github.com/ufoym/deepo.git
cd deepo && docker build -t ufoym/deepo .
```
Note that this may take several hours as it compiles a few libraries from scratch.

<a name="Usage"/>

### Usage
## Usage

Now you can try this command:
```bash
Expand All @@ -72,96 +66,34 @@ nvidia-docker run -it -v /host/data:/data -v /host/config:/config ufoym/deepo ba
```
This will make `/host/data` from the host visible as `/data` in the container, and `/host/config` as `/config`. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.

Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with `--ipc=host` or `--shm-size` command line options to `nvidia-docker run`.
```bash
nvidia-docker run -it --ipc=host ufoym/deepo bash
```


_You are now ready to begin your journey._


#### tensorflow
```$ python```
```python
>>> import tensorflow
>>> print(tensorflow.__name__, tensorflow.__version__)
tensorflow 1.3.0
```

#### sonnet
```$ python```
```python
>>> import sonnet
>>> print(sonnet.__name__, sonnet.__path__)
sonnet ['/usr/local/lib/python3.5/dist-packages/sonnet']
```

#### pytorch
```$ python```
```python
>>> import torch
>>> print(torch.__name__, torch.__version__)
torch 0.2.0_3
```

#### keras
```$ python```
```python
>>> import keras
>>> print(keras.__name__, keras.__version__)
keras 2.0.8
```

#### mxnet
```$ python```
```python
>>> import mxnet
>>> print(mxnet.__name__, mxnet.__version__)
mxnet 0.11.0
```

#### cntk
```$ python```
```python
>>> import cntk
>>> print(cntk.__name__, cntk.__version__)
cntk 2.2
```

#### chainer
```$ python```
```python
>>> import chainer
>>> print(chainer.__name__, chainer.__version__)
chainer 3.0.0
```

#### theano
```$ python```
```python
>>> import theano
>>> print(theano.__name__, theano.__version__)
theano 0.10.0beta4+14.gb6e3768
```

#### lasagne
```$ python```
```python
>>> import lasagne
>>> print(lasagne.__name__, lasagne.__version__)
lasagne 0.2.dev1
```

#### caffe
```$ python```
```python
>>> import caffe
>>> print(caffe.__name__, caffe.__version__)
caffe 1.0.0
```

```$ caffe --version```
```
caffe version 1.0.0
```

#### torch
```$ th```
```
│ ______ __ | Torch7
Expand All @@ -173,9 +105,78 @@ caffe version 1.0.0
│th>
```


<a name="Customization"/>

# Customization

Note that `docker pull ufoym/deepo` mentioned in [Quick Start](#Quick-Start) will give you a standard image containing all available deep learning frameworks. You can customize your own environment as well.

<a name="One"/>

## I hate all-in-one solution

If you prefer a specific framework rather than an all-in-one image, just append a tag with the name of the framework.
Take tensorflow for example:
```bash
docker pull ufoym/deepo:tensorflow
```

<a name="Python"/>

## Other python versions

Note that all python-related images use `Python 3.6` by default. If you are unhappy with `Python 3.6`, you can also specify other python versions:
```bash
docker pull ufoym/deepo:py27
```

```bash
docker pull ufoym/deepo:tensorflow-py27
```

Currently, we support `Python 2.7` and `Python 3.6`.

See [https://hub.docker.com/r/ufoym/deepo/tags/](https://hub.docker.com/r/ufoym/deepo/tags/) for a complete list of all available tags. These pre-built images are all built from `docker/Dockerfile.*` and `circle.yml`. See [How to generate `docker/Dockerfile.*` and `circle.yml`](https://github.com/ufoym/deepo/tree/master/scripts) if you are interested in how these files are generated.

<a name="Build"/>

## Build your own customized image with Lego-like modules

#### Step 1. prepare generator

```bash
git clone https://github.com/ufoym/deepo.git
cd deepo/generator
pip install -r requirements.txt
```

#### Step 2. generate your customized Dockerfile

For example, if you like `pytorch` and `lasagne`, then
```bash
python generate.py Dockerfile pytorch lasagne
```

This should generate a Dockerfile that contains everything for building `pytorch` and `lasagne`. Note that the generator can handle automatic dependency processing and topologically sort the lists. So you don't need to worry about missing dependencies and the list order.

You can also specify the version of Python:
```bash
python generate.py Dockerfile pytorch lasagne python==3.6
```

#### Step 3. build your Dockerfile

```bash
docker build -t my/deepo .
```

This may take several minutes as it compiles a few libraries from scratch.


<a name="Comparison"/>

## Comparison to alternatives
# Comparison to alternatives
. | modern-deep-learning | dl-docker | jupyter-deeplearning | Deepo
:------------------------------------------------: | :------------------: | :----------------: | :------------------: | :----------------:
[ubuntu](https://www.ubuntu.com) | 16.04 | 14.04 | 14.04 | 16.04
Expand All @@ -193,8 +194,15 @@ caffe version 1.0.0
[caffe](http://caffe.berkeleyvision.org) | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:
[torch](http://torch.ch/) | :x: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:


<a name="Contributing"/>

# Contributing

We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us.

<a name="Licensing"/>

## Licensing
# Licensing

Deepo is [MIT licensed](https://github.com/ufoym/deepo/blob/master/LICENSE).
Loading

0 comments on commit e96f620

Please sign in to comment.