Add Deepo2

Add a series of Docker images, and their Dockerfile generator that
allows you to customize your own environment with Lego-like modules, and
automatically resolves the dependencies for you.

Author: ufoym

.gitignore

@@ -0,0 +1,62 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# pyenv python configuration file
+.python-version

README.md

@@ -1,59 +1,53 @@
 ![deepo](https://user-images.githubusercontent.com/2270240/32102393-aecf573c-bb4e-11e7-811c-dc673cae7b9c.png)
 
 [![CircleCI](https://img.shields.io/circleci/project/github/ufoym/deepo.svg)](https://circleci.com/gh/ufoym/deepo)
+[![docker](https://img.shields.io/docker/pulls/ufoym/deepo.svg)](https://hub.docker.com/r/ufoym/deepo)
 ![license](https://img.shields.io/github/license/ufoym/deepo.svg)
 
 
-***Deepo*** is a [*Docker*](http://www.docker.com/) image with a full reproducible deep learning research environment. It contains most popular deep learning frameworks:
-[theano](http://deeplearning.net/software/theano),
-[tensorflow](http://www.tensorflow.org),
-[sonnet](https://github.com/deepmind/sonnet),
-[pytorch](http://pytorch.org),
-[keras](https://keras.io),
-[lasagne](http://lasagne.readthedocs.io),
-[mxnet](http://mxnet.incubator.apache.org),
-[cntk](https://www.microsoft.com/en-us/cognitive-toolkit),
-[chainer](https://chainer.org),
-[caffe](http://caffe.berkeleyvision.org),
-[torch](http://torch.ch/).
+***Deepo*** is a series of [*Docker*](http://www.docker.com/) images that
+- allows you to quickly set up your deep learning research environment
+- supports almost all commonly used deep learning frameworks: [theano](http://deeplearning.net/software/theano), [tensorflow](http://www.tensorflow.org), [sonnet](https://github.com/deepmind/sonnet), [pytorch](http://pytorch.org), [keras](https://keras.io), [lasagne](http://lasagne.readthedocs.io), [mxnet](http://mxnet.incubator.apache.org), [cntk](https://www.microsoft.com/en-us/cognitive-toolkit), [chainer](https://chainer.org), [caffe](http://caffe.berkeleyvision.org), [torch](http://torch.ch/)
 
+and their Dockerfile generator that
+- allows you to customize your own environment with Lego-like modules
+- automatically resolves the dependencies for you
+
+
+# Table of contents
 - [Quick Start](#Quick-Start)
   - [Installation](#Installation)
   - [Usage](#Usage)
+- [Customization](#Customization)
+  - [I hate all-in-one solution](#One)
+  - [Other python versions](#Python)
+  - [Build your own customized image](#Build)
 - [Comparison to Alternatives](#Comparison)
+- [Contributing](#Contributing)
 - [Licensing](#Licensing)
 
 ---
 
 <a name="Quick-Start"/>
 
-## Quick Start
+# Quick Start
 
 
 <a name="Installation"/>
 
-### Installation
+## Installation
 
 #### Step 1. Install [Docker](https://docs.docker.com/engine/installation/) and [nvidia-docker](https://github.com/NVIDIA/nvidia-docker).
 
-#### Step 2. Obtain the Deepo image
-
-You can either directly download the image from Docker Hub, or build the image yourself.
+#### Step 2. Obtain the all-in-one image from [Docker Hub](https://hub.docker.com/r/ufoym/deepo)
 
-##### Option 1: Get the image from Docker Hub (recommended)
 ```bash
 docker pull ufoym/deepo
 ```
-##### Option 2: Build the Docker image locally
-```bash
-git clone https://github.com/ufoym/deepo.git
-cd deepo && docker build -t ufoym/deepo .
-```
-Note that this may take several hours as it compiles a few libraries from scratch.
 
 <a name="Usage"/>
 
-### Usage
+## Usage
 
 Now you can try this command:
 ```bash
@@ -72,96 +66,34 @@ nvidia-docker run -it -v /host/data:/data -v /host/config:/config ufoym/deepo ba
 ```
 This will make `/host/data` from the host visible as `/data` in the container, and `/host/config` as `/config`. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.
 
+Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with `--ipc=host` or `--shm-size` command line options to `nvidia-docker run`.
+```bash
+nvidia-docker run -it --ipc=host ufoym/deepo bash
+```
+
 
 _You are now ready to begin your journey._
 
 
-#### tensorflow
 ```$ python```
 ```python
 >>> import tensorflow
->>> print(tensorflow.__name__, tensorflow.__version__)
-tensorflow 1.3.0
-```
-
-#### sonnet
-```$ python```
-```python
 >>> import sonnet
->>> print(sonnet.__name__, sonnet.__path__)
-sonnet ['/usr/local/lib/python3.5/dist-packages/sonnet']
-```
-
-#### pytorch
-```$ python```
-```python
 >>> import torch
->>> print(torch.__name__, torch.__version__)
-torch 0.2.0_3
-```
-
-#### keras
-```$ python```
-```python
 >>> import keras
->>> print(keras.__name__, keras.__version__)
-keras 2.0.8
-```
-
-#### mxnet
-```$ python```
-```python
 >>> import mxnet
->>> print(mxnet.__name__, mxnet.__version__)
-mxnet 0.11.0
-```
-
-#### cntk
-```$ python```
-```python
 >>> import cntk
->>> print(cntk.__name__, cntk.__version__)
-cntk 2.2
-```
-
-#### chainer
-```$ python```
-```python
 >>> import chainer
->>> print(chainer.__name__, chainer.__version__)
-chainer 3.0.0
-```
-
-#### theano
-```$ python```
-```python
 >>> import theano
->>> print(theano.__name__, theano.__version__)
-theano 0.10.0beta4+14.gb6e3768
-```
-
-#### lasagne
-```$ python```
-```python
 >>> import lasagne
->>> print(lasagne.__name__, lasagne.__version__)
-lasagne 0.2.dev1
-```
-
-#### caffe
-```$ python```
-```python
 >>> import caffe
->>> print(caffe.__name__, caffe.__version__)
-caffe 1.0.0
 ```
 
 ```$ caffe --version```
 ```
 caffe version 1.0.0
 ```
 
-#### torch
 ```$ th```
 ```
  │  ______             __   |  Torch7
@@ -173,9 +105,78 @@ caffe version 1.0.0
  │th>
 ```
 
+
+<a name="Customization"/>
+
+# Customization
+
+Note that `docker pull ufoym/deepo` mentioned in [Quick Start](#Quick-Start) will give you a standard image containing all available deep learning frameworks. You can customize your own environment as well.
+
+<a name="One"/>
+
+## I hate all-in-one solution
+
+If you prefer a specific framework rather than an all-in-one image, just append a tag with the name of the framework.
+Take tensorflow for example:
+```bash
+docker pull ufoym/deepo:tensorflow
+```
+
+<a name="Python"/>
+
+## Other python versions
+
+Note that all python-related images use `Python 3.6` by default. If you are unhappy with `Python 3.6`, you can also specify other python versions:
+```bash
+docker pull ufoym/deepo:py27
+```
+
+```bash
+docker pull ufoym/deepo:tensorflow-py27
+```
+
+Currently, we support `Python 2.7` and `Python 3.6`.
+
+See [https://hub.docker.com/r/ufoym/deepo/tags/](https://hub.docker.com/r/ufoym/deepo/tags/) for a complete list of all available tags. These pre-built images are all built from `docker/Dockerfile.*` and `circle.yml`. See [How to generate `docker/Dockerfile.*` and `circle.yml`](https://github.com/ufoym/deepo/tree/master/scripts) if you are interested in how these files are generated.
+
+<a name="Build"/>
+
+## Build your own customized image with Lego-like modules
+
+#### Step 1. prepare generator
+
+```bash
+git clone https://github.com/ufoym/deepo.git
+cd deepo/generator
+pip install -r requirements.txt
+```
+
+#### Step 2. generate your customized Dockerfile
+
+For example, if you like `pytorch` and `lasagne`, then
+```bash
+python generate.py Dockerfile pytorch lasagne
+```
+
+This should generate a Dockerfile that contains everything for building `pytorch` and `lasagne`. Note that the generator can handle automatic dependency processing and topologically sort the lists. So you don't need to worry about missing dependencies and the list order.
+
+You can also specify the version of Python:
+```bash
+python generate.py Dockerfile pytorch lasagne python==3.6
+```
+
+#### Step 3. build your Dockerfile
+
+```bash
+docker build -t my/deepo .
+```
+
+This may take several minutes as it compiles a few libraries from scratch.
+
+
 <a name="Comparison"/>
 
-## Comparison to alternatives
+# Comparison to alternatives
 .                                                  | modern-deep-learning | dl-docker          | jupyter-deeplearning | Deepo
 :------------------------------------------------: | :------------------: | :----------------: | :------------------: | :----------------:
  [ubuntu](https://www.ubuntu.com)                  | 16.04                | 14.04              | 14.04                | 16.04
@@ -193,8 +194,15 @@ caffe version 1.0.0
  [caffe](http://caffe.berkeleyvision.org)          | :heavy_check_mark:   | :heavy_check_mark: | :heavy_check_mark:   | :heavy_check_mark:
  [torch](http://torch.ch/)                         | :x:                  | :heavy_check_mark: | :heavy_check_mark:   | :heavy_check_mark:
 
+
+<a name="Contributing"/>
+
+# Contributing
+
+We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us.
+
 <a name="Licensing"/>
 
-## Licensing
+# Licensing
 
 Deepo is [MIT licensed](https://github.com/ufoym/deepo/blob/master/LICENSE).