Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Dockerfile #76

Merged
merged 1 commit into from
Jan 30, 2019
Merged

Update Dockerfile #76

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
128 changes: 77 additions & 51 deletions docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,80 +1,106 @@
# Pull base image.
FROM ubuntu:16.04

RUN apt-get -qq update
RUN apt-get -qq install -y python python-dev python-pip python-virtualenv
RUN apt-get -qq install -y bzip2
RUN apt-get install --no-install-recommends -y libjsoncpp-dev build-essential cmake git pkg-config libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler libatlas-base-dev libboost-all-dev libgflags-dev libgoogle-glog-dev liblmdb-dev libgflags-dev libgoogle-glog-dev liblmdb-dev
RUN apt-get install -y scons libjsoncpp-dev flex javacc libbison-dev openjdk-8-jdk
RUN apt-get install -y autoconf automake libtool curl make g++ unzip
RUN apt-get install -y cmake wget git zlib1g-dev libbz2-dev libssl-dev liblz4-dev mpich
RUN apt-get install -y libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev libgtk-3-dev
RUN apt-get install -y zlib1g-dev
RUN apt-get install -y libbz2-dev
RUN apt-get install -y libssl-dev
RUN apt-get install -y liblz4-dev
RUN apt-get install -y autoconf automake libtool curl make g++ unzip
RUN apt-get install -y libopenmpi-dev
RUN apt-get install -y libgtest-dev
RUN apt-get install -y ed
RUN apt-get install -y libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
RUN apt-get install -y libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
RUN pip install scons numpy
RUN apt-get update && apt-get upgrade -y && \
apt-get install -y \
build-essential scons autoconf automake libtool curl make g++ unzip \
bzip2 cmake git pkg-config \
python python-dev python-pip python-virtualenv \
wget zlib1g-dev libbz2-dev libssl-dev liblz4-dev mpich \
libjsoncpp-dev flex javacc libbison-dev openjdk-8-jdk \
libleveldb-dev libsnappy-dev libhdf5-serial-dev \
libatlas-base-dev libboost-all-dev libgflags-dev libgoogle-glog-dev \
liblmdb-dev \
libjpeg8-dev libtiff5-dev libjasper-dev libpng12-dev libgtk-3-dev \
libopenmpi-dev \
libgtest-dev ed \
libgtk2.0-dev pkg-config libavcodec-dev \
libavformat-dev libswscale-dev \
libtbb2 libtbb-dev libjpeg-dev libpng-dev \
libtiff-dev libdc1394-22-dev libopenblas-dev

RUN apt-get remove -y libprotobuf-dev protobuf-compiler
RUN cd / && git clone https://github.com/google/protobuf.git
RUN cd protobuf && git checkout v3.3.0 && ./autogen.sh && ./configure --prefix=/usr/local && make -j $(cat /proc/cpuinfo | wc -l) && make install && ldconfig

# Google Test
RUN cd /usr/src/gtest && cmake . && make && mv libgtest* /usr/lib/

# Protobuf
RUN cd / && git clone https://github.com/google/protobuf.git && \
cd protobuf && git checkout v3.6.1 && ./autogen.sh && \
./configure --prefix=/usr/local && \
make -j $(cat /proc/cpuinfo | wc -l) && \
make install && ldconfig
RUN rm -rf /protobuf

# OpenCV
RUN mkdir /opencv && cd /opencv && \
wget -O opencv.zip https://github.com/opencv/opencv/archive/3.3.0.zip && \
unzip opencv.zip && cd opencv-3.3.0 && \
mkdir build && cd build && \
cmake -D CMAKE_BUILD_TYPE=Release \
-D CMAKE_INSTALL_PREFIX=/usr/local .. && \
make -j $(cat /proc/cpuinfo | wc -l) && make install
RUN rm -rf /opencv

# Blosc
RUN cd / && git clone https://github.com/Blosc/c-blosc.git
RUN cd c-blosc && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX='/usr' .. && cmake --build . && ctest && cmake --build . --target install
RUN cd / && git clone https://github.com/Blosc/c-blosc.git && \
cd c-blosc && mkdir build && cd build && \
cmake -DCMAKE_INSTALL_PREFIX='/usr' .. && cmake --build . && \
cmake --build . --target install
RUN rm -rf /c-blosc

# Zstd
RUN cd / && wget https://github.com/facebook/zstd/archive/v1.1.0.tar.gz && tar xf v1.1.0.tar.gz
RUN cd zstd-1.1.0 && make install PREFIX='/usr'
RUN cd /usr/src/gtest && cmake . && make && mv libgtest* /usr/lib/
RUN cd / && wget https://github.com/facebook/zstd/archive/v1.1.0.tar.gz && \
tar xf v1.1.0.tar.gz && cd zstd-1.1.0 && \
make install PREFIX='/usr'

# TileDB
RUN cd / && wget https://github.com/TileDB-Inc/TileDB/archive/0.6.1.tar.gz && tar xf 0.6.1.tar.gz
RUN cd TileDB-0.6.1 && mkdir build && cd build && cmake .. && make -j $(cat /proc/cpuinfo | wc -l) && make install
RUN rm -rf /TileDB-0.6.1
RUN cd / && wget https://github.com/TileDB-Inc/TileDB/archive/1.3.1.tar.gz && \
tar xf 1.3.1.tar.gz && cd TileDB-1.3.1 && mkdir build && cd build && \
../bootstrap --prefix=/usr/local/ && make -j && make install-tiledb
RUN rm -rf /TileDB-1.3.1

# Valijson
RUN cd / && git clone https://github.com/tristanpenman/valijson.git
RUN cd valijson && cp -r include/* /usr/local/include/
# Faiss
RUN cd / && \
wget https://github.com/facebookresearch/faiss/archive/v1.4.0.tar.gz && \
tar xf v1.4.0.tar.gz && \
cd faiss-1.4.0 && rm CMakeLists.txt && \
wget https://gist.githubusercontent.com/luisremis/758c71c6a86cb3e4fe2311c415e07547/raw/3c51b5ed7d3f16a4d684d9ea524602c9f1f8593a/CMakeLists.txt && \
mkdir build && cd build && cmake ../ && make -j
RUN mkdir /usr/local/include/faiss/
RUN cp -r faiss-1.4.0/* /usr/local/include/faiss/
RUN mv faiss-1.4.0/build/lib/libfaiss.so /usr/local/lib/
RUN rm -rf faiss-1.4*

# OpenCV
RUN mkdir /opencv && cd /opencv && wget -O opencv.zip https://github.com/opencv/opencv/archive/3.3.0.zip
RUN cd /opencv && unzip opencv.zip && cd opencv-3.3.0 && mkdir build && cd build && cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local .. && make -j $(cat /proc/cpuinfo | wc -l) && make install && rm -rf /opencv
# PMGD install
RUN cd / && wget https://github.com/intellabs/pmgd/archive/v2.0.0.tar.gz && \
tar xf v2.0.0.tar.gz && mv pmgd-2.0.0 pmgd && cd pmgd && \
make PMOPT=MSYNC

# VCL install
RUN cd / && wget https://github.com/intellabs/vcl/archive/v0.1.0.tar.gz && tar xf v0.1.0.tar.gz
RUN mv vcl-0.1.0 vcl && cd vcl && scons -j16
RUN cd / && wget https://github.com/intellabs/vcl/archive/v1.0.0.tar.gz && \
tar xf v1.0.0.tar.gz && mv vcl-1.0.0 vcl && cd vcl && scons -j16

# PMGD install
RUN cd / && wget https://github.com/intellabs/pmgd/archive/v1.0.0.tar.gz && tar xf v1.0.0.tar.gz
RUN mv pmgd-1.0.0 pmgd && cd pmgd && make
RUN rm -rf /*.tar.gz
# Valijson
RUN cd / && git clone https://github.com/tristanpenman/valijson.git && \
cd valijson && cp -r include/* /usr/local/include/

# VDMS install
RUN cd / && wget https://github.com/intellabs/vdms/archive/v1.0.0.tar.gz && tar xf v1.0.0.tar.gz
RUN mv vdms-1.0.0 vdms && cd vdms && mkdir db && scons -j16 INTEL_PATH=/
RUN git clone https://github.com/intellabs/vdms/ && \
cd vdms && git checkout develop && \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will you be changing this to 2.0 checkout? Should this commit go after the merge?

mkdir db && scons -j16 INTEL_PATH=/

RUN rm -rf /*.tar.gz
RUN rm -rf /var/lib/apt/lists/* /root/.cache

# Setup entry point

RUN echo '#!/bin/bash' > /start.sh
RUN echo 'export LD_LIBRARY_PATH=/pmgd/lib:$(find /usr/local/lib/ / -type f -name "*.so" | xargs dirname | sort | uniq | tr "\n" ":")' >> /start.sh
RUN echo 'export LD_LIBRARY_PATH=/vdms/utils:/pmgd/lib:$(find /usr/local/lib/ / -type f -name "*.so" | xargs dirname | sort | uniq | tr "\n" ":")' >> /start.sh
RUN echo 'cd /vdms' >> /start.sh
RUN echo 'rm -rf db' >> /start.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CAn we add a line here to mkgraph using reasonable params like multiple allocators? Sadly we cannot create one with indexes and such but something. Maybe in the docker with samples

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will use the default number of allocators that can be set through vdms config file (we could change our default). Also, users can modify this param and re-run vdms in the instance, if needed.

RUN echo 'mkdir db' >> /start.sh
RUN echo 'mkdir db/images' >> /start.sh
RUN echo 'mkdir db/images/pngs' >> /start.sh
RUN echo 'mkdir db/images/jpgs' >> /start.sh
RUN echo 'mkdir db/images/tiledb' >> /start.sh
RUN echo 'mkdir db/images/tiledb/tdb' >> /start.sh
RUN echo './vdms' >> /start.sh
RUN chmod 755 /start.sh
ENTRYPOINT "/start.sh"

EXPOSE 55555
CMD ["/start.sh"]