Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

Latest commit

 

History

History
137 lines (94 loc) · 4.85 KB

how-to-setup-dev-box.md

File metadata and controls

137 lines (94 loc) · 4.85 KB
Raw

Setup dev-box document

Table of Contents

What's dev-box?

Dev-Box is a docker container which contains necessary dependent software for paictl to deploy and manage you cluster. With a dev-box, you no longer need to install the software in your host environment, make your host environment's software package clean.

Install docker on your server

dev-box is a docker container used to boot up or/and maintain a PAI cluster. For convenience, we provide a prebuild Docker image on Docker Hub.

Use prebuild dev-box image

Notice that dev-box should run on a machine outside of PAI cluster, it shouldn't run on any PAI cluster node. replace below v0.x.y to latest release, which can be found here. For example: v0.9.5

# Pull the dev-box image from Docker Hub
sudo docker pull docker.io/openpai/dev-box:v0.x.y

# Run your dev-box
# Assume the path of custom-hadoop-binary-path in your service-configuration is /pathHadoop,
#   and the path of your cluster-configuration is /pathConfiguration.
# By now, you can leave it as it is, we only mount those two directories into docker container for later usage.
sudo docker run -itd \
        -e COLUMNS=$COLUMNS -e LINES=$LINES -e TERM=$TERM \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v /pathConfiguration:/cluster-configuration  \
        -v /hadoop-binary:/hadoop-binary  \
        --pid=host \
        --privileged=true \
        --net=host \
        --name=dev-box \
        docker.io/openpai/dev-box:v0.x.y

# Working in your dev-box
sudo docker exec -it dev-box /bin/bash
cd /pai

# Now you are free to configure your cluster and run PAI commands...

build dev-box docker image on your own

Build dev-box on the latest code

Notice, replace v0.x.y as above, if you are trying to deploy OpenPAI. You can also remove -b parameter, if you are contributing to latest OpenPAI, but it's unstable.

# if you are trying to install latest release, replace v0.x.y like to v0.9.5. If you are trying to contribute on OpenPAI, you can remove -b parameter and clone to default branch.
git clone -b v0.x.y https://github.com/Microsoft/pai.git

# Go into the workdir.
cd pai/src/dev-box/build

# Build your dev-box.
sudo docker build -t dev-box . --file=dev-box.dockerfile

Start your dev-box container

  • Suppose the directory path of your cluster-configuration is /pathConfiguration. Note: Don't change the configuration file name！
# Run your dev-box
sudo docker run -itd \
        -e COLUMNS=$COLUMNS -e LINES=$LINES -e TERM=$TERM \
        -v /var/run/docker.sock:/var/run/docker.sock \
        -v /pathConfiguration:/cluster-configuration  \
        -v /hadoop-binary:/hadoop-binary  \
        --pid=host \
        --privileged=true \
        --net=host \
        --name=dev-box \
        dev-box

# Working in your dev-box
sudo docker exec -it dev-box /bin/bash
cd /pai

# Now you are free to configure your cluster and run PAI commands...

Deploy dev-box over existing K8s

If you want to deploy dev-box in already deployed kubernetes.

Prerequisites: The user has installed kubectl on the current machine.

(1) Create a label for the server to be deployed:

kubectl label --overwrite=true nodes $NODE-IP-ADDRESS dev-box=true

(2) Deploy dev-box to kubernetes

cd pai/src/dev-box

kubectl create -f dev-box-k8s-deploy.yaml