Docker Improvements

Docker/README.md

@@ -1,200 +1,144 @@
-# Run in docker
+# Docker Setup
 
-Simple and fast setup of EOS.IO on Docker is also available.
+This guide explains how to get up and running with Docker.
+Docker Hub image available from [docker hub](https://hub.docker.com/r/eosio/eos/).
 
 ## Install Dependencies
 
 - [Docker](https://docs.docker.com) Docker 17.05 or higher is required
 - [docker-compose](https://docs.docker.com/compose/) version >= 1.10.0
 
-## Docker Requirement
+## Requirements
 
-- At least 7GB RAM (Docker -> Preferences -> Advanced -> Memory -> 7GB or above)
-- If the build below fails, make sure you've adjusted Docker Memory settings and try again.
+- At least 7GB RAM (Docker -> Preferences -> Advanced -> Memory -> 7GB or above). If the build below fails, make sure you've adjusted Docker Memory settings and try again.
+- Ability to edit a text file and execute a bash script
 
-## Build eos image
+## Building the EOS Containers:
 
-```bash
-git clone https://github.com/EOSIO/eos.git --recursive  --depth 1
-cd eos/Docker
-docker build . -t eosio/eos
-```
+There are currently three Docker template directories you can use: `latest`, `testnet`, and `dev`. Inside of `latest`, you'll see a setup.sh, docker-compose.yml, Dockerfile, and a hidden .env file. 
 
-The above will build off the most recent commit to the master branch by default. If you would like to target a specific branch/tag, you may use a build argument. For example, if you wished to generate a docker image based off of the v1.2.2 tag, you could do the following:
+If you look at the .env file, you'll see (which are also the defaults used in the Dockerfile for manual building):
 
 ```bash
-docker build -t eosio/eos:v1.2.2 --build-arg branch=v1.2.2 .
+$ cat eos/Docker/latest/.env
+# GITHUB
+GITHUB_BRANCH=master
+GITHUB_ORG=EOSIO # You can find this on the github url: https://github.com/EOSIO
+# COMPOSE / BUILD
+DOCKERHUB_USERNAME=eosio
+IMAGE_NAME=eos
+IMAGE_TAG=latest
+BUILD_SYMBOL=SYS
+NODEOS_VOLUME_NAME=nodeos-data-volume
+NODEOS_PORT=9876
+NODEOS_API_PORT=8888
+KEOSD_VOLUME_NAME=keosd-data-volume
+KEOSD_API_PORT=8900
+COMPOSE_PROJECT_NAME=main
 ```
 
-By default, the symbol in eosio.system is set to SYS. You can override this using the symbol argument while building the docker image.
-
-```bash
-docker build -t eosio/eos --build-arg symbol=<symbol> .
-```
+These are variables used throughout the scripts, allowing you to simply copy the directory and modify the .env file if you wish to setup a second instance (like running on a public testnet). If you modify these values, like the BUILD_SYMBOL=SYS to =EOS, running `./setup.sh` will set your containers up with that as the "CORE symbol name". You can see that the dev directory has a slightly different .env file if you need more examples.
 
-## Start nodeos docker container only
+If you don't wish to use the setup.sh script, you can manually run docker build. Defaults are included in the Dockerfile, but you can override them with `--build-arg`. 
 
-```bash
-docker run --name nodeos -p 8888:8888 -p 9876:9876 -t eosio/eos nodeosd.sh -e --http-alias=nodeos:8888 --http-alias=127.0.0.1:8888 --http-alias=localhost:8888 arg1 arg2
-```
+*The .env is not used when manually building!*
 
-By default, all data is persisted in a docker volume. It can be deleted if the data is outdated or corrupted:
+Here is an example:
 
 ```bash
-$ docker inspect --format '{{ range .Mounts }}{{ .Name }} {{ end }}' nodeos
-fdc265730a4f697346fa8b078c176e315b959e79365fc9cbd11f090ea0cb5cbc
-$ docker volume rm fdc265730a4f697346fa8b078c176e315b959e79365fc9cbd11f090ea0cb5cbc
+git clone https://github.com/EOSIO/eos.git --recursive
+cd eos/Docker/latest
+docker build -t eosio/eos:v1.1.0 --build-arg COMPOSE_PROJECT_NAME=testnet --build-arg BUILD_SYMBOL=LEOS --build-arg GITHUB_BRANCH=master --build-arg GITHUB_ORG=EOSIO --no-cache .
 ```
 
-Alternately, you can directly mount host directory into the container
+To set up and start the containers, you can run:
 
 ```bash
-docker run --name nodeos -v /path-to-data-dir:/opt/eosio/bin/data-dir -p 8888:8888 -p 9876:9876 -t eosio/eos nodeosd.sh -e --http-alias=nodeos:8888 --http-alias=127.0.0.1:8888 --http-alias=localhost:8888 arg1 arg2
+cd eos/Docker/latest
+./setup.sh
 ```
 
-## Get chain info
+- After running the setup script, two containers named `main_nodeos` and `main_keosd` (if you have `COMPOSE_PROJECT_NAME=main`) will be started in the background (`up -d` = create & detach).
+-- The nodeos container/service will expose ports 8888 and 9876.
+-- The keosd container/service does not expose any port to the host. It does however allow you to alias and run cleos from the local machine: `alias cleos='docker exec main_keosd /opt/eosio/bin/cleos -u http://main_nodeos:8888 --wallet-url http://127.0.0.1:8900'
 
-```bash
-curl http://127.0.0.1:8888/v1/chain/get_info
-```
 
-## Start both nodeos and keosd containers
+### Verify your installation
 
 ```bash
-docker volume create --name=nodeos-data-volume
-docker volume create --name=keosd-data-volume
-docker-compose up -d
+curl http://127.0.0.1:8888/v1/chain/get_info
 ```
 
-After `docker-compose up -d`, two services named `nodeosd` and `keosd` will be started. nodeos service would expose ports 8888 and 9876 to the host. keosd service does not expose any port to the host, it is only accessible to cleos when running cleos is running inside the keosd container as described in "Execute cleos commands" section.
-
-### Execute cleos commands
-
-You can run the `cleos` commands via a bash alias.
+Alternatively, you can run:
 
 ```bash
-alias cleos='docker-compose exec keosd /opt/eosio/bin/cleos -u http://nodeosd:8888 --wallet-url http://localhost:8900'
 cleos get info
-cleos get account inita
+cleos get account eosio
 ```
 
-Upload sample exchange contract
+## (Alternative method) Start nodeos docker container only:
 
 ```bash
-cleos set contract exchange contracts/exchange/
+docker run --name nodeosd -p 8888:8888 -p 9876:9876 -t eosio/eos:latest nodeosd.sh -e --http-alias=nodeos:8888 --http-alias=127.0.0.1:8888 --http-alias=localhost:8888
 ```
 
-If you don't need keosd afterwards, you can stop the keosd service using
+You can also directly mount a host directory into the container:
 
 ```bash
-docker-compose stop keosd
+docker run --name nodeosd -v /path-to-data-dir:/opt/eosio/data-dir -p 8888:8888 -p 9876:9876 -t eosio/eos nodeosd.sh -e --http-alias=nodeos:8888 --http-alias=127.0.0.1:8888 --http-alias=localhost:8888
 ```
 
-### Develop/Build custom contracts
+- Sometimes the nodeos container does not start after the compose or build process. You can check with `docker container ls -a` and manually start it.
+
+## Develop/Build custom contracts:
 
 Due to the fact that the eosio/eos image does not contain the required dependencies for contract development (this is by design, to keep the image size small), you will need to utilize the eosio/eos-dev image. This image contains both the required binaries and dependencies to build contracts using eosiocpp.
 
-You can either use the image available on [Docker Hub](https://hub.docker.com/r/eosio/eos-dev/) or navigate into the dev folder and build the image manually.
+You can either use the image available on [Docker Hub](https://hub.docker.com/r/eosio/eos-dev/) or navigate into the dev folder and build the image manually or with the setup script.
 
 ```bash
 cd dev
-docker build -t eosio/eos-dev .
+./dev_setup.sh
 ```
 
-### Change default configuration
+## Change default configuration
 
-You can use docker compose override file to change the default configurations. For example, create an alternate config file `config2.ini` and a `docker-compose.override.yml` with the following content.
+You can modify the docker-compose.yml file to change the default configurations. For example, add a new local file: `config2.ini`, and point it to the docker container's data-dir:
 
 ```yaml
-version: "2"
+version: "3"
 
 services:
   nodeos:
+. . .
+    expose:
+      - "8888"
     volumes:
-      - nodeos-data-volume:/opt/eosio/bin/data-dir
-      - ./config2.ini:/opt/eosio/bin/data-dir/config.ini
+      - ~/$NODEOS_VOLUME_NAME:/opt/eosio/bin/data-dir
+      - ~/some_dir/config2.ini:/opt/eosio/data-dir/config.ini
 ```
 
-Then restart your docker containers as follows:
+Now, when you modify the ~/some_dir/config2.ini locally, the changes will be included in the docker container, overwriting the config that was used before. This requires a reboot of the containers:
 
 ```bash
+cd eos/Docker/latest
 docker-compose down
-docker-compose up
-```
-
-### Clear data-dir
-
-The data volume created by docker-compose can be deleted as follows:
-
-```bash
-docker volume rm nodeos-data-volume
-docker volume rm keosd-data-volume
-```
-
-### Docker Hub
-
-Docker Hub image available from [docker hub](https://hub.docker.com/r/eosio/eos/).
-Create a new `docker-compose.yaml` file with the content below
-
-```bash
-version: "3"
-
-services:
-  nodeosd:
-    image: eosio/eos:latest
-    command: /opt/eosio/bin/nodeosd.sh --data-dir /opt/eosio/bin/data-dir -e --http-alias=nodeosd:8888 --http-alias=127.0.0.1:8888 --http-alias=localhost:8888
-    hostname: nodeosd
-    ports:
-      - 8888:8888
-      - 9876:9876
-    expose:
-      - "8888"
-    volumes:
-      - nodeos-data-volume:/opt/eosio/bin/data-dir
-
-  keosd:
-    image: eosio/eos:latest
-    command: /opt/eosio/bin/keosd --wallet-dir /opt/eosio/bin/data-dir --http-server-address=127.0.0.1:8900 --http-alias=localhost:8900 --http-alias=keosd:8900
-    hostname: keosd
-    links:
-      - nodeosd
-    volumes:
-      - keosd-data-volume:/opt/eosio/bin/data-dir
-
-volumes:
-  nodeos-data-volume:
-  keosd-data-volume:
-
+docker-compose up -d
 ```
 
-*NOTE:* the default version is the latest, you can change it to what you want
-
-run `docker pull eosio/eos:latest`
+Remember that docker-compose requires a docker-compose.yml in the same directory. You can use `docker-compose -f docker-compose-new.yml` if you want to specify a different file.
 
-run `docker-compose up`
+## Docker Environment Reset / Cleanup
 
-### EOSIO Testnet
+By default, all data is persisted in a docker volume under ~/. There is a script available for an entire reset of your docker environment: `./docker_reset.sh`. It will prompt you before executing just to be safe.
 
-We can easily set up a EOSIO local testnet using docker images. Just run the following commands:
-
-Note: if you want to use the mongo db plugin, you have to enable it in your `data-dir/config.ini` first.
-
-```
-# create volume
-docker volume create --name=nodeos-data-volume
-docker volume create --name=keosd-data-volume
-# pull images and start containers
-docker-compose -f docker-compose-eosio-latest.yaml up -d
-# get chain info
-curl http://127.0.0.1:8888/v1/chain/get_info
-# get logs
-docker-compose logs -f nodeosd
-# stop containers
-docker-compose -f docker-compose-eosio-latest.yaml down
-```
+## EOSIO Testnet:
 
-The `blocks` data are stored under `--data-dir` by default, and the wallet files are stored under `--wallet-dir` by default, of course you can change these as you want.
+We can easily set up a EOSIO local testnet by making a copy of the `latest` directory. Once copied, modify the .env values to match the version and use a different COMPOSE_PROJECT_NAME so it doesn't conflict with other containers. You can setup a second alias for the new container like so:
 
-### About MongoDB Plugin
+`alias test_cleos='docker exec test_keosd /opt/eosio/bin/cleos -u http://test_nodeos:8888 --wallet-url http://127.0.0.1:8900'`
 
-Currently, the mongodb plugin is disabled in `config.ini` by default, you have to change it manually in `config.ini` or you can mount a `config.ini` file to `/opt/eosio/bin/data-dir/config.ini` in the docker-compose file.
+## Miscellaneous / Notes
+- The blocks/state data is stored under --data-dir by default. The wallet files are stored under --wallet-dir by default. These can be changed in you docker-compose.yml
+- Take regular backups of your volume directories
+- If using a public blockchain, you need to wait for the entire blockchain to catch up/sync to the latest blocks before you can perform actions through your keosd/nodeos. Alternatively, you can run keosd on its own and set the cleos `-u` to a public api of an up to date/synced producer

Docker/config.ini

@@ -143,7 +143,7 @@ producer-name = eosio
 #    KEY:<data>      	is a string form of a valid EOSIO private key which maps to the provided public key
 #
 #    KEOSD:<data>    	is the URL where keosd is available and the approptiate wallet(s) are unlocked (eosio::producer_plugin)
-signature-provider = EOS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV=KEY:5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3
+#signature-provider = EOS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV=KEY:5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3
 
 # Limits the maximum time (in milliseconds) that is allowd for sending blocks to a keosd provider for signing (eosio::producer_plugin)
 keosd-provider-timeout = 5

Docker/dev/.env

@@ -0,0 +1,14 @@
+# GITHUB
+GITHUB_BRANCH=master
+GITHUB_ORG=EOSIO # You can find this on the github url: https://github.com/EOSIO
+# COMPOSE / BUILD
+DOCKERHUB_USERNAME=eosio
+IMAGE_NAME=eos-dev
+IMAGE_TAG=latest
+BUILD_SYMBOL=DEVOS
+NODEOS_VOLUME_NAME=nodeos-dev-data-volume
+NODEOS_PORT=9876
+NODEOS_API_PORT=8888
+KEOSD_VOLUME_NAME=keosd-dev-data-volume
+KEOSD_API_PORT=8900
+COMPOSE_PROJECT_NAME=dev

Docker/dev/Dockerfile

@@ -1,11 +1,9 @@
 FROM eosio/builder
-ARG branch=master
-ARG symbol=SYS
 
-RUN git clone -b $branch https://github.com/EOSIO/eos.git --recursive \
-    && cd eos && echo "$branch:$(git rev-parse HEAD)" > /etc/eosio-version \
+RUN git clone -b $GITHUB_BRANCH https://github.com/$GITHUB_ORG/eos.git --recursive \
+    && cd eos && echo "$GITHUB_BRANCH:$(git rev-parse HEAD)" > /etc/eosio-version \
     && cmake -H. -B"/opt/eosio" -GNinja -DCMAKE_BUILD_TYPE=Release -DWASM_ROOT=/opt/wasm -DCMAKE_CXX_COMPILER=clang++ \
-       -DCMAKE_C_COMPILER=clang -DCMAKE_INSTALL_PREFIX=/opt/eosio  -DSecp256k1_ROOT_DIR=/usr/local -DBUILD_MONGO_DB_PLUGIN=true -DCORE_SYMBOL_NAME=$symbol \
+       -DCMAKE_C_COMPILER=clang -DCMAKE_INSTALL_PREFIX=/opt/eosio  -DSecp256k1_ROOT_DIR=/usr/local -DBUILD_MONGO_DB_PLUGIN=true -DCORE_SYMBOL_NAME=$BUILD_SYMBOL \
     && cmake --build /opt/eosio --target install \
     && cp /eos/Docker/config.ini / && ln -s /opt/eosio/contracts /contracts && cp /eos/Docker/nodeosd.sh /opt/eosio/bin/nodeosd.sh && ln -s /eos/tutorials /tutorials
 

Docker/dev/docker-compose.yml

@@ -0,0 +1,39 @@
+version: "3"
+
+services:
+  builder:
+    container_name: "${COMPOSE_PROJECT_NAME}_builder"
+    build:
+      context: ../builder
+    image: eosio/builder
+
+  nodeos:
+    env_file:
+      .env
+    container_name: "${COMPOSE_PROJECT_NAME}_nodeos"
+    image: "${DOCKERHUB_USERNAME}/${IMAGE_NAME}:${IMAGE_TAG}"
+    command: "/opt/eosio/bin/nodeosd.sh --data-dir /opt/eosio/bin/data-dir -e --http-alias=${COMPOSE_PROJECT_NAME}_nodeos:8888 --http-alias=127.0.0.1:${NODEOS_API_PORT} --http-alias=localhost:${NODEOS_API_PORT}"
+    hostname: "${COMPOSE_PROJECT_NAME}_nodeos"
+    ports:
+      - $NODEOS_API_PORT:$NODEOS_API_PORT
+      - $NODEOS_PORT:$NODEOS_PORT
+    expose:
+      - "${NODEOS_API_PORT}"
+    volumes:
+      - ~/$NODEOS_VOLUME_NAME:/opt/eosio/bin/data-dir
+    cap_add:
+      - IPC_LOCK
+    stop_grace_period: 10m
+
+  keosd:
+    env_file:
+      .env
+    container_name: "${COMPOSE_PROJECT_NAME}_keosd"
+    image: "${DOCKERHUB_USERNAME}/${IMAGE_NAME}:${IMAGE_TAG}"
+    command: "/opt/eosio/bin/keosd --wallet-dir /opt/eosio/bin/data-dir --http-server-address=127.0.0.1:${KEOSD_API_PORT} --http-alias=${COMPOSE_PROJECT_NAME}_keosd:${KEOSD_API_PORT} --http-alias=localhost:${KEOSD_API_PORT}"
+    hostname: "${COMPOSE_PROJECT_NAME}_keosd"
+    links:
+      - nodeos
+    volumes:
+      - ~/$KEOSD_VOLUME_NAME:/opt/eosio/bin/data-dir
+    stop_grace_period: 10m

Docker/dev/setup.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+source .env
+echo "[Docker Container Creation Started][${COMPOSE_PROJECT_NAME}]"
+DATAVOLUME="${HOME}/${NODEOS_VOLUME_NAME}"
+# Create data volume if it doesn't exist
+if [ ! -d "${DATAVOLUME}" ]; then
+	mkdir $DATAVOLUME
+	echo "[Created ${DATAVOLUME}]"
+fi
+# If genesis.json is missing, don't allow this to run (adding it later breaks the chain)
+if [ ! -e "${DATAVOLUME}/genesis.json" ]; then
+        printf "Please ensure that you place your genesis.json file into ${DATAVOLUME} before running this script!\n\n"
+        exit
+fi
+docker volume create --name=$NODEOS_VOLUME_NAME 1> /dev/null
+docker volume create --name=$KEOSD_VOLUME_NAME 1> /dev/null
+if [ ! -h "${HOME}/${KEOSD_VOLUME_NAME}" ]; then ln -s $DATAVOLUME $HOME/$KEOSD_VOLUME_NAME; fi
+docker-compose up -d
+# Ensure nodeos is started
+docker start "${COMPOSE_PROJECT_NAME}_nodeos" 1> /dev/null
+echo "[Docker Container Creation Complete][${COMPOSE_PROJECT_NAME}]"