Add docker image #375

chevdor · 2018-07-19T06:44:05Z

Status

Pretty much ready beside the fact that the automatic build on docker hub may timeout due to the time required to make the image.

Usage

see readme to build your own image. Beware, that takes a while...
or docker run -d -p 30333:30333 -p 9933:9933 chevdor/polkadot:latest polkadot

Todos

set version based on Cargo.toml
rebuild as --release
install polkadot so we don´t need to pass the full path
set / document volumes
set / document ports
reduce image size (no one wants to download 8GB images)

pepyakin · 2018-07-19T10:06:14Z

docker/Dockerfile

+FROM phusion/baseimage:0.10.1
+LABEL maintainer "[email protected]"
+
+RUN apt-get update && \


I'm not into docker stuff, but maybe it would be better to split this into separate steps? I found it more useful since if you change some later steps, previous steps could be reused.

That makes it more readable and I would do that also in a regular script but for docker this is different. Having a 'single' command packs all in a single layer.

That could be split however but I have very bad internet atm and testing the build of docker images is a pain :)

If you split apt-get update from upgrade and install, then docker will cache the update step and then when you try to build the rest of the image it may fail because the outcome of the update is outdated.

Yep, very good point, that´s what I did.

pepyakin · 2018-07-19T10:07:10Z

docker/build.sh

@@ -0,0 +1,3 @@
+#!/bin/sh


Better to use #!/usr/bin/env bash shebang since we already use it in other scripts.

ok no problem, I will change that.

ltfschoen · 2018-07-19T11:18:23Z

very pleased to see this

ltfschoen · 2018-07-19T21:38:08Z

docker/Dockerfile

+RUN apt-get update && \
+	apt-get upgrade -y && \
+	apt-get install -y cmake pkg-config libssl-dev && \
+	curl https://sh.rustup.rs -sSf | sh && \


i get the following error when it gets up to running the curl command using Docker Community Edition Version 18.03.1-ce-mac65 (24312) on macOS 10.13.4:

info: downloading installer rustup: Unable to run interactively. Run with -y to accept defaults, --help for additional options The command '/bin/sh -c curl https://sh.rustup.rs -sSf | sh' returned a non-zero code: 1

i fixed the error by changing it to curl https://sh.rustup.rs -sSf | sh -s -- -y (i.e. appending -s -- -y), as recommended in this thread rust-lang/rustup#1031

chevdor · 2018-07-19T21:51:01Z

Yes. This is still a WIP but I fixed the issue you mentioned already with a few others. I still have to do some testing and doc and I will push all. Lean back and enjoy it tomorrow when it all works ;)

ltfschoen · 2018-07-19T22:09:06Z

docker/Dockerfile

+	apt-get upgrade -y && \
+	apt-get install -y cmake pkg-config libssl-dev && \
+	curl https://sh.rustup.rs -sSf | sh && \
+	rustup update nightly && \


i was getting error the following error when trying to run rustup as it wasn't added to the PATH.

i fixed it by adding the following lines of code after the curl command:

RUN echo 'PATH="$/root/.cargo/bin:$PATH";' >> ~/.bash_profile && . ~/.bash_profile && . /root/.cargo/env RUN which rustup && which cargo

ltfschoen · 2018-07-19T22:20:40Z

docker/Dockerfile

+	rustup target add wasm32-unknown-unknown --toolchain nightly && \
+	rustup update stable && \
+	cargo install --git https://github.com/alexcrichton/wasm-gc && \
+	git clone https://github.com/paritytech/polkadot.git && \


i get error /bin/sh: 1: git: not found because you need to install git before you can use it, by adding the following beforehand:

apt-get install -y git && \

…ccept defaults

chevdor · 2018-07-20T06:40:01Z

You are going through all the issues I had and fixed already ;)

Add doc Ref paritytech#375

Add documentation Ref paritytech#375

Fix paritytech#375

chevdor · 2018-07-20T16:19:46Z

So, as I feared, the build on docker hub takes more than 2 hours and times out.
I removed the WIP since the current solution is working and usable, users can either:

build their own image, depending on the perf of their system and docker settings it may take ~30 minutes
use a pre built image

The good news with the last option is that the current (compressed) size of the image is around 300MB. We can probably skim a few MB here and there but the time to do so is probably not worth it.

I will investigate an option to get the automatic docker builds to pass. Maybe someone has an idea/hint beside splitting the image? Most of the time is spent on building rust components, so I put little hope in making a base rust image and if it helps, it may not work very long as polkadot´s complexity increases.

gavofyork

Generally prefer the docker instructions to go later since they're not the first-principles but rather more like a binary installer.

gavofyork · 2018-07-21T20:55:36Z

README.md

+To get up and running with the smallest footprint on your system, you may use the Polkadot Docker image.
+You can either build it yourself (it takes a while...):
+
+


some information missing here?

Yes, hard to split the docker part.
I agree the building part should probably come later.

I moved the whole docker to the end in my last commit, including the (indeed) missing part.

The docker part describing how to 'just run' polkadot may be however interesting for many as it allows starting a node with the least amount of effort and dependencies beside a valid docker install. This is also much much quicker than to build the whole project.

ltfschoen · 2018-07-21T21:34:02Z

@chevdor I tried implementing Docker on Polkadot just after your initial commit. I approached it the way I usually do on projects. I've pushed my proposed approach to the following link. I have poor internet this weekend, but I hope you can take a look at what I've done and choose anything you find useful into your PR https://github.com/paritytech/polkadot/compare/master...ltfschoen:docker?expand=1

chevdor · 2018-07-21T22:07:12Z

@ltfschoen cool thanks. I am pushing a commit to add the missing info as @gavofyork mentioned and I will check your link tomorrow to see if I missed anything.

chevdor · 2018-07-22T16:58:40Z

@ltfschoen here is a little update. I am still working on improving the dockerfile, mainly to improve build speed and final size.

Here is some feedback regarding diff with your branch:

https://github.com/paritytech/polkadot/compare/master...ltfschoen:docker?expand=1#diff-ebacf6f6ae4ee68078bb16454b23247dR6 you make a strong assumption that there will be a tag named vX.Y when cargo contains version = X.Y. That sounds risky and my initial goal is to build the latest version so we can include that in the CI.
https://github.com/paritytech/polkadot/compare/master...ltfschoen:docker?expand=1#diff-a489144702a723431811d56a3c44690aR37 lots of docker cool tips. While tempting to add, I think we should not add too much docker documentation but refer the users to the docker doc. This way, if docker changes details, we don´t need to update.
https://github.com/paritytech/polkadot/compare/master...ltfschoen:docker?expand=1#diff-c48821b747da2e9250236a61b11de958R4 Uhhh I don´t like this one at all, this is assuming the user has only polkadot images on his machine and kicking all the others containers and images in his back. This is not the first time I see that and I find it very uncool :)
https://github.com/paritytech/polkadot/compare/master...ltfschoen:docker?expand=1#diff-072e81fe88d9b23fe00fc189279936a6R1 I savagely steal this one :). Not sure a single contain needs to have a docker compose but a working template as starting point is always nice to have. Cool!
https://github.com/paritytech/polkadot/compare/master...ltfschoen:docker?expand=1#diff-47a46fcdbeeb5726affa190048bd984fR1 Why this 52 LOC script when you can do it in one line?

I added back a WIP on this issue. I will remove it once done.

chevdor · 2018-07-22T17:57:17Z

I did some experiment trying to increase the number of layers to speed up building the image.
This unfortunately has a very negative impact on the size of the image (but makes indeed the build much faster since we can skip the apt updates, rust updates, etc...).

By splitting into more layers, the image went from 337MB to more than 3.3GB 🤢. The reason is mainly that /root/.cargo is huge and I cannot delete it between the layers since it is usually required by the next layer.

I also considered using the official rust slim image but the base image is already bigger than the best image I get today.

The current 2 layers options seem to be the best so far but I still have an idea I want to test.

Fix paritytech#375

Fix doc and reduce image size Fix paritytech#375

…docker

chevdor · 2018-07-26T09:22:25Z

@gavofyork I moved the docker part further down.

* master: (86 commits) Make contract a separate runtime module (#345) Version bump (#450) DB-based blockchain data cache for light nodes (#251) Update libp2p again (#445) Update version on git head change (#444) Fix the public key of bootnode 3 (#441) Update libp2p (#442) Switch to the master branch of libp2p (#427) Export ws port 9944 and add doc (#440) Iterate over overlay to decide which keys to purge (#436) Exit signal gets its own trait (#433) Add docker image (#375) Reset peers.json if the content is not loadable (#405) Limit number of incoming connections (#391) Fix memory leaks in libp2p (#432) Do not queue empty blocks set for import (#431) 5 random fixes (#1) (#435) Chore: fix typo (#434) Prevent building invalid blocks (#430) Better logging for public key mismatch (#429) ...

* Simplify the logger * Fix test * Docs * . * . * . * No line in target

Check bounds in a few places in archiver and document some panics

* WIP update dependencies to crates versions * Use released versions of sp-version and sp-arithmetic * Attempt to align version of sp-keyring used to correspond to 4.0.0 releases. No luck but closer * sp-keyring 4.0.0 * also sp-keyring 4.0.0 for macro crate * simplify cargo.tomls Co-authored-by: James Wilson <[email protected]>

pepyakin reviewed Jul 19, 2018

View reviewed changes

ltfschoen reviewed Jul 19, 2018

View reviewed changes

ltfschoen added a commit that referenced this pull request Jul 20, 2018

fix: Fix #375 errors by installing git, rustup in PATH and flags to a…

4162fd1

…ccept defaults

chevdor added a commit to chevdor/substrate that referenced this pull request Jul 20, 2018

Fix docker image building

ad0793b

Add doc Ref paritytech#375

chevdor changed the title ~~WIP: Add Dockerfile~~ WIP: Add docker image Jul 20, 2018

Add Dockerfile

9e3ce74

Add documentation Ref paritytech#375

chevdor force-pushed the docker branch from 70ecceb to c8b03d4 Compare July 20, 2018 13:28

chevdor added a commit to chevdor/substrate that referenced this pull request Jul 20, 2018

Fix doc and reduce image size

c8b03d4

Fix paritytech#375

chevdor changed the title ~~WIP: Add docker image~~ Add docker image Jul 20, 2018

gavofyork added the A0-please_review Pull request needs code review. label Jul 21, 2018

gavofyork suggested changes Jul 21, 2018

View reviewed changes

chevdor changed the title ~~Add docker image~~ WIP: Add docker image Jul 22, 2018

chevdor force-pushed the docker branch from 6ee3647 to 3b9113a Compare July 23, 2018 07:36

chevdor added a commit to chevdor/substrate that referenced this pull request Jul 23, 2018

Fix doc and reduce image size

3b9113a

Fix paritytech#375

chevdor added 2 commits July 23, 2018 11:16

Add PORT, VOLUME and reduce size of the docker image significantly

26605da

Fix doc and reduce image size Fix paritytech#375

Fix docker compose

e91b1a7

chevdor force-pushed the docker branch from 3b9113a to e91b1a7 Compare July 23, 2018 09:16

Merge branch 'master' into docker

7fdfa6f

chevdor changed the title ~~WIP: Add docker image~~ Add docker image Jul 23, 2018

Merge branch 'master' of https://github.com/paritytech/polkadot into …

d93d2fa

…docker

gavofyork approved these changes Jul 27, 2018

View reviewed changes

Merge branch 'master' into docker

97a418a

gavofyork merged commit 8f414ab into paritytech:master Jul 27, 2018

lamafab pushed a commit to lamafab/substrate that referenced this pull request Jun 16, 2020

Add provides tag to claims transactions. (paritytech#375)

37d32fe

liuchengxu added a commit to chainx-org/substrate that referenced this pull request Aug 23, 2021

Simplify the logger (paritytech#375)

3d81b56

* Simplify the logger * Fix test * Docs * . * . * . * No line in target

liuchengxu pushed a commit to autonomys/substrate that referenced this pull request Jun 3, 2022

Merge pull request paritytech#375 from subspace/check-bounds-in-archiver

f0496fd

Check bounds in a few places in archiver and document some panics

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add docker image #375

Add docker image #375

chevdor commented Jul 19, 2018 •

edited

Loading

pepyakin Jul 19, 2018

chevdor Jul 19, 2018

tomaka Jul 20, 2018

chevdor Jul 20, 2018

pepyakin Jul 19, 2018

chevdor Jul 19, 2018

ltfschoen commented Jul 19, 2018

ltfschoen Jul 19, 2018 •

edited

Loading

ltfschoen Jul 19, 2018

chevdor commented Jul 19, 2018 via email

ltfschoen Jul 19, 2018 •

edited

Loading

ltfschoen Jul 19, 2018

chevdor commented Jul 20, 2018 via email •

edited

Loading

chevdor commented Jul 20, 2018 •

edited

Loading

gavofyork left a comment

gavofyork Jul 21, 2018

chevdor Jul 21, 2018

ltfschoen commented Jul 21, 2018 •

edited

Loading

chevdor commented Jul 21, 2018

chevdor commented Jul 22, 2018

chevdor commented Jul 22, 2018

chevdor commented Jul 26, 2018

		To get up and running with the smallest footprint on your system, you may use the Polkadot Docker image.
		You can either build it yourself (it takes a while...):

Add docker image #375

Add docker image #375

Conversation

chevdor commented Jul 19, 2018 • edited Loading

Status

Usage

Todos

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ltfschoen commented Jul 19, 2018

ltfschoen Jul 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chevdor commented Jul 19, 2018 via email

ltfschoen Jul 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chevdor commented Jul 20, 2018 via email • edited Loading

chevdor commented Jul 20, 2018 • edited Loading

gavofyork left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ltfschoen commented Jul 21, 2018 • edited Loading

chevdor commented Jul 21, 2018

chevdor commented Jul 22, 2018

chevdor commented Jul 22, 2018

chevdor commented Jul 26, 2018

chevdor commented Jul 19, 2018 •

edited

Loading

ltfschoen Jul 19, 2018 •

edited

Loading

ltfschoen Jul 19, 2018 •

edited

Loading

chevdor commented Jul 20, 2018 via email •

edited

Loading

chevdor commented Jul 20, 2018 •

edited

Loading

ltfschoen commented Jul 21, 2018 •

edited

Loading