Enable static builds of binaries

[bobsummerwill]

From @rainbeam on April 30, 2016 22:41

Following on from ethereum/webthree-helpers#157, enable static linking of some more binaries by switching add_executable to eth_simple_add_executable.

This is a master issue. Target binaries:

• eth
  • wrap add_exe link eth binary with custom add_executable webthree#163
• solc Use add exe wrapper for solc, soltest, lllc solidity#528
• soltest ^
• lllc ^
  • fix macro arguments Fix macro arguments. webthree-helpers#164
• ethminer
  • wrap add_exe Use add_exe wrapper for ethminer libethereum#248
  • macro changed. dynamic working. static still has -lcurl ordering issue
• Fix accidental REQUIRED introduced onto curl for dynamic builds (Only looking for ssh2, openssl and lib when static linkage is enabled, to avoid spammy messages webthree-helpers#165)
• Usage of STATIC_LINKING across platforms
  • Test static linkage on Alpine Linux
  • Test static linkage on Debian (working, you said?) (no, curl link problems)
  • Test static linkage on Ubuntu
  • Test static linkage on OS X (partial static-linkage working).
    • Suppress some linker warnings
  • Test static linkage on Windows

Copied from original issue: ethereum/webthree-umbrella#495

[bobsummerwill]

From @rainbeam on April 30, 2016 23:54

Just running a local build now to sanity check the above build with static linking.

[bobsummerwill]

From @rainbeam on May 1, 2016 0:20

Everything works except ethminer which fails on linking with a curl problem:

[100%] Linking CXX executable ethminer
/src/built/lib/libjsonrpccpp-client.a(httpclient.cpp.o): In function `jsonrpc::HttpClient::HttpClient(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
httpclient.cpp:(.text+0x14d): undefined reference to `curl_easy_init'
/src/built/lib/libjsonrpccpp-client.a(httpclient.cpp.o): In function `jsonrpc::HttpClient::~HttpClient()':
httpclient.cpp:(.text+0x1e4): undefined reference to `curl_easy_cleanup'
/src/built/lib/libjsonrpccpp-client.a(httpclient.cpp.o): In function `jsonrpc::HttpClient::SendRPCMessage(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)':
httpclient.cpp:(.text+0x2b5): undefined reference to `curl_easy_setopt'
...

[bobsummerwill]

Link order issue? A la http://stackoverflow.com/questions/28165518/use-libcurl-undefined-reference-to-curl-easy-init?

[bobsummerwill]

From @rainbeam on May 1, 2016 0:32

That's puzzling. Not sure why it isn't an issue when building eth, which also uses jsonrpccpp.

[bobsummerwill]

Well, if it just a command-line order issue, it would all just be down to the order in which targets were executed. So maybe just some fluky alphabetical thing, where -lcurl is ending up early in this case?

[bobsummerwill]

From @rainbeam on May 1, 2016 0:49

This is the actual link command (formatted):

/src/webthree-umbrella/build # cat libethereum/ethminer/CMakeFiles/ethminer.dir/link.txt 

/usr/bin/c++   -std=c++11 -Wall -Wno-unknown-pragmas -Wextra -Werror -pedantic -DSHAREDLIB -fPIC
-std=c++11 -Wall -Wno-unknown-pragmas -Wextra -Werror -pedantic -DSHAREDLIB -fPIC
-Wno-error -fstack-protector-strong -Wstack-protector -O3 -DNDEBUG -DETH_RELEASE  -static

CMakeFiles/ethminer.dir/main.cpp.o  -o ethminer -rdynamic 
/src/built/lib/libjsoncpp.a 

-lssh2 -lssl -lcrypto -lz -lcurl                     <------------- curl link
/src/built/lib/libjsonrpccpp-common.a 
/src/built/lib/libjsonrpccpp-client.a

/src/built/lib/libjsoncpp.a
/src/built/lib/libleveldb.a 
-lboost_thread-mt -lboost_random -lboost_filesystem -lboost_system -lpthread 
../../libweb3core/libdevcore/libdevcore.a 
../../webthree-helpers/utils/libscrypt/libscrypt.a /src/built/lib/libcryptlib.a 
../../webthree-helpers/utils/secp256k1/libsecp256k1.a

../../libweb3core/libdevcrypto/libdevcrypto.a 
../libethcore/libethcore.a 
../libevmcore/libevmcore.a
../libevm/libevm.a 
/src/built/lib/libminiupnpc.a 
../../libweb3core/libp2p/libp2p.a 
-lboost_regex

../libethereum/libethereum.a 
/src/built/lib/libcryptlib.a 
../libethash/libethash.a
../libethashseal/libethashseal.a 
../libethereum/libethereum.a 
../libevm/libevm.a 
../libethcore/libethcore.a
../libevmcore/libevmcore.a 
../../libweb3core/libp2p/libp2p.a 
../../libweb3core/libdevcrypto/libdevcrypto.a
../../libweb3core/libdevcore/libdevcore.a 
/src/built/lib/libjsoncpp.a 
/src/built/lib/libleveldb.a 
-lboost_thread-mt -lboost_random -lboost_filesystem -lboost_system -lpthread
../../webthree-helpers/utils/libscrypt/libscrypt.a 
../../webthree-helpers/utils/secp256k1/libsecp256k1.a 
-lgmp /src/built/lib/libminiupnpc.a -lboost_regex 
../libethash/libethash.a /src/built/lib/libcryptlib.a 

I notice that ethminer isn't linking the json rpc server, only the client, so maybe there is some problem from there.

[bobsummerwill]

From @rainbeam on May 1, 2016 1:2

Manually appending -lcurl -lssh2 -lssl -lcrypto -lssl -lcrypto -lz fixes it (that's the output of pkg-config --libs --static libcurl). Now just need to coerce cmake into adding it.

[bobsummerwill]

'I notice that ethminer isn't linking the json rpc server, only the client, so maybe there is some problem from there."

^ Yeah. I think that this whole stack of "JSON-RPC stuff" which then drags in the curl chain is all a very thin thread of dependencies, but which is a bit messy. I did some minimization before which removed the unnecessary JsonRpc::Server dependency from ethminer. There is probably plenty more which can be trimmed, if we better understood it.

Good luck!

[bobsummerwill]

So it looks like part of my CURL REQUIRED thing was due to only having a partially synced workspace, but I've made one clean up which is building now, and then that's good and everything is healthy for the dynamic builds and you just have that curl link order thing for ethminer left.

I'm going to try doing static builds for OS X and Windows now, for eth at least, and hopefully more if I have time.

[bobsummerwill]

See ethereum/webthree-helpers#167.

[bobsummerwill]

From @rainbeam on May 1, 2016 17:45

I've got a Dockerfile for building (unsuccessfully) on Debian here:

https://github.com/rainbeam/eth-static/blob/debian/Dockerfile

Really minimal curl can be configured with:

./configure --prefix=/opt/curl --enable-static --disable-shared \
--disable-ldap --disable-ldaps --without-libidn --disable-rtsp --without-librtmp \
--disable-manual --disable-tls-srip --without-gnutls \
--without-ssl --without-libssh2 --without-zlib

However, even if we get past the curl linking problems, there is a bigger problem in that glibc is not designed to be statically linked. The STATIC_LINKING that I've implemented tries to static link everything. This works fine on a musl based platform (i.e. Alpine), but isn't going to work on Debian / Ubuntu. I don't know what the story is on Windows.

The binary this produces is completely portable across Linux based distros though - the limitation is only in the build environment.

As such, I don't regard the Debian / Ubuntu problems as part of this issue. The ethminer problem is still relevant though.

As for other platforms (other than x86_64)... I've not looked into it much but I saw that there are musl cross compilers provided here:

http://musl.codu.org/

Might be useful in your mobile adventures 😄 .

[bobsummerwill]

Thanks for that extra info, which will indeed be useful. Flipping to a statically linked musl cross-compiled runtime could be very useful.

I found that OS X as does not want to be completely statically linked (see https://developer.apple.com/library/mac/qa/qa1118/_index.html), so added checks for that to give an error message rather than just failing. I will try to "weaken that off", so that it statically links everything except the C++ runtime library.

Windows can be used with either DLLs or static libraries for the runtime. That will because unlike Linux, there is a binary compatibility guarantee for Windows, which is why old Windows apps work for years and years. Linux, I understand, has never had that, hence the monolithic kernel including drivers for everything under the sun. They have to be in-tree because the ABI can change arbitrarily between kernel and drivers.

Seems the ABI guarantee is a bit weak on OS X too.

As for curl, yeah, would be great to do a minimal compile like that, and I will certainly refer back to your notes here as-and-when I get around to pulling that in as a sub-module.

Did you try building curl like that for your Alpine Linux build? So with such settings, if looks like OpenSSL, ssh2 and zlib would NOT be needed?

[bobsummerwill]

"Manually appending -lcurl -lssh2 -lssl -lcrypto -lssl -lcrypto -lz fixes it (that's the output of pkg-config --libs --static libcurl). Now just need to coerce cmake into adding it."

^ Did you manage to work out the magic spell for this, btw? :-)

[bobsummerwill]

From @rainbeam on May 3, 2016 10:18

No. I messed around a bit with moving the eth_use(CURL) around and with linking against a minimal libcurl, but no luck. Stopped short of doing a full debug of how CMake constructs its linker arguments. It doesn't have a verbose logging mode, so I think you have to debug by adding lots of message calls.

Also tried a bit more with Debian. I reckon it is possible to build fully static by using musl-gcc from musl-tools, but still ran into problems with linking curl.

https://github.com/rainbeam/eth-static/tree/ethminer

https://github.com/rainbeam/eth-static/tree/debian

Did you have a look at Windows? (sorry missed your post above)

[bobsummerwill]

From @rainbeam on May 3, 2016 13:20

Just did an Alpine build with minimal curl - ssh2, openssl and zlib appear not to be needed.

[bobsummerwill]

Thanks for the update.

I've been doing a partially static build for OS X, with some success, and will try Windows as well when I get a chance.

"Just did an Alpine build with minimal curl - ssh2, openssl and zlib appear not to be needed." - That is good to know, and makes sense.

OS X static build is pulling in Snappy as an indirect dependency of leveldb, which is a bit weird, but I suspect it is just an artifact of the fact that leveldb is still a dylib dependency at the partial stage I got to:

[100%] Linking CXX executable eth
[100%] Built target eth
Bobs-MacBook-Air:build bob$ otool -L webthree/eth/eth
webthree/eth/eth:
    /usr/local/opt/jsoncpp/lib/libjsoncpp.1.dylib (compatibility version 1.0.0, current version 1.6.5)
    /usr/local/opt/leveldb/lib/libleveldb.1.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/miniupnpc/lib/libminiupnpc.15.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
    /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
    /usr/local/opt/libjson-rpc-cpp/lib/libjsonrpccpp-common.0.dylib (compatibility version 0.0.0, current version 0.6.0)
    /usr/local/opt/libjson-rpc-cpp/lib/libjsonrpccpp-client.0.dylib (compatibility version 0.0.0, current version 0.6.0)
    /usr/local/opt/libjson-rpc-cpp/lib/libjsonrpccpp-server.0.dylib (compatibility version 0.0.0, current version 0.6.0)
    /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/opt/gmp/lib/libgmp.10.dylib (compatibility version 14.0.0, current version 14.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

So, getting to a static binary isn't possible on OS X, but statically linking all the libs should be possible. The binary I have right now runs just fine, but the following libs are still using .dylib, not .a:

• jsoncpp
• json-rpc-cpp
• leveldb
• miniupnp
• gmp

To get those ones statically linking, I'll need to build them from source. libz and curl ship as part of the platform, but again I could build them from source and statically link them.

[bobsummerwill]

PS. OS X partially statically linked executable is 12MB.

[bobsummerwill]

OK - So testing this for Windows is a horror-show, because we'll need LIB files for all the pre-built external dependencies. I'm going to defer that for now.

So @rainbeam ... are you pretty much done here now?

And you still didn't work out a magic-mantra to get the libcurl ordering thing resolved, right? That was on Debian? Or Alpine?

[bobsummerwill]

From @rainbeam on May 4, 2016 23:41

Yeah I think I'm done here.

Couldn't resolve the libcurl problem, except by manually modifying the link command after cmake was configured. This only affects ethminer. The problem exists on both Alpine and Debian.

Debian has the added problem of glibc, which isn't going to statically link. A similar solution to OSX (i.e. dynamic linking the system libs) might be needed if people want to build on that platform (rather than just use a static binary compiled on a musl system). Alternatively, Debian does have musl-tools etc. which may allow actual building of a static binary from Debian. I couldn't quite get this working.

Thanks for your help on this Bob :)

[bobsummerwill]

FYI ... My first Boost post made it through their filters ...

http://lists.boost.org/Archives/boost/2016/05/229385.php

... the one about thread setname/getname.

And got a first reply:

http://lists.boost.org/Archives/boost/2016/05/229388.php

[bobsummerwill]

See boostorg/thread#84, where there is now an issue tracking the Boost TODO, which I have offered to put on my backlog.

[bobsummerwill]

Hey @rainbeam,

I have been working on an infrastructure-as-code alternative to these tedious per-distro sets of instructions, which will "go live" when we switch back to cpp-ethereum.

You can see my work-in-progress on that here:

• https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/.travis.yml
• https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh

Now for the TravisCI bit, we're stuck with just Ubuntu Trusty automation for the time being, but we might well be able to expand that via Docker in the future.

Whatever the case, I would like to expand this install_deps.sh pattern pattern to cover more distros, so that our ethdocs.org instructions for the distros which we have bothered to do that can be simplified to, with the steps being executed directly and committed in with the code itself.

• git clone --recursive
• ./install_deps.sh
• cmake
• make

So maybe for Alpine we can take that approach instead, rather than bothering with written instructions?

Please could help me out and copy https://github.com/bobsummerwill/cpp-ethereum/blob/merge_repos/install_deps.sh locally and run it, does it go into the right conditional?

I suspect that it won't work, and that we will need to do something different for Alpine detection.

[bobsummerwill]

install_deps request split out into ethereum/webthree-umbrella#633.

[bobsummerwill]

Hey @rainbeam!
How are you doing?

See http://doublethinkco.github.io/cpp-ethereum-cross/images/dependency_graph.svg which I've just generated as part of the oh-so-nearly-finished cpp-ethereum Homecoming, including those tricky little bastard curl dependencies!

[bobsummerwill]

Are you up for helping me out on the install.sh stuff for Alpine?

I want to get the Alpine flow to be super-simple, so I can add Alpine static-build automation via Docker into our flow. Because I think that those statically-linked Linux executables are actually going to be THE BEST executables which we can supply on Linux.

See also ethereum/webthree-umbrella#664.

All that bullshit is pretty analogous to this bullshit which I have still to resolve for the cross-builds on Tizen for the Gear S2 smartwatch, which is where I started my whole Ethereum adventure nearly a year ago and still have not solved!

doublethinkco/cpp-ethereum-cross#20

[bobsummerwill]

Damn you, GLIBC!

./eth: /usr/lib/libstdc++.so.6: version 'GLIBCXX_3.4.21' not found (required by ./eth)

[bobsummerwill]

Tizen 2.4 ships with C runtime from the ancient GCC 4.6.

[bobsummerwill]

From @rainbeam on July 22, 2016 21:25

hey @bobsummerwill: sure I can help out with static Alpine builds. Will be great when I can just download the latest solidity static binary :)

Are we targeting just eth and solc or is it wider than that?

I'm getting a bit lost with the repo reorg. Will there be a install.sh in both solidity and cpp-ethereum repos?

[bobsummerwill]

Great!

So the reorg changes aren't live in the final repos, yet, though they will be in the next few days

These are my working branches:

• https://github.com/bobsummerwill/cpp-ethereum/tree/merge_repos
• https://github.com/bobsummerwill/solidity/tree/standalone_prior_to_splat

So, yes, both will have scripts/install.sh files, which will be the ones shown above.

[bobsummerwill]

And I want to get all executables working with static builds :-)

Thanks for the offer of help. I'll give you a poke when the repo switches are done and it is less head-scratchy!

[bobsummerwill]

From @nerdralph on August 2, 2016 0:43

I have been looking into building a static ethminer (based on Genoil's fork), and got almost there.
https://github.com/nerdralph/ethminer-nr/blob/110/releases/ethminer-genoil-1.1.8nr.tgz

I ran into the libcurl issue (without having read this thread first), and my initial "solution" was just to dynamically link libcurl. Here's what I've figured out in the process:
Ethminer doesn't directly depend on libcurl, that dependency arises from libjson-rpc. If you check the external dependencies of /usr/lib/libjsonrpccpp-client.so with ldd you'll see libcurl.so.4. And since there is no direct dependency on curl, it's possible to do a dynamic linked build with curl removed from EthDependencies.cmake. Doing a mostly static build (with curl still dynamically-linked) isn't as simple as adding "set(CMAKE_FIND_LIBRARY_SUFFIXES .a .so)", and that's because there is no libcurl.so. When libjsonrpcccpp-client.a is statically linked, libcurl.so needs to be on the link line to resolve the undefined symbols. The convention in Linux is that libcurl.so would be a symlink to libcurl.so.4, and libcurl.so.4 would be a symlink to the actual file, which in the case of Ubuntu 14.04 is libcurl.so.4.3.0 (which is part of the oddly-named libcurl3 package). But as there's no libcurl.so, my hacked solution was to explicitly list libcurl.so.4 in FindCURL.cmake. FindOpenCL.cmake does the same kind of thing, so apparently someone before me considered to be a suitable (or the least offensive) work-around.
Today I noticed cmake has a FindCURL module, and thought that might be smart enough to find libcurl.so.X, but it doesn't.
https://cmake.org/cmake/help/v3.0/module/FindCURL.html

I tried using libcurl.a from libcurl4-openssl-dev instead of libcurl4-gnutls-dev, but that wasn't any better. I think the issue is the plethora of dependencies for ssl support. My next though is that a custom build of libcurl without ssl may work, since ethminer doesn't use ssl connections. I had to do something similar (do a local build) for cryptopp. I still haven't cleanly integrated that into the cmake files, as I plan to test including cyrptopp as a git submodule.

If anyone has some less-hackish ideas for getting this to work, I'm all ears. Otherwise, I'll continue to pick away at it.

[bobsummerwill]

From @rainbeam on August 2, 2016 9:5

@nerdralph: great!

As I recall, I could get ethminer to compile when I manually fiddled with the gcc command that cmake generated, but couldn't get cmake to actually generate it. Also had issues with gnutls, but only on Debian - I found building on an Alpine docker image to be simpler. I think the only curl dep was in jsonrpccpp, and that only seems to need curl itself (no ssl etc.), but cmake doesn't seem to be able to figure that out (even when explicitly told about a bare build of libcurl).

Please keep hacking away! My approach was completely hacky. It'll probably help that you actually use ethminer.

[bobsummerwill]

From @nerdralph on August 2, 2016 15:36

@rainbeam Thanks for the feedback. I had started by patching into Genoil's fork the cmake changes that I guess you had committed for static linking. I was confused at first because some of the changes didn't seem to do anything, but now that I know it's still a work in progress it makes more sense. I, too, got my first static build by hacking the makefile generated by cmake. Then I did some more debugging to understand a bit more about the order of execution of the cmake files.
The current state of things is that under Ubuntu 14.04 I use the following command line for my mostly-static build:
cmake -DBUNDLE=miner -DCRYPTOPP_ROOT_DIR=pwd -DETH_STATIC=1 ..

I've added a ETH_SHARED to enable building shared libs (for libdevcore, libethcore, etc).
https://github.com/nerdralph/ethminer-nr/blob/110/cmake/EthCompilerSettings.cmake#L95
The default is now to generate static libs for them (i.e. .a files under Unix), but search for and link shared libs for things like boost.

[bobsummerwill]

Thanks to you both!

I think it will be such a big win when we get to the end of this adventure.

Ultimately, I think we're going to want to build all of the external dependencies from source for the degree of control of build options we're going to need for things like libcurl, following @chfast's path with LLVM for evmjit.

You will have seen this updated dependency-graph, right?

https://camo.githubusercontent.com/b59d6b4aa624edfa51c707ac543369c1d90b2e3e/687474703a2f2f646f75626c657468696e6b636f2e6769746875622e696f2f6370702d657468657265756d2d63726f73732f696d616765732f646570656e64656e63795f67726170682e737667

So I was looking at ethminer's dependencies, and I think that it is very likely that we can do some minor refactoring within ethashseal to sever that hard dependency onto libethereum when we are building ethminer. That will make the ethminer dependency tree much more manageable, but you can still see the mess of indirect dependencies dangling off json-rpc-cpp. If we can build a very constrained subset of libcurl from source then maybe that wouldn't be so bad.

[chfast]

EVMJIT will provide evmjit-static target if you want to link statically.

[Prostu2]

Did anyone managed to make it static ,i cant compile it on centos and redhat ,i really need it to work on these,thanks

[chfast]

I'm working on it here: #3518. It's pretty close.

I also added support for CentOS in install_deps.sh script. Adding RHEL should not be difficult.

[nerdralph]

I was able to get my fork building mostly static:

 ldd ethminer
        linux-vdso.so.1 =>  (0x00007ffc57b47000)
        libOpenCL.so.1 => /usr/lib/x86_64-linux-gnu/libOpenCL.so.1 (0x00007f26ecea2000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f26ecc84000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f26ec971000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f26ec66b000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f26ec455000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f26ec08f000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f26ebe8b000)
        /lib64/ld-linux-x86-64.so.2 (0x000055b75bfce000)

https://github.com/nerdralph/ethminer-nr

[chfast]

Nice. I don't think you can go much further. These libraries, except OpenCL should be available on any Linux.

What do you want to do with it if I may ask?

[nerdralph]

@chfast I've been releasing Linux binaries for Ubuntu 14.
https://github.com/nerdralph/ethminer-nr/tree/110/releases

ETH mining is looking like a dead end, so I've been more involved in ZEC miner development over the last few months.

[chfast]

What do you mean by the dead end?
Have you modified the ethminer in any way?

[nerdralph]

@chfast In 6 months from now eth mining rewards will likely be 1/2 of what they are now. Some time "soon" after that, mining rewards will approach 0 with the switch to PoS.
ethereum/EIPs#186

As for modifications, I improved on JW(Genoil's) stratum code. He merged virtually all of my changes back into version, so the primary difference is that he released Windows binaries while I released Linux binaries.