WIP: ./configure

[rustyrussell]

[ On top of gossip changes, so ignore all patches up to and including 'gossipd: minor cleanups.' ]

This adds a ./configure step for those of us who hate remembering to 'make DEVELOPER=1'. It's not autoconf, but if you can write shell it should be pretty simple to enhance.

Main user-visible change is that you need to use 'VALGRIND=0' instead of 'NO_VALGRIND=1' to override valgrind testing.

@jsarenik might want to comment, since he's doing exotic cross-builds!

[jsarenik]

Will check it out today. Thanks!

[jsarenik]

Hi Rusty! I'm still not at the computer but so far I've read the commit names. On Mon, Jun 4, 2018, 08:32 Rusty Russell wrote via GitHub,:

- ccan: update to have new configurator. - configure: new script. - configure: add --enable-valgrind/disable-valgrind. - Makefile: import config.vars. - Makefile: not running configure first is so yesterday: - travis: use VALGRIND rather than NO_VALGRIND. - devtools/Makefile: add devtools/onion.c to DEVTOOLS_TOOL_SRC - external/Makefile: call git submodule sync and update with --remote. - libbacktrace: pull from rustyrussell version until config.guess fix applied.

Thank you very much especially for the last one in libbacktrace! The rest I'll have a look at when I'm at the computer.

[jsarenik]

Already compiling. Not a lot of success yet. I have a note which is not really related to this particular HEAD, but it is still present in it: There is a race condition when make -j# is issued (where # is a natural number higher than two, e.g. 4).

[jsarenik]

Two things. Here is the fix for the race condition. Start with cleanly cloned lightning repository. The fix lies in initiating the git submodules early and in non-simultaneous way by calling

git submodule update --init

There could be also --recursive but it is not needed in our case (yet). By the way this can be simplified into one clone command as (excuse the usage of jsarenik fork, but for the sake of a working Alpine Linux example):

lomidrevo:/tmp$ rm -rf lightning; git clone https://github.com/jsarenik/lightning -b rusty-guilt-configure --recurse-submodules
...
lomidrevo:/tmp$ cd lightning
lomidrevo:/tmp/lightning$ ./configure --enable-developer --disable-valgrind
...

Then after make -j4 I have a nicely compiled and perfectly working lightning. My conclusion is that no git magic should happen from inside the Makefiles.

Try to run the same as above, but without the --recurse-submodules. I get unfinished makes so that I can run while ! make; do :; done and still do not get any successfully finished make. See #1542.

The other thing is that only the latest config.guess and config.sub fixes the compilation issue for me on Alpine Linux. I have sent a PR rustyrussell/libbacktrace#2

And have checked it locally with changing your (@rustyrussell) commit to (already found in above-mentioned jsarenik/rusty-guilt-compile branch):

diff --git a/.gitmodules b/.gitmodules
index cada6332..c0410770 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -9,7 +9,7 @@
        url = https://github.com/jedisct1/libsodium.git
 [submodule "external/libbacktrace"]
        path = external/libbacktrace
-       url = https://github.com/ianlancetaylor/libbacktrace.git
+       url = https://github.com/jsarenik/libbacktrace.git
 [submodule "external/libwally-core"]
        path = external/libwally-core
        url = https://github.com/ElementsProject/libwally-core.git
diff --git a/external/libbacktrace b/external/libbacktrace
index 14d377e9..6aecc10c 160000
--- a/external/libbacktrace
+++ b/external/libbacktrace
@@ -1 +1 @@
-Subproject commit 14d377e9be7c89511c472d728e1b88b4e96f1946
+Subproject commit 6aecc10cb6830fcb04d890c7f907c91bb5e65dea

[jsarenik]

Oh, by the way my so-called race condition is not really a race condition because it happens even when make is called without the -j flag. The test (while ! make; do :; done) really did not end after many rounds...

Here is how to fix it in the same directory just after pressing ^C breaking the infinite failure loop:

git submodule deinit --all -f
git submodule update --init

Now run make, even parallel (with -j4) and it compiles on the first try. Yeah.

[jsarenik]

And because I am knowingly mixing two issues here, I will make a new one for the "race condition" with a full log.

[jsarenik]

The race-condition issue is fully documented in #1543

[jsarenik]

Upstream libbacktrace already updated. I've sent a PR to this branch. rustyrussell#2

[rustyrussell]

"This time for sure"!

[bitonic-cjp]

See #1544: I have a feature request for the configure script: choose between pytest-3 and pytest (whichever is available; use pytest-3 if both exist and user didn't manually override).

What is the preferred way to do this? Try to make it myself and make a pull request for rustyrussell/lightning/tree/guilt/configure ? Wait until someone else (@rustyrussell ?) adds this? Something else?

[jsarenik]

Oops. Seems like there is some problem when I do a clean clone and build. See further comments...

[jsarenik]

@rustyrussell The whole problem lies in checksum for libbacktrace, which was changed to the tip in your fork (rustyrussell/libbacktrace@078f841) in f660271, but the submodule uses the original repository which does not contain that commit.

Here is the ammended and corrected commit:

From acd087192ea0dd84b3694796f61110d1a6669463 Mon Sep 17 00:00:00 2001
From: Rusty Russell <[email protected]>
Date: Wed, 6 Jun 2018 16:11:49 +0930
Subject: [PATCH] libbacktrace: pull latest version, which has config.guess fix
 applied.

Signed-off-by: Rusty Russell <[email protected]>
---
 external/libbacktrace | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/external/libbacktrace b/external/libbacktrace
index 14d377e9..5a99ff7f 160000
--- a/external/libbacktrace
+++ b/external/libbacktrace
@@ -1 +1 @@
-Subproject commit 14d377e9be7c89511c472d728e1b88b4e96f1946
+Subproject commit 5a99ff7fed66b8ea8f09c9805c138524a7035ece
-- 
2.17.1

After this is applied instead of f660271 and the code is rebased to the current master branch, and VALGRIND=0 is change is done also in Python tests, then as for me and my house it is ready to be merged.

Full example:

• Script: https://gist.github.com/jsarenik/13a8bdee119df1d1eb40c09cd39fb1d1
• Full log from Ubuntu 18.04: https://gist.github.com/jsarenik/deac2376402cfb07ca6726bd7086cef2

[jsarenik]

Mystery solved: rustyrussell/libbacktrace@078f841 was your local commit.

I have edited/removed my previous comments accordingly.

[jsarenik]

@bitonic-cjp I would suggest you to wait until this gets merged into master. Then rebase your work on top of it.

[jsarenik]

Another issue is that when configured with --disable-valgrind, the Python tests still try to run with Valgrind.

Fix available in jsarenik@690c366

PR sent to Rusty's branch: rustyrussell#4

[jsarenik]

@rustyrussell @cdecker @ZmnSCPxj

Should the default for Python tests run without environment be with or without Valgrind?
https://github.com/rustyrussell/lightning/blob/guilt/configure/tests/test_lightningd.py#L34

Sorry for spam. I realized this is totally off-topic. Have a look at rustyrussell@b8e26af without which Valgrind is still run for me.

[jsarenik]

Last changes done. Let's call it a day here in the old Prague. Who will test the tests?

https://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3F

[rustyrussell]

Thanks! I rebased and applied your fixes.

Turns out guilt doesn't play well with submodules, but who does?

[jsarenik]

Thanks for sharing! I didn't know about that vulnerability.

[jsarenik]

ACK 0a094b9 (no idea what the flags mean and if I can ack at all, but here is my 0.00000002 BTC).

[cdecker]

Damn, sorry about that, I didn't see the WIP in the PR title and went ahead and merged it ☹️

[jsarenik]

Damn, sorry about that, I didn't see the WIP in the PR title and went ahead and merged it

And here I am :-) again