From 9414219d510a3ec22f6712d98f4a9c2ad2dc140b Mon Sep 17 00:00:00 2001 From: Alex Rousskov Date: Mon, 5 Aug 2024 12:18:02 +0000 Subject: [PATCH] Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build (#1877) gmake[3]: Entering directory .../libltdl .../cfgaux/missing: line 85: aclocal-1.16: command not found gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127 During bootstrap.sh run, libtoolize copies prepackaged configure and Makefile.in files into our libltdl directory: * libltdl/configure from libtool v2.4 has aclocal version set to 1.16; * libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal version to build aclocal.m4 Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs aclocal-1.16 if "make" needs to build libltdl/aclocal.m4. Normally, "make" does not need to build libltdl/aclocal.m4 because that file was created by libtoolize. However, libtool v2.4 is packaged with (generated by packaging folks) libltdl/Makefile.in that makes libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory. Squid does not have that ./m4 directory, so "make" attempts to re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16. Our bootstrap.sh generated new ./configure but preserved copied libltdl/configure with its aclocal version set to 1.16. In other words, our bootstrap.sh did not bootstrap libltdl sub-project. In build environments without aclocal-1.16, Squid build failed. Several solutions or workarounds have been tried or considered: * Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt to do that was reverted in commit bfd6b6a9 with "better to fix libtool installation" rationale. Another potential argument against this option is that packages should be bootstrapped by their distributors, not "users". We are not distributing libtool, but this is a gray area because we do distribute files that libtoolize creates. Finally, libtool itself does not provide a bootstrapping script and does not explicitly recommend bootstrapping in documentation. * "Fix libtool installation". We failed to find a good way to do that on MacOS (without building and installing newer libtool from sources). * Place m4 files where libtool v2.4 expects to find them. That change fixes MacOS builds that use automake v1.17, but breaks Gentoo builds because Gentoo libtool installs a buggy libltdl/Makefile.in that must be regenerated by automake before it can work. Fixing m4 files location prevents that regeneration. We picked the first option despite its drawbacks because the third option did not work on Gentoo, and asking Squid developers to install libtool from sources (i.e. the second option) felt like a greater evil. This old problem was exposed by recently introduced CI MacOS tests that started to fail when MacOS brew updated automake package from v1.16 without the corresponding libtoolize package changes. Also work around what seems to be a libtool packaging bug affecting MacOS/brew environments, including GitHub Actions runners we use for CI: libtool (2.4.7) : glibtool libtool path : /opt/homebrew/bin Bootstrapping glibtoolize: error: creating 'libltdl/configure.ac' ... failed glibtoolize: error: creating 'libltdl/configure' ... failed glibtoolize failed That workaround will be removed after libtool package is fixed. Also removed a single-iteration "for dir" loop with several stale hacks from bootstrap.sh: With only two directories to bootstrap and with a directory-specific mkdir command, source comments, and progress messages, it is best to unroll that loop. --- .github/workflows/default.yaml | 15 +++++++ bootstrap.sh | 71 +++++++++++++++++++--------------- 2 files changed, 55 insertions(+), 31 deletions(-) diff --git a/.github/workflows/default.yaml b/.github/workflows/default.yaml index f3da5e41e72..0204f95a2c4 100644 --- a/.github/workflows/default.yaml +++ b/.github/workflows/default.yaml @@ -159,6 +159,21 @@ jobs: export CPPFLAGS="-I$HOMEBREW_PREFIX/include${CPPFLAGS:+ $CPPFLAGS}" export LDFLAGS="-L$HOMEBREW_PREFIX/lib${LDFLAGS:+ $LDFLAGS}" export CFLAGS="-Wno-compound-token-split-by-macro${CFLAGS:+ $CFLAGS}" # needed fir ltdl with Xcode + + # libtool package referenced below fails to copy its configure* + # files, possibly due to a packaging/brewing bug. The following sed + # command restores installed libtoolize code to its earlier (and + # working) variation. + echo "brew libtool package details:" + brew info libtool --json | grep -E 'rebuild|tap_git_head' + # This hack was tested on libtoolize package with the following output: + # "rebuild": 2, + # "tap_git_head": "5cede8ea3b7b12c7f68215f75a951430b38d945f", + # + editable=$HOMEBREW_CELLAR/libtool/2.4.7/bin/glibtoolize + sed -i.bak 's@ltdl_ac_aux_dir=""@ltdl_ac_aux_dir="../build-aux"@' $editable || true + diff -u $editable.bak $editable || true + ./test-builds.sh ${{ matrix.layer.name }} - name: Publish build logs diff --git a/bootstrap.sh b/bootstrap.sh index d7317f32452..23025de7429 100755 --- a/bootstrap.sh +++ b/bootstrap.sh @@ -117,38 +117,47 @@ echo "autoconf ($acversion) : autoconf$acver" echo "libtool ($ltversion) : ${LIBTOOL_BIN}${ltver}" echo "libtool path : $ltpath" -for dir in \ - "" -do - if [ -z "$dir" ] || [ -d $dir ]; then - if ( - echo "Bootstrapping $dir" - cd ./$dir - if [ -n "$dir" ] && [ -f bootstrap.sh ]; then - ./bootstrap.sh - elif [ ! -f $dir/configure ]; then - # Make sure cfgaux exists - mkdir -p cfgaux - - if test -n "$ltpath"; then - acincludeflag="-I $ltpath/../share/aclocal" - else - acincludeflag="" - fi - - # Bootstrap the autotool subsystems - bootstrap aclocal$amver $acincludeflag - bootstrap autoheader$acver - bootstrap_libtoolize ${LIBTOOL_BIN}ize${ltver} - bootstrap automake$amver --foreign --add-missing --copy -f - bootstrap autoconf$acver --force - fi ); then - : # OK - else - exit 1 - fi +if test -n "$ltpath"; then + acincludeflag="-I $ltpath/../share/aclocal" +else + acincludeflag="" +fi + +# bootstrap primary or subproject sources +bootstrap_dir() { + dir="$1" + cd $dir || exit $? + + bootstrap aclocal$amver $acincludeflag + bootstrap autoheader$acver + + # Do not libtoolize ltdl + if grep -q '^LTDL_INIT' configure.ac + then + bootstrap_libtoolize ${LIBTOOL_BIN}ize${ltver} fi -done + + bootstrap automake$amver --foreign --add-missing --copy --force + bootstrap autoconf$acver --force + + cd - > /dev/null +} + +echo "Bootstrapping primary Squid sources" +mkdir -p cfgaux || exit $? +bootstrap_dir . + +# The above bootstrap_libtoolize step creates or updates libltdl. It copies +# (with minor adjustments) configure.ac and configure, Makefile.am and +# Makefile.in from libtool installation, but does not regenerate copied +# configure from copied configure.ac and copied Makefile.in from Makefile.am. +# We get libltdl/configure and libltdl/Makefile.in as they were bootstrapped +# by libtool authors or package maintainers. Low-level idiosyncrasies in those +# libtool files result in mismatches between copied code expectations and +# Squid sub-project environment, leading to occasional build failures that +# this bootstrapping addresses. +echo "Bootstrapping libltdl sub-project" +bootstrap_dir libltdl # Make a copy of SPONSORS we can package if test -f SPONSORS.list; then