Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build (squid-c…

…ache#1877)

    gmake[3]: Entering directory .../libltdl
    .../cfgaux/missing: line 85: aclocal-1.16: command not found
    gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127

During bootstrap.sh run, libtoolize copies prepackaged configure and
Makefile.in files into our libltdl directory:

* libltdl/configure from libtool v2.4 has aclocal version set to 1.16;
* libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal
  version to build aclocal.m4

Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs
aclocal-1.16 if "make" needs to build libltdl/aclocal.m4.

Normally, "make" does not need to build libltdl/aclocal.m4 because that
file was created by libtoolize. However, libtool v2.4 is packaged with
(generated by packaging folks) libltdl/Makefile.in that makes
libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory.
Squid does not have that ./m4 directory, so "make" attempts to
re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16.

Our bootstrap.sh generated new ./configure but preserved copied
libltdl/configure with its aclocal version set to 1.16. In other words,
our bootstrap.sh did not bootstrap libltdl sub-project. In build
environments without aclocal-1.16, Squid build failed.

Several solutions or workarounds have been tried or considered:

* Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt
  to do that was reverted in commit bfd6b6a with "better to fix libtool
  installation" rationale. Another potential argument against this
  option is that packages should be bootstrapped by their distributors,
  not "users". We are not distributing libtool, but this is a gray area
  because we do distribute files that libtoolize creates. Finally,
  libtool itself does not provide a bootstrapping script and does not
  explicitly recommend bootstrapping in documentation.

* "Fix libtool installation". We failed to find a good way to do that on
  MacOS (without building and installing newer libtool from sources).

* Place m4 files where libtool v2.4 expects to find them. That change
  fixes MacOS builds that use automake v1.17, but breaks Gentoo builds
  because Gentoo libtool installs a buggy libltdl/Makefile.in that must
  be regenerated by automake before it can work. Fixing m4 files
  location prevents that regeneration.

We picked the first option despite its drawbacks because the third
option did not work on Gentoo, and asking Squid developers to install
libtool from sources (i.e. the second option) felt like a greater evil.

This old problem was exposed by recently introduced CI MacOS tests that
started to fail when MacOS brew updated automake package from v1.16
without the corresponding libtoolize package changes.


Also work around what seems to be a libtool packaging bug affecting
MacOS/brew environments, including GitHub Actions runners we use for CI:

    libtool  (2.4.7) : glibtool
    libtool path : /opt/homebrew/bin
    Bootstrapping
    glibtoolize:   error: creating 'libltdl/configure.ac' ... failed
    glibtoolize:   error: creating 'libltdl/configure' ... failed
    glibtoolize failed

That workaround will be removed after libtool package is fixed.


Also removed a single-iteration "for dir" loop with several stale hacks
from bootstrap.sh: With only two directories to bootstrap and with a
directory-specific mkdir command, source comments, and progress
messages, it is best to unroll that loop.

.github/workflows/default.yaml

@@ -159,6 +159,21 @@ jobs:
           export CPPFLAGS="-I$HOMEBREW_PREFIX/include${CPPFLAGS:+ $CPPFLAGS}"
           export LDFLAGS="-L$HOMEBREW_PREFIX/lib${LDFLAGS:+ $LDFLAGS}"
           export CFLAGS="-Wno-compound-token-split-by-macro${CFLAGS:+ $CFLAGS}" # needed fir ltdl with Xcode
+
+          # libtool package referenced below fails to copy its configure*
+          # files, possibly due to a packaging/brewing bug. The following sed
+          # command restores installed libtoolize code to its earlier (and
+          # working) variation.
+          echo "brew libtool package details:"
+          brew info libtool --json | grep -E 'rebuild|tap_git_head'
+          # This hack was tested on libtoolize package with the following output:
+          # "rebuild": 2,
+          # "tap_git_head": "5cede8ea3b7b12c7f68215f75a951430b38d945f",
+          #
+          editable=$HOMEBREW_CELLAR/libtool/2.4.7/bin/glibtoolize
+          sed -i.bak 's@ltdl_ac_aux_dir=""@ltdl_ac_aux_dir="../build-aux"@' $editable || true
+          diff -u $editable.bak $editable || true
+
           ./test-builds.sh ${{ matrix.layer.name }}
 
       - name: Publish build logs

bootstrap.sh

@@ -117,38 +117,47 @@ echo "autoconf ($acversion) : autoconf$acver"
 echo "libtool  ($ltversion) : ${LIBTOOL_BIN}${ltver}"
 echo "libtool path : $ltpath"
 
-for dir in \
-	""
-do
-    if [ -z "$dir" ] || [ -d $dir ]; then
-	if (
-	echo "Bootstrapping $dir"
-	cd ./$dir
-	if [ -n "$dir" ] && [ -f bootstrap.sh ]; then
-	    ./bootstrap.sh
-	elif [ ! -f $dir/configure ]; then
-	    # Make sure cfgaux exists
-	    mkdir -p cfgaux
-
-            if test -n "$ltpath"; then
-              acincludeflag="-I $ltpath/../share/aclocal"
-            else
-              acincludeflag=""
-            fi
-
-	    # Bootstrap the autotool subsystems
-	    bootstrap aclocal$amver $acincludeflag
-	    bootstrap autoheader$acver
-	    bootstrap_libtoolize ${LIBTOOL_BIN}ize${ltver}
-	    bootstrap automake$amver --foreign --add-missing --copy -f
-	    bootstrap autoconf$acver --force
-	fi ); then
-	    : # OK
-	else
-	    exit 1
-	fi
+if test -n "$ltpath"; then
+    acincludeflag="-I $ltpath/../share/aclocal"
+else
+    acincludeflag=""
+fi
+
+# bootstrap primary or subproject sources
+bootstrap_dir() {
+    dir="$1"
+    cd $dir || exit $?
+
+    bootstrap aclocal$amver $acincludeflag
+    bootstrap autoheader$acver
+
+    # Do not libtoolize ltdl
+    if grep -q '^LTDL_INIT' configure.ac
+    then
+        bootstrap_libtoolize ${LIBTOOL_BIN}ize${ltver}
     fi
-done
+
+    bootstrap automake$amver --foreign --add-missing --copy --force
+    bootstrap autoconf$acver --force
+
+    cd - > /dev/null
+}
+
+echo "Bootstrapping primary Squid sources"
+mkdir -p cfgaux || exit $?
+bootstrap_dir .
+
+# The above bootstrap_libtoolize step creates or updates libltdl. It copies
+# (with minor adjustments) configure.ac and configure, Makefile.am and
+# Makefile.in from libtool installation, but does not regenerate copied
+# configure from copied configure.ac and copied Makefile.in from Makefile.am.
+# We get libltdl/configure and libltdl/Makefile.in as they were bootstrapped
+# by libtool authors or package maintainers. Low-level idiosyncrasies in those
+# libtool files result in mismatches between copied code expectations and
+# Squid sub-project environment, leading to occasional build failures that
+# this bootstrapping addresses.
+echo "Bootstrapping libltdl sub-project"
+bootstrap_dir libltdl
 
 # Make a copy of SPONSORS we can package
 if test -f SPONSORS.list; then