Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build #1877

Conversation

rousskov
Copy link
Contributor

@rousskov rousskov commented Aug 2, 2024

gmake[3]: Entering directory .../libltdl
.../cfgaux/missing: line 85: aclocal-1.16: command not found
gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127

During bootstrap.sh run, libtoolize copies prepackaged configure and
Makefile.in files into our libltdl directory:

  • libltdl/configure from libtool v2.4 has aclocal version set to 1.16;
  • libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal
    version to build aclocal.m4

Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs
aclocal-1.16 if "make" needs to build libltdl/aclocal.m4.

Normally, "make" does not need to build libltdl/aclocal.m4 because that
file was created by libtoolize. However, libtool v2.4 is packaged with
(generated by packaging folks) libltdl/Makefile.in that makes
libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory.
Squid does not have that ./m4 directory, so "make" attempts to
re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16.

Our bootstrap.sh generated new ./configure but preserved copied
libltdl/configure with its aclocal version set to 1.16. In other words,
our bootstrap.sh did not bootstrap libltdl sub-project. In build
environments without aclocal-1.16, Squid build failed.

Several solutions or workarounds have been tried or considered:

  • Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt
    to do that was reverted in commit bfd6b6a with "better to fix libtool
    installation" rationale. Another potential argument against this
    option is that packages should be bootstrapped by their distributors,
    not "users". We are not distributing libtool, but this is a gray area
    because we do distribute files that libtoolize creates. Finally,
    libtool itself does not provide a bootstrapping script and does not
    explicitly recommend bootstrapping in documentation.

  • "Fix libtool installation". We failed to find a good way to do that on
    MacOS (without building and installing newer libtool from sources).

  • Place m4 files where libtool v2.4 expects to find them. That change
    fixes MacOS builds that use automake v1.17, but breaks Gentoo builds
    because Gentoo libtool installs a buggy libltdl/Makefile.in that must
    be regenerated by automake before it can work. Fixing m4 files
    location prevents that regeneration.

We picked the first option despite its drawbacks because the third
option did not work on Gentoo, and asking Squid developers to install
libtool from sources (i.e. the second option) felt like a greater evil.

This old problem was exposed by recently introduced CI MacOS tests that
started to fail when MacOS brew updated automake package from v1.16
without the corresponding libtoolize package changes.

Also work around what seems to be a libtool packaging bug affecting
MacOS/brew environments, including GitHub Actions runners we use for CI:

libtool  (2.4.7) : glibtool
libtool path : /opt/homebrew/bin
Bootstrapping
glibtoolize:   error: creating 'libltdl/configure.ac' ... failed
glibtoolize:   error: creating 'libltdl/configure' ... failed
glibtoolize failed

That workaround will be removed after libtool package is fixed.

Also removed a single-iteration "for dir" loop with several stale hacks
from bootstrap.sh: With only two directories to bootstrap and with a
directory-specific mkdir command, source comments, and progress
messages, it is best to unroll that loop.

    gmake[3]: Entering directory .../libltdl
    .../cfgaux/missing: line 85: aclocal-1.16: command not found

During bootstrap.sh run, libtoolize copies prepackaged configure.ac and
Makefile.in files into our libltdl directory:

* libltdl/configure from libtool v2.4 has aclocal version set to 1.16
* libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal
  version to build aclocal.m4

Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs
aclocal-1.16 if "make" needs to build libltdl/aclocal.m4.

Our bootstrap.sh generates new ./configure but preserves copied
libltdl/configure with its aclocal version set to 1.16. In other words,
our bootstrap.sh does not bootstrap libltdl. If build environment lacks
aclocal-1.16, Squid build fails.

Several solutions or workarounds have been considered:

* Adjust bootstrap.sh to bootstrap libltdl. 2008 attempt to do that was
  reverted in commit bfd6b6a with "better to fix libtool installation"
  rationale. Another potential argument against this option is that
  packages should be bootstrapped by their distributors, not users. We
  are not distributing libtoolize; this is a gray area because we do
  distribute the files libtoolize copies.

* "Fix libtool installation". We failed to find a good way to do that on
  MacOS. One can build and install newer libtool from sources, but
  asking all Squid developers to do that if they face this problem feels
  like a less desirable option.

* Place m4 files where libtool v2.4 expects to find them (this change).

This old problem was exposed by recently introduced CI MacOS tests that
started to fail when MacOS brew updated automake package from v1.16
while keeping libtoolize package intact.
Also removed AC_CONFIG_MACRO_DIRS([m4]) suspected in cause that new
libtoolize failure.

Also added LT_CONFIG_LTDL_DIR(libltdl) because libtoolize started
from autoreconf tells us to do that.
... because I cannot figure out what causes the new bug. Did GitHub
update something again?!
... to avoid requiring Autoconf v2.70 instead of v2.61.
Copy link
Contributor Author

@rousskov rousskov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hope this PR fixes CI.

These comments do not request any PR changes.

Comment on lines +167 to +171
echo "brew libtool package details:"
brew info libtool --json | grep -E 'rebuild|tap_git_head'
# This hack was tested on libtoolize package with the following output:
# "rebuild": 2,
# "tap_git_head": "5cede8ea3b7b12c7f68215f75a951430b38d945f",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you do not want to see this noise in CI logs, I will remove it. I added it to make package changes easier to detect while this temporary sed workaround is in place.

# "tap_git_head": "5cede8ea3b7b12c7f68215f75a951430b38d945f",
#
editable=/opt/homebrew/Cellar/libtool/2.4.7/bin/glibtoolize
sed -i.bak 's@ltdl_ac_aux_dir=""@ltdl_ac_aux_dir="../build-aux"@' $editable || true
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sooner or later, this hack will fail. I did not make it conditional on brew libtool package variation because doing so is difficult (for me) and the failure is very unlikely to produce false positives; CI tests will just start failing again instead. We will simply remove this hack when it is no longer needed.

#
editable=/opt/homebrew/Cellar/libtool/2.4.7/bin/glibtoolize
sed -i.bak 's@ltdl_ac_aux_dir=""@ltdl_ac_aux_dir="../build-aux"@' $editable || true
diff -u $editable.bak $editable || true
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sed-based edit restores the earlier/working variation of libtoolize script:

-    ltdl_ac_aux_dir=""
+    ltdl_ac_aux_dir="../build-aux"

The actual name of this directory is misleading in Squid context. Ignore it.

@rousskov rousskov added M-cleared-for-merge https://github.com/measurement-factory/anubis#pull-request-labels S-could-use-an-approval An approval may speed this PR merger (but is not required) labels Aug 2, 2024
yadij
yadij previously approved these changes Aug 3, 2024
configure.ac Outdated Show resolved Hide resolved
@yadij yadij added the S-waiting-for-author author action is expected (and usually required) label Aug 3, 2024
@rousskov rousskov added S-waiting-for-more-authors needs developer help and removed S-waiting-for-author author action is expected (and usually required) labels Aug 3, 2024
@rousskov
Copy link
Contributor Author

rousskov commented Aug 3, 2024

Status update: This PR cannot be merged until we find a way to address the following Jenkins build test failure.

14:56:29 make[4]: Entering directory '/srv/jenkins/workspace/5-pr-test/gentoo/clang/layer-00-default/btlayer-00-default/libltdl'
14:56:29 make[4]: *** No rule to make target 'config.guess', needed by 'distdir-am'.  

FWIW, config.guess should be in cfgaux/ directory (it is on two different lab machines). That directory should not be affected by PR changes to the m4/ directory location, but evidently is.

@rousskov
Copy link
Contributor Author

rousskov commented Aug 3, 2024

Jenkins, retest this please.

@kinkie
Copy link
Contributor

kinkie commented Aug 4, 2024

Can't reproduce this with my build script. I'll try again with something closer to the standard build system

@rousskov
Copy link
Contributor Author

rousskov commented Aug 4, 2024

I can reproduce gentoo problem using your docker image, but I did not have enough time to research why libltdl/Makefile.in there gets (the wrong location for config.guess) an extra config.guess dependency entry (with the wrong location) for am__DIST_COMMON target.

Reproducing using your script probably doesn't work because your script bootstraps libltdl, generating new libltdl/Makefile.in.

@rousskov
Copy link
Contributor Author

rousskov commented Aug 4, 2024

I now know why libltdl/Makefile.in on gentoo has an extra config.guess dependency entry (with the wrong location) for am__DIST_COMMON target, leading to failed "make dist" checks on that platform. That Makefile.in file is not the file installed by bootstrap.sh/libtoolize. Due to the m4 dependency bug (that this PR is working around), running "make" on Gentoo regenerates our libltdl/Makefile.in:

make[3]: Entering directory .../squid/libltdl'
... /bin/sh '...squid/cfgaux/missing' aclocal-1.16 -I m4
... /bin/sh .../squid/cfgaux/missing automake-1.16 --foreign

During that regeneration, the problematic config.guess dependency gets removed and subsequent "make dist" (using new/regenerated libltdl/Makefile) succeeds.

Thus, working around m4 dependency bug (i.e. applying this PR primary changes):

  • fixes GitHub Actions CI because regeneration code (that fails on Github MacOS runners that lack automake-1.16) is no longer executed but also
  • breaks Jenkins Gentoo CI because same regeneration code (that works on Gentoo that has automake-1.16) is no longer executed!

You may wonder why fixed MacOS CI does not subsequently fail like Gentoo does. After all, MacOS regeneration code does not run, and Gentoo fails because its regeneration code does not run. MacOS CI does not fail because its libltdl/Makefile.in (generated by libtoolize) is not broken -- it does not have config.guess dependency entry with the wrong path. Both OSes have the same version of libtool package and the same libltdl/Makefile.am, so the difference can probably be explained by how libtoolize was bootstrapped when it was packaged for those two OSes:

  • MacOS libtoolize installs Makefile.in "generated by automake 1.16.3"
  • Gentoo libtoolize installs Makefile.in "generated by automake 1.16.5"

Automake version difference itself may not bee important, but that or some other difference in libtool package bootstrapping environment leads to a buggy Makefile.in installed into our libltdl by libtoolize on Gentoo.

To address both MacOS and Gentoo problems, I am considering adjusting this PR to bootstrap libltdl (i.e. implementing the first solution itemized in the current PR description). That way, we will become responsible for correct libltdl bootstrapping outcome (something we can control) rather than being at the mercy of libtool packaging problems (outside of our control). Given the wide variety of those problems, bootstrapping may be the lesser evil.

@kinkie
Copy link
Contributor

kinkie commented Aug 4, 2024

To address both MacOS and Gentoo problems, I am considering adjusting this PR to bootstrap libltdl (i.e. implementing the first solution itemized in the current PR description). That way, we will become responsible for correct libltdl bootstrapping outcome

I support this proposal - unsurprisingly. I'm just sorry it took all this effort for troubleshooting

rousskov added a commit to measurement-factory/squid that referenced this pull request Aug 5, 2024
Rationale:
squid-cache#1877 (comment)

Also removed a single-iteration "for dir" loop with several stale hacks.
With only two directories to bootstrap, and with directory-specific
mkdir commands, source comments, and progress messages, it is best to
unroll that loop.
Rationale:
github.com/squid-cache/pull/1877#issuecomment-2267627355

Also removed a single-iteration "for dir" loop with several stale hacks.
With only two directories to bootstrap, and with directory-specific
mkdir commands, source comments, and progress messages, it is best to
unroll that loop.
... after we started bootstrapping libltdl/ sub-project
... because that bug bites even _before_ (i.e. while) top-level
bootstrapping creates libltdl sub-project that we can bootstrap.
@rousskov rousskov changed the title Fix build with libtool v2.4 and automake v1.17 combo Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build Aug 5, 2024
@rousskov rousskov removed S-waiting-for-more-authors needs developer help M-cleared-for-merge https://github.com/measurement-factory/anubis#pull-request-labels labels Aug 5, 2024
@rousskov
Copy link
Contributor Author

rousskov commented Aug 5, 2024

To address both MacOS and Gentoo problems, I am considering adjusting this PR to bootstrap libltdl (i.e. implementing the first solution itemized in the current PR description). That way, we will become responsible for correct libltdl bootstrapping outcome

I support this proposal

Done. I have adjusted PR title/description accordingly. Please re-review.

We may also be able to replace our custom bootstrap.sh auto* commands with two autoreconf calls, but, in my tests, autoreconf runs a very different sequence of auto* commands and generates slightly different files, so I think it is best to make that replacement in a dedicated PR that will focus on simplifying bootstrap.sh further (rather than on fixing build -- the focus of this PR).

@kinkie kinkie added S-waiting-for-author author action is expected (and usually required) and removed M-cleared-for-merge https://github.com/measurement-factory/anubis#pull-request-labels S-could-use-an-approval An approval may speed this PR merger (but is not required) labels Aug 5, 2024
@rousskov rousskov added S-waiting-for-reviewer ready for review: Set this when requesting a (re)review using GitHub PR Reviewers box and removed S-waiting-for-author author action is expected (and usually required) labels Aug 5, 2024
@rousskov rousskov requested a review from kinkie August 5, 2024 12:18
@squid-anubis squid-anubis added the M-abandoned-staging-checks https://github.com/measurement-factory/anubis#pull-request-labels label Aug 5, 2024
Copy link
Contributor

@kinkie kinkie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@kinkie kinkie added M-cleared-for-merge https://github.com/measurement-factory/anubis#pull-request-labels S-could-use-an-approval An approval may speed this PR merger (but is not required) and removed S-waiting-for-reviewer ready for review: Set this when requesting a (re)review using GitHub PR Reviewers box M-cleared-for-merge https://github.com/measurement-factory/anubis#pull-request-labels labels Aug 5, 2024
@rousskov rousskov removed the S-could-use-an-approval An approval may speed this PR merger (but is not required) label Aug 5, 2024
@kinkie kinkie added the backport-to-v6 maintainer has approved these changes for v6 backporting label Aug 5, 2024
squid-anubis pushed a commit that referenced this pull request Aug 5, 2024
    gmake[3]: Entering directory .../libltdl
    .../cfgaux/missing: line 85: aclocal-1.16: command not found
    gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127

During bootstrap.sh run, libtoolize copies prepackaged configure and
Makefile.in files into our libltdl directory:

* libltdl/configure from libtool v2.4 has aclocal version set to 1.16;
* libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal
  version to build aclocal.m4

Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs
aclocal-1.16 if "make" needs to build libltdl/aclocal.m4.

Normally, "make" does not need to build libltdl/aclocal.m4 because that
file was created by libtoolize. However, libtool v2.4 is packaged with
(generated by packaging folks) libltdl/Makefile.in that makes
libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory.
Squid does not have that ./m4 directory, so "make" attempts to
re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16.

Our bootstrap.sh generated new ./configure but preserved copied
libltdl/configure with its aclocal version set to 1.16. In other words,
our bootstrap.sh did not bootstrap libltdl sub-project. In build
environments without aclocal-1.16, Squid build failed.

Several solutions or workarounds have been tried or considered:

* Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt
  to do that was reverted in commit bfd6b6a with "better to fix libtool
  installation" rationale. Another potential argument against this
  option is that packages should be bootstrapped by their distributors,
  not "users". We are not distributing libtool, but this is a gray area
  because we do distribute files that libtoolize creates. Finally,
  libtool itself does not provide a bootstrapping script and does not
  explicitly recommend bootstrapping in documentation.

* "Fix libtool installation". We failed to find a good way to do that on
  MacOS (without building and installing newer libtool from sources).

* Place m4 files where libtool v2.4 expects to find them. That change
  fixes MacOS builds that use automake v1.17, but breaks Gentoo builds
  because Gentoo libtool installs a buggy libltdl/Makefile.in that must
  be regenerated by automake before it can work. Fixing m4 files
  location prevents that regeneration.

We picked the first option despite its drawbacks because the third
option did not work on Gentoo, and asking Squid developers to install
libtool from sources (i.e. the second option) felt like a greater evil.

This old problem was exposed by recently introduced CI MacOS tests that
started to fail when MacOS brew updated automake package from v1.16
without the corresponding libtoolize package changes.


Also work around what seems to be a libtool packaging bug affecting
MacOS/brew environments, including GitHub Actions runners we use for CI:

    libtool  (2.4.7) : glibtool
    libtool path : /opt/homebrew/bin
    Bootstrapping
    glibtoolize:   error: creating 'libltdl/configure.ac' ... failed
    glibtoolize:   error: creating 'libltdl/configure' ... failed
    glibtoolize failed

That workaround will be removed after libtool package is fixed.


Also removed a single-iteration "for dir" loop with several stale hacks
from bootstrap.sh: With only two directories to bootstrap and with a
directory-specific mkdir command, source comments, and progress
messages, it is best to unroll that loop.
@squid-anubis squid-anubis added M-waiting-staging-checks https://github.com/measurement-factory/anubis#pull-request-labels and removed M-abandoned-staging-checks https://github.com/measurement-factory/anubis#pull-request-labels labels Aug 5, 2024
@squid-anubis squid-anubis added M-merged https://github.com/measurement-factory/anubis#pull-request-labels and removed M-waiting-staging-checks https://github.com/measurement-factory/anubis#pull-request-labels M-cleared-for-merge https://github.com/measurement-factory/anubis#pull-request-labels labels Aug 5, 2024
kinkie pushed a commit to kinkie/squid that referenced this pull request Sep 15, 2024
…ache#1877)

    gmake[3]: Entering directory .../libltdl
    .../cfgaux/missing: line 85: aclocal-1.16: command not found
    gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127

During bootstrap.sh run, libtoolize copies prepackaged configure and
Makefile.in files into our libltdl directory:

* libltdl/configure from libtool v2.4 has aclocal version set to 1.16;
* libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal
  version to build aclocal.m4

Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs
aclocal-1.16 if "make" needs to build libltdl/aclocal.m4.

Normally, "make" does not need to build libltdl/aclocal.m4 because that
file was created by libtoolize. However, libtool v2.4 is packaged with
(generated by packaging folks) libltdl/Makefile.in that makes
libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory.
Squid does not have that ./m4 directory, so "make" attempts to
re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16.

Our bootstrap.sh generated new ./configure but preserved copied
libltdl/configure with its aclocal version set to 1.16. In other words,
our bootstrap.sh did not bootstrap libltdl sub-project. In build
environments without aclocal-1.16, Squid build failed.

Several solutions or workarounds have been tried or considered:

* Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt
  to do that was reverted in commit bfd6b6a with "better to fix libtool
  installation" rationale. Another potential argument against this
  option is that packages should be bootstrapped by their distributors,
  not "users". We are not distributing libtool, but this is a gray area
  because we do distribute files that libtoolize creates. Finally,
  libtool itself does not provide a bootstrapping script and does not
  explicitly recommend bootstrapping in documentation.

* "Fix libtool installation". We failed to find a good way to do that on
  MacOS (without building and installing newer libtool from sources).

* Place m4 files where libtool v2.4 expects to find them. That change
  fixes MacOS builds that use automake v1.17, but breaks Gentoo builds
  because Gentoo libtool installs a buggy libltdl/Makefile.in that must
  be regenerated by automake before it can work. Fixing m4 files
  location prevents that regeneration.

We picked the first option despite its drawbacks because the third
option did not work on Gentoo, and asking Squid developers to install
libtool from sources (i.e. the second option) felt like a greater evil.

This old problem was exposed by recently introduced CI MacOS tests that
started to fail when MacOS brew updated automake package from v1.16
without the corresponding libtoolize package changes.

Also work around what seems to be a libtool packaging bug affecting
MacOS/brew environments, including GitHub Actions runners we use for CI:

    libtool  (2.4.7) : glibtool
    libtool path : /opt/homebrew/bin
    Bootstrapping
    glibtoolize:   error: creating 'libltdl/configure.ac' ... failed
    glibtoolize:   error: creating 'libltdl/configure' ... failed
    glibtoolize failed

That workaround will be removed after libtool package is fixed.

Also removed a single-iteration "for dir" loop with several stale hacks
from bootstrap.sh: With only two directories to bootstrap and with a
directory-specific mkdir command, source comments, and progress
messages, it is best to unroll that loop.
squid-anubis pushed a commit that referenced this pull request Sep 30, 2024
Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build

    gmake[3]: Entering directory .../libltdl
    .../cfgaux/missing: line 85: aclocal-1.16: command not found
    gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4]
    Error 127

During bootstrap.sh run, libtoolize copies prepackaged
configure and Makefile.in files into our libltdl directory:

* libltdl/configure from libtool v2.4 has aclocal version set
  to 1.16;
* libltdl/Makefile.in from libtool v2.4 uses
  configure-set aclocal version to build aclocal.m4

Thus, libltdl/Makefile (generated from libltdl/Makefile.in
above) runs aclocal-1.16 if "make" needs to build
libltdl/aclocal.m4.

Normally, "make" does not need to build libltdl/aclocal.m4
because that file was created by libtoolize. However, libtool
v2.4 is packaged with (generated by packaging folks)
libltdl/Makefile.in that makes libltdl/aclocal.m4 target
dependent on files in libltld/../m4 directory. Squid does not
have that ./m4 directory, so "make" attempts to re-generate
libltdl/aclocal.m4. When it does, it uses aclocal-1.16.

Our bootstrap.sh generated new ./configure but preserved
copied libltdl/configure with its aclocal version set to
1.16. In other words, our bootstrap.sh did not bootstrap
libltdl sub-project. In build environments without
aclocal-1.16, Squid build failed.

Several solutions or workarounds have been tried or
considered:

* Adjust bootstrap.sh to bootstrap libltdl (this change).
  2008 attempt to do that was reverted in commit bfd6b6a with
  "better to fix libtool installation" rationale. Another
  potential argument against this option is that packages
  should be bootstrapped by their distributors, not "users". We
  are not distributing libtool, but this is a gray area because
  we do distribute files that libtoolize creates. Finally,
  libtool itself does not provide a bootstrapping script and
  does not explicitly recommend bootstrapping in documentation.

* "Fix libtool installation". We failed to find a good way to
  do that on MacOS (without building and installing newer
  libtool from sources).

* Place m4 files where libtool v2.4 expects to find them.
  That change fixes MacOS builds that use automake v1.17, but
  breaks Gentoo builds because Gentoo libtool installs a buggy
  libltdl/Makefile.in that must be regenerated by automake
  before it can work. Fixing m4 files location prevents that
  regeneration.

We picked the first option despite its drawbacks because the
third option did not work on Gentoo, and asking Squid
developers to install libtool from sources (i.e. the second
option) felt like a greater evil.

This old problem was exposed by recently introduced CI MacOS
tests that started to fail when MacOS brew updated automake
package from v1.16 without the corresponding libtoolize
package changes.

Also work around what seems to be a libtool packaging bug
affecting MacOS/brew environments, including GitHub Actions
runners we use for CI:

    libtool  (2.4.7) : glibtool libtool path :
    /opt/homebrew/bin Bootstrapping glibtoolize:   error:
    creating 'libltdl/configure.ac' ... failed glibtoolize:
    error: creating 'libltdl/configure' ... failed glibtoolize
    failed

That workaround will be removed after libtool package is
fixed.

Also removed a single-iteration "for dir" loop with several
stale hacks from bootstrap.sh: With only two directories to
bootstrap and with a directory-specific mkdir command, source
comments, and progress messages, it is best to unroll that
loop.

----
Backport of PR #1877
@kinkie kinkie removed the backport-to-v6 maintainer has approved these changes for v6 backporting label Oct 2, 2024
@kinkie
Copy link
Contributor

kinkie commented Oct 2, 2024

backported manually

kinkie pushed a commit to kinkie/squid that referenced this pull request Oct 12, 2024
…ache#1877)

    gmake[3]: Entering directory .../libltdl
    .../cfgaux/missing: line 85: aclocal-1.16: command not found
    gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127

During bootstrap.sh run, libtoolize copies prepackaged configure and
Makefile.in files into our libltdl directory:

* libltdl/configure from libtool v2.4 has aclocal version set to 1.16;
* libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal
  version to build aclocal.m4

Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs
aclocal-1.16 if "make" needs to build libltdl/aclocal.m4.

Normally, "make" does not need to build libltdl/aclocal.m4 because that
file was created by libtoolize. However, libtool v2.4 is packaged with
(generated by packaging folks) libltdl/Makefile.in that makes
libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory.
Squid does not have that ./m4 directory, so "make" attempts to
re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16.

Our bootstrap.sh generated new ./configure but preserved copied
libltdl/configure with its aclocal version set to 1.16. In other words,
our bootstrap.sh did not bootstrap libltdl sub-project. In build
environments without aclocal-1.16, Squid build failed.

Several solutions or workarounds have been tried or considered:

* Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt
  to do that was reverted in commit bfd6b6a with "better to fix libtool
  installation" rationale. Another potential argument against this
  option is that packages should be bootstrapped by their distributors,
  not "users". We are not distributing libtool, but this is a gray area
  because we do distribute files that libtoolize creates. Finally,
  libtool itself does not provide a bootstrapping script and does not
  explicitly recommend bootstrapping in documentation.

* "Fix libtool installation". We failed to find a good way to do that on
  MacOS (without building and installing newer libtool from sources).

* Place m4 files where libtool v2.4 expects to find them. That change
  fixes MacOS builds that use automake v1.17, but breaks Gentoo builds
  because Gentoo libtool installs a buggy libltdl/Makefile.in that must
  be regenerated by automake before it can work. Fixing m4 files
  location prevents that regeneration.

We picked the first option despite its drawbacks because the third
option did not work on Gentoo, and asking Squid developers to install
libtool from sources (i.e. the second option) felt like a greater evil.

This old problem was exposed by recently introduced CI MacOS tests that
started to fail when MacOS brew updated automake package from v1.16
without the corresponding libtoolize package changes.


Also work around what seems to be a libtool packaging bug affecting
MacOS/brew environments, including GitHub Actions runners we use for CI:

    libtool  (2.4.7) : glibtool
    libtool path : /opt/homebrew/bin
    Bootstrapping
    glibtoolize:   error: creating 'libltdl/configure.ac' ... failed
    glibtoolize:   error: creating 'libltdl/configure' ... failed
    glibtoolize failed

That workaround will be removed after libtool package is fixed.


Also removed a single-iteration "for dir" loop with several stale hacks
from bootstrap.sh: With only two directories to bootstrap and with a
directory-specific mkdir command, source comments, and progress
messages, it is best to unroll that loop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
M-merged https://github.com/measurement-factory/anubis#pull-request-labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants