-
Notifications
You must be signed in to change notification settings - Fork 528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build #1877
Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build #1877
Conversation
gmake[3]: Entering directory .../libltdl .../cfgaux/missing: line 85: aclocal-1.16: command not found During bootstrap.sh run, libtoolize copies prepackaged configure.ac and Makefile.in files into our libltdl directory: * libltdl/configure from libtool v2.4 has aclocal version set to 1.16 * libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal version to build aclocal.m4 Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs aclocal-1.16 if "make" needs to build libltdl/aclocal.m4. Our bootstrap.sh generates new ./configure but preserves copied libltdl/configure with its aclocal version set to 1.16. In other words, our bootstrap.sh does not bootstrap libltdl. If build environment lacks aclocal-1.16, Squid build fails. Several solutions or workarounds have been considered: * Adjust bootstrap.sh to bootstrap libltdl. 2008 attempt to do that was reverted in commit bfd6b6a with "better to fix libtool installation" rationale. Another potential argument against this option is that packages should be bootstrapped by their distributors, not users. We are not distributing libtoolize; this is a gray area because we do distribute the files libtoolize copies. * "Fix libtool installation". We failed to find a good way to do that on MacOS. One can build and install newer libtool from sources, but asking all Squid developers to do that if they face this problem feels like a less desirable option. * Place m4 files where libtool v2.4 expects to find them (this change). This old problem was exposed by recently introduced CI MacOS tests that started to fail when MacOS brew updated automake package from v1.16 while keeping libtoolize package intact.
Also removed AC_CONFIG_MACRO_DIRS([m4]) suspected in cause that new libtoolize failure. Also added LT_CONFIG_LTDL_DIR(libltdl) because libtoolize started from autoreconf tells us to do that.
... because I cannot figure out what causes the new bug. Did GitHub update something again?!
Cherry-picked branch commit 9225d1c.
... to avoid requiring Autoconf v2.70 instead of v2.61.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I hope this PR fixes CI.
These comments do not request any PR changes.
echo "brew libtool package details:" | ||
brew info libtool --json | grep -E 'rebuild|tap_git_head' | ||
# This hack was tested on libtoolize package with the following output: | ||
# "rebuild": 2, | ||
# "tap_git_head": "5cede8ea3b7b12c7f68215f75a951430b38d945f", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you do not want to see this noise in CI logs, I will remove it. I added it to make package changes easier to detect while this temporary sed
workaround is in place.
# "tap_git_head": "5cede8ea3b7b12c7f68215f75a951430b38d945f", | ||
# | ||
editable=/opt/homebrew/Cellar/libtool/2.4.7/bin/glibtoolize | ||
sed -i.bak 's@ltdl_ac_aux_dir=""@ltdl_ac_aux_dir="../build-aux"@' $editable || true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sooner or later, this hack will fail. I did not make it conditional on brew libtool package variation because doing so is difficult (for me) and the failure is very unlikely to produce false positives; CI tests will just start failing again instead. We will simply remove this hack when it is no longer needed.
# | ||
editable=/opt/homebrew/Cellar/libtool/2.4.7/bin/glibtoolize | ||
sed -i.bak 's@ltdl_ac_aux_dir=""@ltdl_ac_aux_dir="../build-aux"@' $editable || true | ||
diff -u $editable.bak $editable || true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This sed-based edit restores the earlier/working variation of libtoolize script:
- ltdl_ac_aux_dir=""
+ ltdl_ac_aux_dir="../build-aux"
The actual name of this directory is misleading in Squid context. Ignore it.
Status update: This PR cannot be merged until we find a way to address the following Jenkins build test failure.
FWIW, config.guess should be in cfgaux/ directory (it is on two different lab machines). That directory should not be affected by PR changes to the m4/ directory location, but evidently is. |
Jenkins, retest this please. |
Can't reproduce this with my build script. I'll try again with something closer to the standard build system |
I can reproduce gentoo problem using your docker image, but I did not have enough time to research why libltdl/Makefile.in there gets Reproducing using your script probably doesn't work because your script bootstraps libltdl, generating new libltdl/Makefile.in. |
I now know why libltdl/Makefile.in on gentoo has an extra config.guess dependency entry (with the wrong location) for am__DIST_COMMON target, leading to failed "make dist" checks on that platform. That Makefile.in file is not the file installed by bootstrap.sh/libtoolize. Due to the m4 dependency bug (that this PR is working around), running "make" on Gentoo regenerates our libltdl/Makefile.in:
During that regeneration, the problematic config.guess dependency gets removed and subsequent "make dist" (using new/regenerated libltdl/Makefile) succeeds. Thus, working around m4 dependency bug (i.e. applying this PR primary changes):
You may wonder why fixed MacOS CI does not subsequently fail like Gentoo does. After all, MacOS regeneration code does not run, and Gentoo fails because its regeneration code does not run. MacOS CI does not fail because its libltdl/Makefile.in (generated by libtoolize) is not broken -- it does not have config.guess dependency entry with the wrong path. Both OSes have the same version of libtool package and the same libltdl/Makefile.am, so the difference can probably be explained by how libtoolize was bootstrapped when it was packaged for those two OSes:
Automake version difference itself may not bee important, but that or some other difference in libtool package bootstrapping environment leads to a buggy Makefile.in installed into our libltdl by libtoolize on Gentoo. To address both MacOS and Gentoo problems, I am considering adjusting this PR to bootstrap libltdl (i.e. implementing the first solution itemized in the current PR description). That way, we will become responsible for correct libltdl bootstrapping outcome (something we can control) rather than being at the mercy of libtool packaging problems (outside of our control). Given the wide variety of those problems, bootstrapping may be the lesser evil. |
I support this proposal - unsurprisingly. I'm just sorry it took all this effort for troubleshooting |
Rationale: squid-cache#1877 (comment) Also removed a single-iteration "for dir" loop with several stale hacks. With only two directories to bootstrap, and with directory-specific mkdir commands, source comments, and progress messages, it is best to unroll that loop.
Rationale: github.com/squid-cache/pull/1877#issuecomment-2267627355 Also removed a single-iteration "for dir" loop with several stale hacks. With only two directories to bootstrap, and with directory-specific mkdir commands, source comments, and progress messages, it is best to unroll that loop.
... after we started bootstrapping libltdl/ sub-project
... because that bug bites even _before_ (i.e. while) top-level bootstrapping creates libltdl sub-project that we can bootstrap.
Done. I have adjusted PR title/description accordingly. Please re-review. We may also be able to replace our custom bootstrap.sh |
Co-authored-by: Francesco Chemolli <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks
gmake[3]: Entering directory .../libltdl .../cfgaux/missing: line 85: aclocal-1.16: command not found gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127 During bootstrap.sh run, libtoolize copies prepackaged configure and Makefile.in files into our libltdl directory: * libltdl/configure from libtool v2.4 has aclocal version set to 1.16; * libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal version to build aclocal.m4 Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs aclocal-1.16 if "make" needs to build libltdl/aclocal.m4. Normally, "make" does not need to build libltdl/aclocal.m4 because that file was created by libtoolize. However, libtool v2.4 is packaged with (generated by packaging folks) libltdl/Makefile.in that makes libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory. Squid does not have that ./m4 directory, so "make" attempts to re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16. Our bootstrap.sh generated new ./configure but preserved copied libltdl/configure with its aclocal version set to 1.16. In other words, our bootstrap.sh did not bootstrap libltdl sub-project. In build environments without aclocal-1.16, Squid build failed. Several solutions or workarounds have been tried or considered: * Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt to do that was reverted in commit bfd6b6a with "better to fix libtool installation" rationale. Another potential argument against this option is that packages should be bootstrapped by their distributors, not "users". We are not distributing libtool, but this is a gray area because we do distribute files that libtoolize creates. Finally, libtool itself does not provide a bootstrapping script and does not explicitly recommend bootstrapping in documentation. * "Fix libtool installation". We failed to find a good way to do that on MacOS (without building and installing newer libtool from sources). * Place m4 files where libtool v2.4 expects to find them. That change fixes MacOS builds that use automake v1.17, but breaks Gentoo builds because Gentoo libtool installs a buggy libltdl/Makefile.in that must be regenerated by automake before it can work. Fixing m4 files location prevents that regeneration. We picked the first option despite its drawbacks because the third option did not work on Gentoo, and asking Squid developers to install libtool from sources (i.e. the second option) felt like a greater evil. This old problem was exposed by recently introduced CI MacOS tests that started to fail when MacOS brew updated automake package from v1.16 without the corresponding libtoolize package changes. Also work around what seems to be a libtool packaging bug affecting MacOS/brew environments, including GitHub Actions runners we use for CI: libtool (2.4.7) : glibtool libtool path : /opt/homebrew/bin Bootstrapping glibtoolize: error: creating 'libltdl/configure.ac' ... failed glibtoolize: error: creating 'libltdl/configure' ... failed glibtoolize failed That workaround will be removed after libtool package is fixed. Also removed a single-iteration "for dir" loop with several stale hacks from bootstrap.sh: With only two directories to bootstrap and with a directory-specific mkdir command, source comments, and progress messages, it is best to unroll that loop.
…ache#1877) gmake[3]: Entering directory .../libltdl .../cfgaux/missing: line 85: aclocal-1.16: command not found gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127 During bootstrap.sh run, libtoolize copies prepackaged configure and Makefile.in files into our libltdl directory: * libltdl/configure from libtool v2.4 has aclocal version set to 1.16; * libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal version to build aclocal.m4 Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs aclocal-1.16 if "make" needs to build libltdl/aclocal.m4. Normally, "make" does not need to build libltdl/aclocal.m4 because that file was created by libtoolize. However, libtool v2.4 is packaged with (generated by packaging folks) libltdl/Makefile.in that makes libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory. Squid does not have that ./m4 directory, so "make" attempts to re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16. Our bootstrap.sh generated new ./configure but preserved copied libltdl/configure with its aclocal version set to 1.16. In other words, our bootstrap.sh did not bootstrap libltdl sub-project. In build environments without aclocal-1.16, Squid build failed. Several solutions or workarounds have been tried or considered: * Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt to do that was reverted in commit bfd6b6a with "better to fix libtool installation" rationale. Another potential argument against this option is that packages should be bootstrapped by their distributors, not "users". We are not distributing libtool, but this is a gray area because we do distribute files that libtoolize creates. Finally, libtool itself does not provide a bootstrapping script and does not explicitly recommend bootstrapping in documentation. * "Fix libtool installation". We failed to find a good way to do that on MacOS (without building and installing newer libtool from sources). * Place m4 files where libtool v2.4 expects to find them. That change fixes MacOS builds that use automake v1.17, but breaks Gentoo builds because Gentoo libtool installs a buggy libltdl/Makefile.in that must be regenerated by automake before it can work. Fixing m4 files location prevents that regeneration. We picked the first option despite its drawbacks because the third option did not work on Gentoo, and asking Squid developers to install libtool from sources (i.e. the second option) felt like a greater evil. This old problem was exposed by recently introduced CI MacOS tests that started to fail when MacOS brew updated automake package from v1.16 without the corresponding libtoolize package changes. Also work around what seems to be a libtool packaging bug affecting MacOS/brew environments, including GitHub Actions runners we use for CI: libtool (2.4.7) : glibtool libtool path : /opt/homebrew/bin Bootstrapping glibtoolize: error: creating 'libltdl/configure.ac' ... failed glibtoolize: error: creating 'libltdl/configure' ... failed glibtoolize failed That workaround will be removed after libtool package is fixed. Also removed a single-iteration "for dir" loop with several stale hacks from bootstrap.sh: With only two directories to bootstrap and with a directory-specific mkdir command, source comments, and progress messages, it is best to unroll that loop.
Bootstrap libltdl to fix libtool v2.4 + automake v1.17 build gmake[3]: Entering directory .../libltdl .../cfgaux/missing: line 85: aclocal-1.16: command not found gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127 During bootstrap.sh run, libtoolize copies prepackaged configure and Makefile.in files into our libltdl directory: * libltdl/configure from libtool v2.4 has aclocal version set to 1.16; * libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal version to build aclocal.m4 Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs aclocal-1.16 if "make" needs to build libltdl/aclocal.m4. Normally, "make" does not need to build libltdl/aclocal.m4 because that file was created by libtoolize. However, libtool v2.4 is packaged with (generated by packaging folks) libltdl/Makefile.in that makes libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory. Squid does not have that ./m4 directory, so "make" attempts to re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16. Our bootstrap.sh generated new ./configure but preserved copied libltdl/configure with its aclocal version set to 1.16. In other words, our bootstrap.sh did not bootstrap libltdl sub-project. In build environments without aclocal-1.16, Squid build failed. Several solutions or workarounds have been tried or considered: * Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt to do that was reverted in commit bfd6b6a with "better to fix libtool installation" rationale. Another potential argument against this option is that packages should be bootstrapped by their distributors, not "users". We are not distributing libtool, but this is a gray area because we do distribute files that libtoolize creates. Finally, libtool itself does not provide a bootstrapping script and does not explicitly recommend bootstrapping in documentation. * "Fix libtool installation". We failed to find a good way to do that on MacOS (without building and installing newer libtool from sources). * Place m4 files where libtool v2.4 expects to find them. That change fixes MacOS builds that use automake v1.17, but breaks Gentoo builds because Gentoo libtool installs a buggy libltdl/Makefile.in that must be regenerated by automake before it can work. Fixing m4 files location prevents that regeneration. We picked the first option despite its drawbacks because the third option did not work on Gentoo, and asking Squid developers to install libtool from sources (i.e. the second option) felt like a greater evil. This old problem was exposed by recently introduced CI MacOS tests that started to fail when MacOS brew updated automake package from v1.16 without the corresponding libtoolize package changes. Also work around what seems to be a libtool packaging bug affecting MacOS/brew environments, including GitHub Actions runners we use for CI: libtool (2.4.7) : glibtool libtool path : /opt/homebrew/bin Bootstrapping glibtoolize: error: creating 'libltdl/configure.ac' ... failed glibtoolize: error: creating 'libltdl/configure' ... failed glibtoolize failed That workaround will be removed after libtool package is fixed. Also removed a single-iteration "for dir" loop with several stale hacks from bootstrap.sh: With only two directories to bootstrap and with a directory-specific mkdir command, source comments, and progress messages, it is best to unroll that loop. ---- Backport of PR #1877
backported manually |
…ache#1877) gmake[3]: Entering directory .../libltdl .../cfgaux/missing: line 85: aclocal-1.16: command not found gmake[3]: *** [Makefile:561: .././../libltdl/aclocal.m4] Error 127 During bootstrap.sh run, libtoolize copies prepackaged configure and Makefile.in files into our libltdl directory: * libltdl/configure from libtool v2.4 has aclocal version set to 1.16; * libltdl/Makefile.in from libtool v2.4 uses configure-set aclocal version to build aclocal.m4 Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs aclocal-1.16 if "make" needs to build libltdl/aclocal.m4. Normally, "make" does not need to build libltdl/aclocal.m4 because that file was created by libtoolize. However, libtool v2.4 is packaged with (generated by packaging folks) libltdl/Makefile.in that makes libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory. Squid does not have that ./m4 directory, so "make" attempts to re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16. Our bootstrap.sh generated new ./configure but preserved copied libltdl/configure with its aclocal version set to 1.16. In other words, our bootstrap.sh did not bootstrap libltdl sub-project. In build environments without aclocal-1.16, Squid build failed. Several solutions or workarounds have been tried or considered: * Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt to do that was reverted in commit bfd6b6a with "better to fix libtool installation" rationale. Another potential argument against this option is that packages should be bootstrapped by their distributors, not "users". We are not distributing libtool, but this is a gray area because we do distribute files that libtoolize creates. Finally, libtool itself does not provide a bootstrapping script and does not explicitly recommend bootstrapping in documentation. * "Fix libtool installation". We failed to find a good way to do that on MacOS (without building and installing newer libtool from sources). * Place m4 files where libtool v2.4 expects to find them. That change fixes MacOS builds that use automake v1.17, but breaks Gentoo builds because Gentoo libtool installs a buggy libltdl/Makefile.in that must be regenerated by automake before it can work. Fixing m4 files location prevents that regeneration. We picked the first option despite its drawbacks because the third option did not work on Gentoo, and asking Squid developers to install libtool from sources (i.e. the second option) felt like a greater evil. This old problem was exposed by recently introduced CI MacOS tests that started to fail when MacOS brew updated automake package from v1.16 without the corresponding libtoolize package changes. Also work around what seems to be a libtool packaging bug affecting MacOS/brew environments, including GitHub Actions runners we use for CI: libtool (2.4.7) : glibtool libtool path : /opt/homebrew/bin Bootstrapping glibtoolize: error: creating 'libltdl/configure.ac' ... failed glibtoolize: error: creating 'libltdl/configure' ... failed glibtoolize failed That workaround will be removed after libtool package is fixed. Also removed a single-iteration "for dir" loop with several stale hacks from bootstrap.sh: With only two directories to bootstrap and with a directory-specific mkdir command, source comments, and progress messages, it is best to unroll that loop.
During bootstrap.sh run, libtoolize copies prepackaged configure and
Makefile.in files into our libltdl directory:
version to build aclocal.m4
Thus, libltdl/Makefile (generated from libltdl/Makefile.in above) runs
aclocal-1.16 if "make" needs to build libltdl/aclocal.m4.
Normally, "make" does not need to build libltdl/aclocal.m4 because that
file was created by libtoolize. However, libtool v2.4 is packaged with
(generated by packaging folks) libltdl/Makefile.in that makes
libltdl/aclocal.m4 target dependent on files in libltld/../m4 directory.
Squid does not have that ./m4 directory, so "make" attempts to
re-generate libltdl/aclocal.m4. When it does, it uses aclocal-1.16.
Our bootstrap.sh generated new ./configure but preserved copied
libltdl/configure with its aclocal version set to 1.16. In other words,
our bootstrap.sh did not bootstrap libltdl sub-project. In build
environments without aclocal-1.16, Squid build failed.
Several solutions or workarounds have been tried or considered:
Adjust bootstrap.sh to bootstrap libltdl (this change). 2008 attempt
to do that was reverted in commit bfd6b6a with "better to fix libtool
installation" rationale. Another potential argument against this
option is that packages should be bootstrapped by their distributors,
not "users". We are not distributing libtool, but this is a gray area
because we do distribute files that libtoolize creates. Finally,
libtool itself does not provide a bootstrapping script and does not
explicitly recommend bootstrapping in documentation.
"Fix libtool installation". We failed to find a good way to do that on
MacOS (without building and installing newer libtool from sources).
Place m4 files where libtool v2.4 expects to find them. That change
fixes MacOS builds that use automake v1.17, but breaks Gentoo builds
because Gentoo libtool installs a buggy libltdl/Makefile.in that must
be regenerated by automake before it can work. Fixing m4 files
location prevents that regeneration.
We picked the first option despite its drawbacks because the third
option did not work on Gentoo, and asking Squid developers to install
libtool from sources (i.e. the second option) felt like a greater evil.
This old problem was exposed by recently introduced CI MacOS tests that
started to fail when MacOS brew updated automake package from v1.16
without the corresponding libtoolize package changes.
Also work around what seems to be a libtool packaging bug affecting
MacOS/brew environments, including GitHub Actions runners we use for CI:
That workaround will be removed after libtool package is fixed.
Also removed a single-iteration "for dir" loop with several stale hacks
from bootstrap.sh: With only two directories to bootstrap and with a
directory-specific mkdir command, source comments, and progress
messages, it is best to unroll that loop.