Libtool errors with recent libtool

[saraedum]

When building with libtool 2.4.7 we saw the following error at configure time:

config.status: executing libtool commands
+ make
    * new C compiler flags
    * new preprocessor flags
   C       src/ariths.c => obj/src/ariths.lo
libtool: Version mismatch error.  This is libtool 2.4.7, but the
libtool: definition of this LT_INIT comes from libtool 2.4.6.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.7
libtool: and run autoconf again.

For the conda-forge distribution, we fixed this with this patch, i.e., rm aclocal.m4 and use AC_CONFIG_MAROS_DIR to tell autoconf where it's macros can be found instead.

What is the reason for GAP's non-standard setup here?

[fingolfin]

It seems you are replacing the libtool bundled with GAP, which makes it then necessary to replace the bundled m4 files. To paraphrase your question, and with tongue in cheek: "Is there a reason for this non-standard treatment?"

Anyway, it seems to me that simple fix is to not just replace libtool and libtool.m4, but also ltoptions.m4, etc. -- or possibly to re-run ./autogen.sh to ensure all m4 files are updated. UPDATE: I misremembered, apologies: autogen.sh does not update any m4 files. So if you want a different libtool than the one bundled, you need to replace all those m4 files manually. END OF UPDATE

As to why we are not using AC_CONFIG_MACRO_DIRS: I am pretty sure this was to workaround problematic autotools behavior and/or bugs in it -- unfortunately I don't remember the details -- I really should have added this as a comment somewhere :-(. It might have been one of the various bugs that cause problems for anyone who tries to use autoconf + libtool without automake -- all kinds of things start to break because apparently that is a rare thing to do and not well-tested on the autotools side. Some of these bugs are fixed in current autotools versions, but unfortunately enough people are using system with e.g. autoconf 2.68 that I am reluctant to touch these things. If at all, I am trying to reduce our exposure to them (and thus e.g. get rid of libtool altogether).

In this particular case, perhaps the fact that aclocal, which is used to handled AC_CONFIG_MACRO_DIRS is part of automake, was part of the reason: I don't want to force people to install automake just for that. But again, I am speculating here.

[fingolfin]

OK, looking at the autoconf docs clarifies it. To quote:

Packages that use aclocal to generate aclocal.m4 should declare where local macros can be found using AC_CONFIG_MACRO_DIRS.

But we are not using aclocal to generate aclocal.m4 (as it is part of automake), hence we don't use AC_CONFIG_MACRO_DIRS.

Note that usage of a hand-written aclocal.m4 is explicitly covered in the autoconf manual.

[fingolfin]

OK, so I guess to force use of a different libtool version, you'd need to update all of these files:

• cnf/ltmain.sh
• cnf/m4/libtool.m4
• cnf/m4/ltoptions.m4
• cnf/m4/ltsugar.m4
• cnf/m4/ltversion.m4
• cnf/m4/lt~obsolete.m4

And to properly update these files, with AC_CONFIG_MACRO_DIRS, one would have to use autoreconf, which also requires automake, and has bugs in various versions of autotools, which lead to it "forgetting" to installs certain files (such as install-sh) if the package it is used on dares to not use automake....

Again, I kinda wish I had documented all the issues I've run into, but at the time the main focus was to get something working, not to figure out which versions and parts of autotools had precisely which bugs, etc. :-/.

But I am happy to report that within 10 minutes of trying out some things to jog my memory, I've run into several ones I don't recall seeing before:

• after adding AC_CONFIG_MACRO_DIRS([cnf/m4]) to configure.ac, running autoreconf -fiv ends up creating symlinks for install-sh, config.guess, config.sub, even though I explicitly asked it not do that :-(
• glibtoolize -icv updates cnf/ltmain.sh, cnf/m4/libtool.m4, cnf/m4/ltversion.m4 but for some reason not the other files that need updating
• glibtoolize -ficv updates all files but also downgrades cnf/config.guess and cnf/config.sub

Well, while I don't remember these, they might be part of the reason we don't run (g)libtoolize in autogen.sh.

[fingolfin]

After patching all files listed above, GAP builds just fine for me in GNU libtool 2.4.7.

But at the end of my investigation, I've got to come back to the question: Why? Looking at the NEWS for libtool 2.4.7, I found nothing that suggests upgrading to it is beneficial. Maybe you are aware of something that's not listed there or that I missed?

[saraedum]

Why am I using libtool 2.4.7? Just because it's installed here. And having it installed means that configure fails for me unfortunately.

[saraedum]

Oh, and yes, we are not using GAP's libtool. That's probably the issue.

If I had to guess, we do this for some cross-compilation/Windows issues. But I have no memory why we are doing this. Maybe @isuruf knows?

Maybe we just shouldn't be doing that in the first place. We'll keep doing that I think but in any case this issue can be closed then.

[saraedum]

Adding some more documentation to the build scripts to explain this would be great I think.
(Sorry for my somewhat garbled messages above. Your reactions reached me by email out of order and I had a hard time to follow your line of thought initially.)

[fingolfin]

I also have libtool 2.4.7 installed and GAP builds just fine anyway. I think it only fails for you because your build scripts choose to update two out of six files in GAP's libtool. Of course this breaks! That seems like a bug in your build script, which should either update all or none...

[fingolfin]

Adding some more documentation to the build scripts to explain this would be great I think.

I am in principle absolutely willing to improve the documentation, to make live easier for packagers like you (BTW thank you for that!)

But I am honestly at a loss what you'd like me to cover, i.e., what "this" refers to: "How to replace the libtool bundled with GAP by a different version"? But that seems really special and I am not sure this should be encouraged. At the very least, I'd like to know the usecase. Note that we build GAP 4.12.2 just fine for Windows (with Cygwin, of course, it doesn't support mingw or "native"). So if there is a problem, we'd like to know about it, and fix it on our side.

Or maybe something else.

(Sorry for my somewhat garbled messages above. Your reactions reached me by email out of order and I had a hard time to follow your line of thought initially.)

Same here :-). My last comment was based on what I read in your emails, not what I now see in the edited comments here. Either way I should have phrased it more nicely, sorry for that :-/.

Since we perhaps got a bit off on the wrong foot, I'd like to say that I am grateful that you are filing this as an issue so we can discuss it! Far too often, packagers apply all kinds of patches without talking to upstream even once, and this results in all kind of problems.

[saraedum]

But I am honestly at a loss what you'd like me to cover

I think just a line in configure.ac that you don't use AC_CONFIG_MACRO_DIRS but aclocal.m4 instead to load a vendored libtool because … would have been very helpful. I.e., explaining what's non-standard about the setup and why.

Far too often, packagers apply all kinds of patches without talking to upstream even once, and this results in all kind of problems.

Yes I fully agree. At the same time, authors of libraries tend to completely abuse build systems without talking to downstream (not talking about GAP here…)

[fingolfin]

I think just a line in configure.ac that you don't use AC_CONFIG_MACRO_DIRS but aclocal.m4 instead to load a vendored libtool because … would have been very helpful. I.e., explaining what's non-standard about the setup and why.

OK, that's a good suggestion, will do, thanks!

Yes I fully agree. At the same time, authors of libraries tend to completely abuse build systems without talking to downstream (not talking about GAP here…)

I've been on both sides of this often enough. In general some understanding for each other would go a long way... Library authors don't do this out of spite. But because they have other constraints to wrangle with. And for a long time, GNU autotools documentation was worse than useless in some regards... Even today, there are many things which are just not covered and which one has to figure out by looking at the implementation. Of course autotools are not unique in that regard, I've had similar experiences with cmake and others 😢 .

[fingolfin]

As an update, https://github.com/conda-forge/gap-feedstock has dropped the patch adding AC_CONFIG_MAROS_DIR and the code updating libtool, and all seems to work fine now :-).

But I am not closing this issue yet because I still want to add those docs we talked about.

[fingolfin]

Just out of completeness, I should also mention here that long-term goal is to just eliminate use of GNU libtool, see #5306

[fingolfin]

We have removed usage of GNU libtool completely, so hopefully with GAP 4.13.0 this will be an issue of the past. I also added a few comments to the build system in PR #5602