Dependency Handling Whiteboard

[textshell]

[Feel free to mercilessly edit this when parts are resolved or for any other reason]
[Partially based on irc conversations in the meson irc channel]

Reasons absolute paths in pkg-config files might break things.

There is interest in investigation if generating pc files that point to the actual files rather than doing -L/-l magic is viable.

• not sure the ecosystem is happy with full pathes in .pc files. This would be a very visible change in the installed files.
• It seems hard to support one file that allows both static and dynamic linking. This is possible now and should be still supported
• the whole point of pkg-config files is that they allow for a relocatable prefix.
  • this should be doable with transforming libs: -L${libdir} -lsomething to 'Libs: ${libdir}/libsomething.so'
• but then the file name gets hard-coded
  • there seem to be people that share pkg-config files across architectures/platforms depending on each one to resolve to different filename conventions. Also mixed MSVC and cygwin/mingw seems to rely on late resolving of file name conventions.
  • on Windows shared libraries might be libfoo.dll.a or foo.lib, and you won't know till you try to link
  • This is literally why gstreamer needed meson
• the build system might be the only place you can do this, because only you know what the libraries can be called, whether to force static or prefer shared, and whether to override the search path and look elsewhere first

Alternatives:

• Do nothing
• Change meson to resolve each -L -l pair to a filename with preference and then drop back to looking in all -L output by pkg-config, to get a meson resolved 'hacky micro syntax' alternative
• Push new features in pkg-config.

Static vs dynamic linking selection (user intend)

• We likely want to offer strict static linking to dependencies incl. secondary dependencies
  • Some parts of libc (-lm) or system libraries (user32.dll) might never be available for static linking.
    • Although some libcs allow fully static linking with reduced capability
• We likely want to offer strict static linking to dependencies excl. secondary dependencies
• Users might want to be able to set preference for static linking from the command line
• It was suggested to control static linking depending on the directories the libraries is resolved to (PkgConfigDependency: Add option to choose between shared or static #2816, Don't want platform libraries to be included when linking statically to a dependency #2765)
• The same dependencies can come into a target via different pathes, meson needs to resolve different static / dynamic preferences.

static linking specific cflags

Some platforms (i.e. windows) require different cflags when compiling code that will be statically linked against a library (i.e. no declspec dll import).
(pkgconf has Cflags.private for that, for pkg-config see https://bugs.freedesktop.org/show_bug.cgi?id=47996)

Resolving library pathes as early as possible

Meson wants to resolve libraries to file pathes as early as possible to avoid relying on the shared state in the linker commandline where search pathes from many different dependencies can mix in hard to predict ways.

• pkg-config --static seems to recursivly add .private variants. So meson would have a way to get more fine grained information than i think it currently gets.
• it seems we might need to run pkg-config without --static and then once again with --static
• pkg-config --libs-only-l is also recursive
• So the next possibility is to do pkg-config --print-requires, then run pkg-config --libs-only-l on each, and subtract from the original list
• i think a future solution should convert dynamic libs to full pathes too and drop -L from pkg-config as well. (PkgConfigDependency: Also try to eagerly resolve libraries path without static mode. #3414)

Secondary Dependencies

Dependencies themselves often have their own dependencies

Static linking

For static linking these dependencies need to be added as direct dependencies while linking.

• The needs telling pkg-config to also include "private" dependencies when linking statically.

Dynamic linking

At least gnu binutils' ld insists that dependencies of shared libraries (DT_NEEDED) need to be available and findable at link time. Practically this means all locations (containing directory) of secondary dependencies need to be either specified either as -rpath or -rpath-link

• Currently not handled at all

Pkg-config file generation

• The result of the pkg-config file generation module should always be usable with just dependency('name').
  • static only libraries now add dependencies in a way that does not need the consumer to know that only static linking is available ( Improve generation of pkg-config files for static only libraries. (closes #3074) #3093 )
• It should be possible to generate a .pc file that allows consumers to select between static and shared linking
  • currently only both_libraries is supported here.

Link argument deduplication

Real builds can get very slow by massive duplications of libraries in dependency chains. Some discussion here #2156
Recent changes should at least reduce this problem: #3392 #3394 #3395
Open issues: #2150 #3071

Dependencies for relinking on change.

• Meson handles target outputs well, even with optimization for shared libraries that don't change symbols
  • Does this need to be extended to more than symbol names. Like symbol sizes where the linker can warn on mismatch?
• External dependencies are harder.
  • Can be specified in various linker command line syntax formated options.
  • Lobby for linkers to learn to output dependencies in ninja compatible syntax (https://sourceware.org/bugzilla/show_bug.cgi?id=22843)
    • Would undo the symbol optimization if it includes libraries build by meson
  • Guess at files at configure time
    • merged in ninjabackend: Try to guess library dependencies for linker invocation. #3094 (ninja backend)

Build rpath

Meson should ensure for all dependencies that targets are runnable from the build directory.

• works well for build dependencies
• Currently only works for external dependencies found via legacy find_library not pkg-config
• No code for external secondary dependencies.
• We need to investigate if/when --disable-new-dtags is needed. --enable-new-dtags/DT_RUNPATH is not really recursive (or maybe not if any library has DT_RUNPATH entries on it's own)

Windows std library variants

(from irc): """just realised that everything about "debug" is broken. There are at least two different things that it means on Windows. The first is debug information (pdb files) and the second is whether deps are built against the debug STL or not. We currently don't deal with the latter in any way. Qt ships release and debug deps and we need to know which one to use. I would imagine the only way to get that done is to somehow look at the value of b_vscrt and make it part of the dependency lookup. But the question then becomes what to do if the mode is /MTd or /MT?"""

[jpakkane]

The best solution would be to update pkg-config to provide a richer set of data. That will be probably be tricky. Is it even actively maintained?

[nirbheek]

Is it even actively maintained?

Kinda, but not really.

However, we can recommend pkgconf as a future replacement. It already has a Python API too that we can if it's available for greater speed.

I think it's fine if we support older pkg-config with fallback code that calls it multiple times to get all the info we need.

[xclaesse]

This is related too: #2816

[bredelings]

In addition to static vs shared, do we need to allow selection between other variants? I am thinking of boost here, which gives debug and non-debug variants different names on windows.

[bluetech]

There is interest in investigation if generating pc files that point to the actual files rather than doing -L/-l magic is viable.

I'll mention that include paths can also lead to trouble - here's an example where I got it wrong: xkbcommon/libxkbcommon@776cb52. Getting rid of shared include paths is probably harder than library paths. But maybe you have some ideas.

Alternatively, we can all switch to NixOS.

[textshell]

just edited in:

static linking specific cflags

Some platforms (i.e. windows) require different cflags when compiling code that will be statically linked against a library (i.e. no declspec dll import).
(pkgconf has Cflags.private for that, for pkg-config see https://bugs.freedesktop.org/show_bug.cgi?id=47996)

[textshell]

• We need to investigate if/when --disable-new-dtags is needed. --enable-new-dtags/DT_RUNPATH is not really recursive (or maybe not if any library has DT_RUNPATH entries on it's own)

[textshell]

Might come in handy to know that pkg-config --variable=pc_path pkg-config can print the search path used by pkg-config in (most?) cases.

[jhgit]

Here's a case where absolute paths failed where -L would be better:

https://groups.google.com/forum/#!topic/mesonbuild/USAviy4NZSw

Basically, meson converted pkg-config's library spec to an absolute path, but the "library" is a linker script which points to an actual library file. But the path to the library file is not in the system ldconfig path. So since meson did not use -L, the actual library file is not found - thus linker error.