Support for "special" libraries

[dcbaker]

One that as a build system implementor I have to deal with often, is pkg-config files with dependencies on "special" libraries. Whether this is libdl or libm, which are part of libc on some implementations (or versions of implementations) or separate libraries on others; to things like OpenGL, which are part of the base OS in Windows and MacOS, or provide standard lookup mechanisms on Linux and Unicies; to systems like OpenMP which are part of the compiler implementation, but can still be useful to abstract as a dependency since they are not part of the language standard, but an additional feature that some compilers implement (some versions of), and there may be specific requirements for (such as, I require OpenMP of at least version 3.2 to function).

It would be nice to avoid having to do what is currently done in for pkg-config with these libaries (or flags) and just shove them into the cflags/libs lines and hope for the best.

I don't have a vision of how this would work, but maybe others have ideas?

[bretbrownjr]

My thoughts on this would be to stop the flag flinging and just expect m.cps, dl.cps, and so on.

If need be, we can design new "special" types of CPS libraries, but I expect in most cases, these are well modeled by existing CPS mechanisms.

Of course, the challenge, especially early in the adoption curve for CPS, is that there won't exist any /usr/lib/cps/m.cps file. Bill Hoffman talked about this challenge for early CMake adoption in the CPS talk at CppCon 2023. Here's a link at the approximate correct time that discusses how QT and CMake evolved side-by-side until QT just supported CMake natively. That sort of path is possible here as well. Some combination of CPS, build systems, and thoughtful packaging will have to make up the difference for a while, I expect, by providing cps files for libraries.

Importantly, we need CPS file discovery rules that can model "this is a best-effort CPS file until something official comes along" and "this is an official CPS file shipped with the library itself". We'll also need "this is a user-written CPS file to work around bad/missing CPS data in the packages I'm using", of course.

Anyway, if the goal of this issue is to start a conversation, those are my current thoughts. If you have other specifics you're looking for, @dcbaker, it'd be helpful to clarify what would resolve this issue.

[dcbaker]

I really wanted to start a conversation.

I had dismissed the m.cps plan out of hand because the main provider of these libs is glibc, who has, in the past, been dismissive of the idea of having pkg-config files, so I doubt they'll be much more receptive to having cps files. That leaves two options: 1. someone maintains the glibc cps files and relies on distros to shipe them, 2. they're somehow speced that the cps implementor deals with them. I don't know which is a better idea

[bretbrownjr]

There are additional complications like libstdc++ having dependencies like tbb that are only needed conditionally if you're using certain parts of the standard library (parallel algorithms).

My current philosophy is we just provide a lot of options for others to work out any "last mile" problems like this that pop up. I think we focus on accurately describing the system as configured. I expect if CPS is adopted widely enough, opportunities to converge will arise, especially:

• new best practices being developed and taught now that there's a model to actually consider
• when possible, tools being developed to point out bugs and nudge maintainers away from antipatterns
• ergonomic tools based on the data... for instance simple things like autocomplete and spell checking based on cps-query list-all when adding dependencies in a target_link_libraries in a CMakeLists.txt command from your favorite editor

[mwoehlke]

I'm not sure this is the "right" answer, but the question is very similar to how to deal with threading support, which (at present) is explicitly a compiler feature. The difficulty with this, of course, is that it doesn't scale well and probably isn't the right approach when the ultimate resolution is whether or not to link to a particular library (i.e. -lsomething). I think it may be necessary when the resolution has to do with compiler-specific flags; e.g. on Windows, threading may determine which runtime library is used, and on both Windows and Linux, threading support isn't just a -l flag.

IIRC there has even been discussion about representing language runtimes as components rather than their own special magic. Particularly in such a case, I'd be inclined to say that the solution to some of these questions is that users depend on something like stdc:dynamic_loader.

[dcbaker]

I would conceptually put threading support here as well. It's even more special since you have winpthreads.

[bretbrownjr]

There are also link-wide decisions that affect compile commands like the C++ standard version, the toolchain to use, etc.

I think an issue exploring these issues is justified. The reality may end up being a lot of messy options instead of a single right way to do things, at least initially.

As to this issue, the subject seems a little broad to me? When do we know if this issue is "done"? Maybe we can spell out specific use cases and track each with specific issues? The threading one is a good one.

[dcbaker]

It seems like at the moment we're discussing three issues:

1. dependencies that are actually compiler features (threading, OpenMP)
2. libraries that may be provided by the base OS (but might also come from an external project) (OpenGL)
3. libraries that are critical, but which upstream may not be willing to ship CPS files for (libm, libdl)

My original issue is 3

[bretbrownjr]

When you mean upstream, do you mean the package maintainers or the library maintainers? Either way, there will be a mix of approaches that should work, including writing CPS files on the side and setting CPS_PATH accordingly. I suspect having a repo ship tools to output fallback versions of those files makes sense as well. Like cps-gen-fallback, it pokes around and finds undocumented libs, and makes educated deductions about what might work. This is more or less how CMake deals with this issue. It has "Find" modules that take relatively low precedence and provide bridge support for important things like QT.

…

On Tue, Feb 27, 2024, 17:16 Dylan Baker ***@***.***> wrote: It seems like at the moment we're discussing three issues: 1. dependencies that are actually compiler features (threading, OpenMP) 2. libraries that may be provided by the base OS (OpenGL) 3. libraries that are critical, but which upstream may not be willing to ship CPS files for My original issue is 3 — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQFLYRR23RHD3DULRN477DYVZLNBAVCNFSM6AAAAABCTUZYN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRXG4ZTEMRXGA> . You are receiving this because you commented.Message ID: ***@***.***>

[dcbaker]

When you mean upstream, do you mean the package maintainers or the library maintainers?

Library maintainers. When package maintainers provide their own pkg-config files it ends up being a giant pain, because now instead of whatever the library maintainers what you to do (like use a lib-config script), there's now pkg-config files which are different depending on which package distributor you're getting them from. Meson runs into this all the time in the form of conan generating pkg-config files that don't work the same way the upstream does.

Either way, there will be a mix of approaches that should work, including writing CPS files on the side and setting CPS_PATH accordingly.

I'm have concerns about Linux distro engineering, where inserting CPS on the side is probably not acceptable. The expectation is that they puil in the dependency pancakes and everything Just Works™.

Eventually, when the CPS spec is relatively stable I will implement a CPS resolver inside of Meson itself, which will allow us to side-step some of these issues because I can plug in our custom dependency lookup mechanisms, but the near term solution will likely involve cps-config, which will not have such an option.

[bretbrownjr]

I'm fully expecting some amount of headaches like you describe. I expect CPS will need to be flexible enough to let people build software by any means necessary, especially to start out.

But I also expect the standards around metadata will help us declare intention in various ways that aren't really possible with pkg-config or CMake modules. We could directly model the concept of a best-effort CPS file, for instance. And we could ship validation tools that warn against common antipatterns like lists of bare -lsomelibs in linker options, pointing instead at recommended practices documents and such.

Keep in mind also that the plan is to have CMake, at least, generate CPS files for users. Ideally Meson could do likewise. After a little critical mass and some attractive features, I think we can get critical mass with reasonable effort. Or, at least, I think this is our best option for success. All the other ideas seem to be "everyone port to this specific packaging ecosystem" or "everyone make your own monorepos", both of which seem to have at least soft ceilings for market share.

[dcbaker]

Yeah, Meson will generate them in the future, we already to generate both pkg-config and cmake-config files.