Add support for BLAS and LAPACK dependencies

[rgommers]

This is a start on implementing the BLAS and LAPACK support discussed in gh-2835.

This PR in its current form passes the added test case, and dependency('openblas') works for me now with SciPy in 3 cases beyond pkg-config:

• CMake detecting OpenBLAS
• Detecting OpenBLAS installed in the default environment lib/ and include/ dirs, but no pkg-config nor CMake installed
• Using a native or cross file to specify the include and library directories for OpenBLAS

This is already nice to have in its current form, however there's a lot more to do. This includes dealing with 32 vs. 64 bit interfaces, supporting CBLAS and (maybe) LAPACKE, and implementing support for other implementations (MKL, Netlib, ATLAS, etc.). The docs I added try to sketch this out - I'm very much not sure about the API to expose here, some feedback would be great.

This is also the first time I'm implementing a new dependency, so I may have taken a few wrong turns. If there's better ways of doing any of this, I'd love to hear them.

[codecov]

Codecov Report

Attention: 46 lines in your changes are missing coverage. Please review.

Comparison is base (5b317c5) 70.87% compared to head (484833b) 64.79%.
Report is 28 commits behind head on master.

❗ Current head 484833b differs from pull request most recent head f6ca5f0. Consider uploading reports for the commit f6ca5f0 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10921      +/-   ##
==========================================
- Coverage   70.87%   64.79%   -6.08%     
==========================================
  Files         218      416     +198     
  Lines       48356    90350   +41994     
  Branches    11485    20721    +9236     
==========================================
+ Hits        34271    58541   +24270     
- Misses      11568    27350   +15782     
- Partials     2517     4459    +1942     

Files	Coverage Δ
mesonbuild/dependencies/__init__.py	100.00% <100.00%> (ø)
mesonbuild/dependencies/blas_lapack.py	51.06% <51.06%> (ø)

... and 400 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lgtm-com]

This pull request introduces 2 alerts when merging 5c2bb31 into 7912901 - view on LGTM.com

new alerts:

• 2 for Unused import

[eli-schwartz]

Thanks for looking at this! Miscellaneous hodgepodge of review remarks that I found when taking a hurried glance (it's the holiday season right now):

[rgommers]

Thanks for the quick feedback @eli-schwartz. Enjoy your holiday season!

[rgommers]

The UNEXSKIP is because OpenBLAS isn't installed in CI yet. Before I mess with that, let me ask in which jobs you want me to add it. Maybe the same ones as hdf5 or scalapack, since they're similar scientific computing focused dependencies?

[rgommers]

As far as I could tell, adding keywords or methods/attributes to a dependency object is not allowed by design, and I should be using modules and .get_variable. modules is not a 100% natural fit, because not everything we're looking for is an optional/add-on component (e.g., the 32/64-bit interface is "give me this one" and symbol suffix is optional data about the library one is building against). However, it seems close enough.

From that follows the current draft API:

openblas_dep = dependency('openblas',
  version : '>=0.3.21',
  language: 'c',          # can be c/cpp/fortran
  modules: [
    'interface: ilp64',   # can be lp64 or ilp64 (or auto?)
    'symbol-suffix: 64_', # check/auto-detect? default to 64_ or no suffix?
    'cblas',
  ]
)
# Query properties as needed:
has_cblas = openblas.get_variable('cblas')
is_ilp64 = openblas_dep.get_variable('interface') == 'ilp64'
blas_symbol_suffix = openblas_dep.get_variable('symbol-suffix')

It'd be great to get some feedback on this.

I'll probably implement Netlib and MKL as well in this PR, because those are both quite different from OpenBLAS. So if the API works for all three, then that's a good sign.

[eli-schwartz]

As far as I could tell, adding keywords or methods/attributes to a dependency object is not allowed by design

I would say returned object methods definitely should be avoided, and really in nearly all cases .get_variable() should work or should be made to work.

Adding keywords to dependency() is more iffy, there are a couple cases where we have pretty boutique kwargs. Here's the full list of kwargs supported by dependency():

meson/mesonbuild/interpreter/interpreter.py

Lines 252 to 272 in b8e53ed

	permitted_dependency_kwargs = {
	'allow_fallback',
	'cmake_args',
	'cmake_module_path',
	'cmake_package_version',
	'components',
	'default_options',
	'fallback',
	'include_type',
	'language',
	'main',
	'method',
	'modules',
	'native',
	'not_found_message',
	'optional_modules',
	'private_headers',
	'required',
	'static',
	'version',
	}

Many of these are even documented. 🤣 The rest are carefully disclaimed in the docs with:

This function supports additional library-specific keyword arguments that may also be accepted

I think my personal opinion on the matter is I'd like it if we could avoid new kwargs, but if it would be painful or gross without them, then we might as well add them.

[rgommers]

Thanks for the feedback!

I think my personal opinion on the matter is I'd like it if we could avoid new kwargs, but if it would be painful or gross without them, then we might as well add them.

I think we can. If we compare, e.g., modules: ['interface: ilp64'] with interface: '64' the latter is a little nicer probably. But I'm happy enough with the former, and I also like that Meson enforces consistency in its API design.

I will continue down the current path then. Just a heads up that I won't have much time to work on this for the next 10 days or so. So I'll probably continue to enhance this PR after that, and then split off pieces that are ready into more digestible new PRs.

[tylerjereddy]

Will NumPy need 64-bit ILP64 OpenBLAS support for 3.12 wheels in the next release cycle? I think Sebastian was thinking they might in the meeting today? Not sure if that increases priority here.

[rgommers]

@tylerjereddy yes, but this one will be quick compared to dealing with the SIMD stuff. I've been too swamped with other topics, but I should revisit this in the near future. We'll probably miss the 1.25.0 release window and will have to do a 1.25.1/2 to complete the switchover, just like for SciPy (IIRC 1.9.3 was the first complete release without setup.py parts).

[rgommers]

Hi all, an update on the status of this: this is pretty much done, and actually already shipped in the Meson that is vendored in NumPy, in the numpy 1.26.1 release (last week, Oct 14th). I decided on doing it that way because this has turned out to be quite a complex feature - more so than I had anticipated - and I wanted to make sure to get a good amount of real-world feedback before proposing to merge this. So far we've had one bug report that needs addressing, regarding a Netlib BLAS install where libblas and libcblas are split libraries.

The docs in this PR contain the API that Meson will expose, most of which was already pre-discussed higher up and in gh-2835. A review of that API now would be quite useful though.

[dnicolodi]

@rgommers I haven't looked at the implementation, but the the documentation needs some finishing touches: there are several TBD notes, some TODOs, and some undefined things left.

[rgommers]

Sorry, I meant "please look at Dependencies.md for the API". I have some local changes to fix the one bug reported against NumPy as well as make MyPy happy - and that will address the couple of comments with TODO as well.

[drhpc]

Can you design this in a way that supports specifying which blas, cblas, lapack, lapacke to use via pkg-config module names? In pkgsrc, we rely on custom pkg-config module names to install differing builds of openblas into the same prefix. You have differing libray names and include directories. The CMake FindBLAS and FindLAPACK (missing FindCBLAS and FindLAPACKE yet, AFAIK) support that via BLA_PREFER_PKGCONFIG and BLA_PKGCONFIG_BLAS / BLA_PKGCONFIG_LAPACK.

I'd love the same for meson, for all components. Right now we're scratching heads over the scipy build which uses the vendored meson. When using the netlib implementation, which comes as 4 libraries with accompanying pkg-config file, it works for numpy buidl to tell it that blas=cblas. It finds libcblas.so, which brings libblas.so in and offers the full API. NumPy uses CBLAS, actually.

But SciPy seems to need BLAS API directly, specifically for its vendored superlu, where a build with blas=cblas will result in a _superlu.so without any BLAS library linkage (as -lcblas is silently dropped because of no used symbols, perhaps?).

I just noticed that blas=blas does fix the build, but only because the proposed meson code has a special case for BLAS being named blas and assumes a separate libcblas in that case. this guessing of the library name is dangerous, as you just never know all implementations out there.

Can we have separate choice for CBLAS and LAPACKE pkg-config module names? The application would have the differing netlib library names and most often all 4 settings being identical (openblas_openmp), which might be optimized a bit to clean up duplicated compiler/linker flags, but shouldn't cause trouble in any case. The guesswork to ease the life for users not specifying anything is nice to have for those users, but as packager, I need to be able to be specific and preferrably without diving into the detection logic of each build system as used by each package. I want numerous hours of my life back that I spent hunting down differing FindLAPACK.cmake copies/forks/idiosyncratic variants.

[rgommers]

Can you design this in a way that supports specifying which blas, cblas, lapack, lapacke to use via pkg-config module names?

Yes for the main part, as commented on in #2835 (comment). For separate CBLAS/LAPACKE, see below.

The CMake FindBLAS and FindLAPACK (missing FindCBLAS and FindLAPACKE yet, AFAIK) support that via BLA_PREFER_PKGCONFIG and BLA_PKGCONFIG_BLAS / BLA_PKGCONFIG_LAPACK.

It's already the first thing tried in Meson, so nothing else should be needed.

Right now we're scratching heads over the scipy build which uses the vendored meson.

I'll comment on this elsewhere (your numpy-discussion post probably) since it's SciPy-specific. Just to say here that SciPy, as of now, does not use the code in this PR or a vendored Meson (NumPy does). As a result, it's less flexible. This PR should solve SciPy's issues.

Can we have separate choice for CBLAS and LAPACKE pkg-config module names?

I'd much prefer not to unless it's absolutely necessary - and I've never come across this need before. There is only one known BLAS implementation AFAIK (namely Netlib) which ships CBLAS symbols as a separate library. And in all cases (including Netlib) a package needs BLAS and CBLAS pointing to the same implementation (cblas.so will depend on blas.so), so completely separating BLAS from CBLAS doesn't make sense to me.

If you have a case that is not covered here (it sounds like blas=blas did), can you please explain in detail what the libraries are and how they are built and named?

[drhpc]

It's already the first thing tried in Meson, so nothing else should be needed.

Could you, for the sake of compleness here, mention how meson does implement CMake's BLA_PKGCONFIG_BLAS choice if the -Dblas=myfunkyblas is a feature of the numpy use of meson, not meson itself (I confused that, as I am rather ignorant on generic meson features, just knowing that there are options defined in projects)? Or do you mean that meson will call CMake's FindBLAS anyway (without arguments that would specify my choice, though)? As I see it, if you want a dependency 'blas', meson will find a pkg-config blas.pc and stop there, despite me wanting openblas_openmp.pc.

Can I tell it, generically, to use myfunkyblas.pc when it encounters a desired dependency named 'blas'? I assumed the-Dblas=foo feature was supposed to be something generic, for any named dependency, to choose a certain variant. An ignorant assumption of an nonexisting generalisation?

[rgommers]

Could you, for the sake of compleness here, mention how meson does implement CMake's BLA_PKGCONFIG_BLAS choice if the -Dblas=myfunkyblas is a feature of the numpy use of meson, not meson itself

The description of that is: "If set, the pkg-config method will look for this module name instead of just blas." That is not difficult, however it does require something from the package author, because by design Meson avoids use of environment variables, and prefers CLI options. So the package author has to expose a -Dblas CLI option. If they do, and the basic implementation in the package's meson.build file is to use dependency(cli_blas_name), then using -Dblas=myfunkyblas works as expected.

Can I tell it, generically, to use myfunkyblas.pc when it encounters a desired dependency named 'blas'? I assumed the-Dblas=foo feature was supposed to be something generic, for any named dependency, to choose a certain variant. An ignorant assumption of an nonexisting generalisation?

You can't tell it generically to do that indeed. And that's by design, because of: https://mesonbuild.com/Contributing.html#environment-variables

[drhpc]

Thanks for clarifying. So what I imagined is some kind of default cli by meson along -Dmeson-rename-dep=foo=bar acting on a dependency named foo in meson.build.

[rgommers]

So what I imagined is some kind of default cli by meson along -Dmeson-rename-dep=foo=bar

There is no such thing. Looking at this again, the closest things to this are a few dependency-specific env vars that do exist (Boost has 2, CUDA 1: https://mesonbuild.com/Dependencies.html#boost), or CLI flags specific to a module like the python or pkgconfig one (you can see those if you run meson configure on a project).

I guess the thing is that in this PR I'm implementing several new dependencies for BLAS libraries plus a uniform API for treating things like asking for CBLAS or ILP64 - but not a generic "give me any BLAS lib you know about" (yet, at least). I'm not sure if that would be appropriate later on, because there's some things that are a bit opinionated in the NumPy/SciPy way of doing things. It's significantly more flexible than https://cmake.org/cmake/help/latest/module/FindBLAS.html (e.g., setting the search order, looking for a selected set of options, handling MKL threading options, etc.).

There are multiple options here I think:

1. Add "BLAS dependency options" in Meson similarly to "python module options"
2. Add a host of env vars, a la CMake
3. Take what NumPy/SciPy do and put it in a separate git repo that can be easily vendored by other projects, should they want the same full flexibility
4. Leave any CLI flags up to individual packages

Right now we're at (4). I am reasonably sure that (2) is a no-go, given Meson's preference for no env vars and the number of knobs that are possible (NumPy has 8 CLI flags now: https://github.com/numpy/numpy/blob/main/meson_options.txt). (3) or (1) could be done.

For now I'd like to finish this with (4), and then see if it makes sense to go to (3) to share code between NumPy and SciPy, since those are close in behavior.

[kloczek]

Just free thought .. why at all meson needs such extension if both librariesa are just libraries.
On first look all what is needed here is just add to blas and lapack pkgconfig interface to be able detect them 🤔
All details of exact library like is it 64 or not typu build is possible to put in pkconfig variables.
Nothing special is in those two libraries which could not be described in .pc files

IMO scipy and numpy should stick to regular meson + take care of provide .pc files by needed libraries (they are just libraries) instead maintaining own fork of meson. I know that to speedup some computations blas and lapack are used quite often as static libraries however pkgconfig has no any issues to describe static libraries interrace as well (Requires.provate:, Libs.private: and Cflags.private: are dedicated to to describe static library interface).

[dnicolodi]

It seems that the effort on upstreaming this has stalled. I have the impression that one of the issues is that the dependecy() interface is not the best suited to describe all the nuisances of the BLAS and LAPACK implementations desired and the preferred way to look it up. What about moving the look-up to a Meson module? It could work in a way similar to

blas = import('blas').find_dependency(...)

This has the advantage that the find_dependency() function can use the most proper arguments and is not limited to what makes sense for a generic dependency. If there is interest in seeing how this approach could work, I'll be happy to work on it.

[dnicolodi]

Another advantage is that this could easily expose command line options like the ones exposed by the python module.

[rgommers]

It seems that the effort on upstreaming this has stalled.

Yeah, that's mostly because I've been quite short on time, after I was happy with the maturity of the code. There are some problems though with integration, primarily that it is impossible to make Mypy happy with the way multiple inheritance is used in the current state of this PR. And I've been dreading a large refactor a bit.

I have the impression that one of the issues is that the dependecy() interface is not the best suited to describe all the nuisances of the BLAS and LAPACK implementations desired and the preferred way to look it up

Yes, the modules: interface is acceptable, but not really optimal. Another annoyance is the caching behavior of dependency(), so it's not possible to for example search for dependency('openblas', modules: ['interface: ilp64']) and then later try again with say 'interface: lp64'. The caching keys only on the dependency name, not on the content of modules IIRC. That hasn't been a problem in practice for NumPy yet, but it's undesirable behavior and I can see use cases for wanting to try the same dependency name multiple times.

What about moving the look-up to a Meson module?

Hmm, that's an interesting idea. It would make the API nicer, avoid the caching problem, and probably also make it easier to add an auto method that searches multiple BLAS libraries in a given order and with certain criteria. Because there's still quite a bit of code needed to do that nicely now: https://github.com/numpy/numpy/blob/88a6a0070d9d50f034c3f31f5e1d9ceea5f1065e/numpy/meson.build#L71-L155

@eli-schwartz WDYT?

[rgommers]

@kloczek your input really isn't helpful or to the point. pkg-config isn't always available (which is why there are SystemDependency's), nor can it do things like deal with symbol suffixes automatically or selecting something like an ILP64 build in a uniform/sane way across various BLAS libraries. This feature is needed and builds on a lot of real-world experience.

[eli-schwartz]

Yes, the modules: interface is acceptable, but not really optimal. Another annoyance is the caching behavior of dependency(), so it's not possible to for example search for dependency('openblas', modules: ['interface: ilp64']) and then later try again with say 'interface: lp64'. The caching keys only on the dependency name, not on the content of modules IIRC. That hasn't been a problem in practice for NumPy yet, but it's undesirable behavior and I can see use cases for wanting to try the same dependency name multiple times.

meson/mesonbuild/dependencies/detect.py

Lines 41 to 67 in b8cdd06

	def get_dep_identifier(name: str, kwargs: T.Dict[str, T.Any]) -> 'TV_DepID':
	identifier: 'TV_DepID' = (('name', name), )
	from ..interpreter import permitted_dependency_kwargs
	assert len(permitted_dependency_kwargs) == 19, \
	'Extra kwargs have been added to dependency(), please review if it makes sense to handle it here'
	for key, value in kwargs.items():
	# 'version' is irrelevant for caching; the caller must check version matches
	# 'native' is handled above with `for_machine`
	# 'required' is irrelevant for caching; the caller handles it separately
	# 'fallback' and 'allow_fallback' is not part of the cache because,
	# once a dependency has been found through a fallback, it should
	# be used for the rest of the Meson run.
	# 'default_options' is only used in fallback case
	# 'not_found_message' has no impact on the dependency lookup
	# 'include_type' is handled after the dependency lookup
	if key in {'version', 'native', 'required', 'fallback', 'allow_fallback', 'default_options',
	'not_found_message', 'include_type'}:
	continue
	# All keyword arguments are strings, ints, or lists (or lists of lists)
	if isinstance(value, list):
	for i in value:
	assert isinstance(i, str)
	value = tuple(frozenset(listify(value)))
	else:
	assert isinstance(value, (str, bool, int))
	identifier = (*identifier, (key, value),)
	return identifier

[eli-schwartz]

@kloczek

I offer you fair warning: your input is not going to be considered for this PR, so anything you say is just talking into the void.

[drhpc]

Regarding symbol suffixes for ILP64 … Reference-LAPACK/lapack#666 got marked as resolved now. I wonder if we are approaching a world where BLAS library symbols are standardized and we could just drop in a library (or two libraries that can be trivially linked together) that offers both 32 bit and 64 bit indices.

As a packager/admin, I just want to stress that whatever magic is implemented in the dependency code of the build system, I need some switch to override the magic and make it just follow a central decision to use THIS BLAS (equivalent to the undesired traditional environment variables like BLAS_LIBS). That's why I added the override to tell CMake to use a specific pkg-config module.

I must admit hat my head is at some distance from the meson/numpy/etc building business right now, but I'll have to work on refreshing a HPC cluster install soon and hope that things are at least not harder than last time. Many users do use pip, conda, or whatever, where this is not my problem, but I'd like to continue to offer central install with a predictable config.

[rgommers]

Regarding symbol suffixes for ILP64 … Reference-LAPACK/lapack#666 got marked as resolved now. I wonder if we are approaching a world where BLAS library symbols are standardized and we could just drop in a library (or two libraries that can be trivially linked together) that offers both 32 bit and 64 bit indices.

I'd love to see us all get there, but it won't happen any time soon I'm afraid. OpenBLAS will need a lot of work that isn't planned. There's some CMake-only support in Netlib BLAS/LAPACK now I believe, but I don't know if any distros are trying to build like that already.

As a packager/admin, I just want to stress that whatever magic is implemented in the dependency code of the build system, I need some switch to override the magic and make it just follow a central decision to use THIS BLAS

Definitely, that's a hard requirement. This PR doesn't even add a 'magic auto-detect' option (yet), and while NumPy and SciPy will have that, the current API for selecting an exact library won't change (e.g., -Dblas=blas -Dlapack=lapack, as documented in https://numpy.org/devdocs/building/blas_lapack.html#selecting-specific-blas-and-lapack-libraries).

[drhpc]

I'd love to see us all get there, but it won't happen any time soon I'm afraid. OpenBLAS will need a lot of work that isn't planned. There's some CMake-only support in Netlib BLAS/LAPACK now I believe, but I don't know if any distros are trying to build like that already.

Well, pkgsrc is, and perhaps others, as it's just the default on the CMake build. Excerpt from their CMakeLists.txt:

option(BUILD_INDEX64 "Build Index-64 API libraries" OFF)
if(BUILD_INDEX64)
  set(BLASLIB "blas64")
  set(CBLASLIB "cblas64")
  set(LAPACKLIB "lapack64")
  set(LAPACKELIB "lapacke64")
  set(TMGLIB "tmglib64")
  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DWeirdNEC -DLAPACK_ILP64 -DHAVE_LAPACK_CONFIG_H")
  set(FORTRAN_ILP TRUE)
else()
  set(BLASLIB "blas")
  set(CBLASLIB "cblas")
  set(LAPACKLIB "lapack")
  set(LAPACKELIB "lapacke")
  set(TMGLIB "tmglib")
endif()
# By default build standard API and extended _64 API
option(BUILD_INDEX64_EXT_API "Build Index-64 API as extended API with _64 suffix" ON)

This results in

$ nm -D /data/pkg/lib/libblas.so | grep dgemm
0000000000019be0 T dgemm_
000000000006b600 T dgemm_64_
$ nm -D /data/pkg/lib/libblas64.so | grep dgemm
0000000000019730 T dgemm_
00000000000698c0 T dgemm_64_

Both 32 bit and 64 bit build (BUILD_INDEX64) contain the added _64 (or 64_, depending on viewpoint) symbols. (I still patch for library naming, though.)

Definitely, that's a hard requirement. This PR doesn't even add a 'magic auto-detect' option (yet), and while NumPy and SciPy will have that, the current API for selecting an exact library won't change (e.g., -Dblas=blas -Dlapack=lapack, as documented in https://numpy.org/devdocs/building/blas_lapack.html#selecting-specific-blas-and-lapack-libraries).

My only gripe there is the implicit handling of cblas. It works now with our openblas builds including cblas (pkgsrc used to have separate builds of cblas on top of optimized blas) and only netlib having it as separate lib. But it's a funky special case that the 'magic' guesses that it will need to link to libcblas when I tell it that -Dblas=blas. Not sure what happens when I try our INDEX64 builds. For those I have that note:

# Check if the numpy/scipy ecosystem is ready for full ILP64 before
# allowing/setting BLAS_INDEX64=yes. Scipy needs both variants right now,
# wich does not work without standardized symbol suffix.

This might be moot of if we just start using the default build of netlib BLAS with CMake (and possibly with plain Makefile build, too, if someone ports that) now, as indicated above. Major packages making use of the 32 and 64 bit symbols as standardized is a strong case for OpenBLAS and others to add them to their builds. After all, it's not Rocket Science, it's just Computer Science (or Algebra, whatever seems more trivial to you;-)

The needed logic for Meson then just boils down again to just requiring a BLAS and presuming or late checking that it either provides 64 bit symbols (numpy etc. enabling big indices then) or not. Meson ocating a named pkg-config module is just fine, again. Then we can talk about me still building libcblas separately and if that should always be wrapped into one library with libblas, like it is for OpenBLAS, normally.

[rgommers]

Note to self: this should be fixed in 1.6.0, see https://mesonbuild.com/Release-notes-for-1-6-0.html#support-for-variable-in-system-dependencies.