Repair use of system BLAS broken by #29025: remove numpy site.cfg [DEFAULT/ALL] sections

[mkoeppe]

This ticket fixes what was broken by #29025.

Previous related tickets:

• make numpy and scipy use pkg-config to find blas/lapack #20157: make numpy and scipy use pkg-config to find blas/lapack
• Scipy fails to build with ATLAS #22047: Scipy fails to build with ATLAS
• numpy: site.cfg needs a [DEFAULT] section #29025: numpy: site.cfg needs a [DEFAULT] section

---

References:

scipy/setup.py at master · scipy/scipy
https://github.com/scipy/scipy/blob/master/scipy/setup.py

numpy/system_info.py at master · numpy/numpy
https://github.com/numpy/numpy/blob/master/numpy/distutils/system_info.py

python - How to check BLAS/LAPACK linkage in NumPy and SciPy? - Stack Overflow
https://stackoverflow.com/questions/9000164/how-to-check-blas-lapack-linkage-in-numpy-and-scipy

CC: @kiwifb @dimpase @mwageringel @vbraun

Component: packages: standard

Author: Dima Pasechnik

Branch/Commit: dba7aef

Reviewer: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/29051

[dimpase]

comment:2

I now see this on Arch on #29071 (with openblas from the system)

[dimpase]

comment:3

#29025 broke buidling scipy with system's OpenBLAS. I'm now looking into how to tweak
numpy/distutils/site.cfg to make it work, but it is not obvious. Probably there are more bugs there (in numpy).

[kiwifb]

comment:4

Something that weirded me a bit in #29025. Why keep the [ALL] section at all? I cannot be sure that's the root cause but I would start by cleaning that.

[dimpase]

comment:5

it's a bloody mess, if you read the numpy issue linked at #29025 - some numpy config functions use [ALL] section.

Removing [DEFAULT] section allows the build with system OpenBLAS to work again,
so there is clearly something amiss. I messed up while testing #29025 and so it went through.

[dimpase]

comment:6

Should we just revert #29025 ?
(at the moment it's not needed anywhere expect in not yet ready tickers)

I don't see an easy fix for it, without digging into numpy mess.

[mkoeppe]

comment:7

After reverting, is [ALL] needed?

[dimpase]

comment:8

No, [ALL] with just paths in it has no effect, according to numpy people.
So it should be removed.

What was the orginal need for [DEFAULT] in #29025 ?

[dimpase]

comment:9

[blas]
libraries  = openblas
[lapack]
libraries  = openblas

seems to work just as well.

[kiwifb]

comment:10

If we did the work properly with the like of CPATH and LIBRARY_PATH we should be able to drop [ALL]/[DEFAULT] completely. These section just define additional default paths for headers and libraries to be used in numpy.

[dimpase]

New commits:

9d1770a

revert #29025, remove useless [ALL] section

[dimpase]

Author: Dima Pasechnik

[dimpase]

Commit: 9d1770a

[dimpase]

Branch: public/packages/numpy/no_DEFAULT_and_ALL_numpy_site_cfg

[dimpase]

comment:12

Replying to @kiwifb:

If we did the work properly with the like of CPATH and LIBRARY_PATH we should be able to drop [ALL]/[DEFAULT] completely. These section just define additional default paths for headers and libraries to be used in numpy.

IMHO all these paths are better set up per library-specific section ([blas] etc.)
than via "globals" like LIBRARY_PATH.

[videlec]

Attachment: vdelecroix-config.log

[videlec]

comment:13

Attachment: vdelecroix-numpy-1.16.1.log

Thanks to Dima (and #29071 and the branch here), I got Sage compile up to numpy. It ends up with some linking trouble. See

• config.log
• numpy-1.16.1.log

[dimpase]

comment:14

Replying to @videlec:

Thanks to Dima (and #29071 and the branch here), I got Sage compile up to numpy. It ends up with some linking trouble. See

• config.log
• numpy-1.16.1.log

thanks, it's the same Arch-specific issue again - more precisely, numpy does not know about cblas. Testing on Debian wasn't enough.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

dba7aef

revert from "libraries=" back to section-specific "BLAH_libs="

[sagetrac-git]

Changed commit from 9d1770a to dba7aef

[dimpase]

comment:16

the commit at comment:15 fixes the latest problem.
I've also left a comment on this here: numpy/numpy#15338 (comment)

[mkoeppe]

comment:17

Testing at https://github.com/mkoeppe/sage/actions?query=workflow%3A%22Run+SAGE_ROOT%2Ftox.ini%22
with #29053

[videlec]

comment:18

At ​dba7aef, success for me!

[mkoeppe]

comment:19

Errors on various platforms tested at https://github.com/mkoeppe/sage/actions/runs/31716135. Examples:

• archlinux-latest-standard at https://github.com/mkoeppe/sage/runs/410329835:
  configure finds system openblas, then installing R from source fails with openblas-related errors

• ubuntu-bionic-minimal
  builds openblas from source, then installing R from source fails with openblas-related errors

• fedora-32-standard
  configure fails to detect openblas, error building openblas from source

[kiwifb]

comment:20

Replying to @mkoeppe:

Errors on various platforms tested at https://github.com/mkoeppe/sage/actions/runs/31716135. Examples:

• archlinux-latest-standard at https://github.com/mkoeppe/sage/runs/410329835:
  configure finds system openblas, then installing R from source fails with openblas-related errors

• ubuntu-bionic-minimal
  builds openblas from source, then installing R from source fails with openblas-related errors

• fedora-32-standard
  configure fails to detect openblas, error building openblas from source

I'll be honest. None of these failures can be a result of this ticket. May be the git branch needs rebasing to include some fix for those separate issues.

[mkoeppe]

Reviewer: Matthias Koeppe

[mkoeppe]

comment:21

Yes, this fix should definitely go into the next beta.

[mkoeppe]

comment:22

Follow-up for the other build issues related to BLAS at #29088.

[kiwifb]

comment:23

Replying to @mkoeppe:

Yes, this fix should definitely go into the next beta.

Note that Volker silently released 9.1.beta2 yesterday. Should ping him to make a proper announcement.

[jhpalmieri]

comment:24

I just downloaded the 9.1.beta2 tarball today, and on an Ubuntu virtual machine it fails on scipy, complaining about blas:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-7agkks4g/setup.py", line 492, in <module>
        setup_package()
      File "/tmp/pip-req-build-7agkks4g/setup.py", line 488, in setup_package
        setup(**metadata)
      File "/home/john/Sage/sage-9.1.beta2/local/lib/python3.7/site-packages/numpy/distutils/core.py", line 137, in setup
        config = configuration()
      File "/tmp/pip-req-build-7agkks4g/setup.py", line 395, in configuration
        raise NotFoundError(msg)
    numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

The main Sage config.log file says

configure:22549: result:     openblas-0.3.6.p0 will not be installed (configure check)

Am I doing something wrong?

[kiwifb]

comment:25

Replying to @jhpalmieri:

Am I doing something wrong?

No you are not. This ticket is supposed to deal with the stuff you just experienced. But this ticket is not included in 9.1.beta2, so it is effectively broken for external openblas.

[jhpalmieri]

comment:26

Replying to @kiwifb:

Replying to @jhpalmieri:

Am I doing something wrong?

No you are not. This ticket is supposed to deal with the stuff you just experienced. But this ticket is not included in 9.1.beta2, so it is effectively broken for external openblas.

I just realized that, too. When I merge this, acipy builds.

[mkoeppe]

comment:27

Follow-up = #29071.

[vbraun]

Changed branch from public/packages/numpy/no_DEFAULT_and_ALL_numpy_site_cfg to dba7aef