Fix a build problem with numpy 1.23

[lfarv]

Problem:

The recent release of numpy 1.23 (available for python 3.8 to 3.10) introduced a problem for users of python 3.8+ building from source. pip install runs in its own environment, and installs there the most recent numpy version, to compile the C-extensions of AT (so numpy 1.23 now). At run time, the code links with the release installed by the user in its environment, possibly an older one. It appears that numpy 1.23 changes something so that at fails to import:

RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xe

The only solution for the user is to upgrade numpy. Though usually this can be done without side effects, this should not be requested!

Proposed solution:

The solution proposed here is to select for building with each python version the oldest compatible numpy release. The version used at runtime will then always be more recent, ensuring full compatibility (except for documented deprecations). The following table is used:

Python version	numpy release
3.6	1.16.6
3.7	1.16.6
3.8	1.17.3
3.9	1.19.3
3.10	1.21.3

This the version used for building only (compilation of C extensions), configured in the [build-system] section of pyproject.toml. The requirement for running PyAT (install_requires in setup.cfg) stays as it is: numpy > 1.16.6, to get the necessary features. Practically, it will be the current version at the time of 1st installation, but the user may upgrade as he wants.

Possible side effects:

• python setup.py install does not run in a separated environment. Instead, it will downgrade numpy in the user's environment. So always use "pip" for installation !
• This may prevent the C extensions to use the most recent features of numpy. However the AT C code does not call any numpy function, it just needs access to the internal structure of a ndarray. It's very unlikely that any new feature may improve (accelerate) this significantly. So this is not relevant.

[swhite2401]

• python setup.py install does not run in a separated environment. Instead, it will downgrade numpy in the user's environment. So always use "pip" for installation !

@lfarv in #363 you showed that for the moment we cannot build with MPI using pip, does this mean that we will not be able to use the latest numpy, even worse that the environment that might be used for other applications will be affected?

In which case I would suggest that either #363 is solved (I do not see much problem in adding mpi4py to the requirements) or that another solution is found here.

[lfarv]

@swhite2401 : in this situation, you could still upgrade numpy again after the installation is done… But I agree, that's not nice. Anyway python setup.py install should soon stop working completely, so we'll need to find another solution.

For the moment, I think the problem you and @simoneliuzzo encountered recently, and that everybody running python 3.8+ will encounter, is much worse.

So I'm looking at solutions for building with MPI.

[lfarv]

@swhite2401 : here is a solution to build with MPI using pip:

MPI=1 pip install --no-build-isolation .

with the --no-build-isolation flag, pip will ignore the build specs from pyproject.toml and use the installed packages as they are. You just have to ensure yourself that numpy and setuptools_scm have been previously installed (which should be the case if you already ran pip install normally).

There is still a hope to find a cleaner solution, but does this allow to merge and solve this critical problem?

[swhite2401]

Yes ok for me.
Not necessarily very urgent, but it would be nice to document this and other build option such as OpenMP on the github pages.

[lfarv]

@swhite2401 : did you try running with MPI ? I just compiled and checked that at.DConstant.mpi is True, but I have no example to run…

[swhite2401]

Not since it was introduced a while ago...should I try again now?

[simoneliuzzo]

I haven't got sufficient python skills to approve or not this pull request

[MJGaughran]

How does this affect the building of wheels as part of our release process?

[lfarv]

@MJGaughran : if we were to produce a wheel now, the C integrators for python3.8+ would use the numpy 1.23 ABI, so would be incompatible with any older version. As mentioned above, this would force the users to upgrade their numpy to 1.23.

With this PR, wheels will be compiled versus the numpy version in the table above, that is the oldest numpy compatible with the python version. Consequently they will be compatible at run time with any newer numpy version.

We had no problem until now because apparently the numpy ABI was stable since long. Now we can profit from the fact that with the new packaging method (pyproject.toml), we can have different requirements for building (freeze a version) and running (allowing any version recent enough).

[MJGaughran]

Looking at #346 I can remember how it works now. The githubproject.toml file is used for configuration.

As the github workflow specifies Python 3.9, it builds using 1.16.6, which is what we need for ABI compatibility.