Compiling PyAT with MPI

[lfarv]

The last PyAT release uses the new python packaging method based on pyproject.toml. With this setup, the requirements for building the project (not installing) must be declared in pyproject.toml. So at the moment, at attempt for building with MPI using:

$ MPI=1 pip install .

fails because mpi4py is not declared as a build dependency in pyproject.toml. This even if mpi4py is already installed, because the build occurs in a virtual env. And mpi4py cannot be declared unconditionally, which would make it mandatory for all builds. The installation requirement can be declared in setup.cfg in the [options.extras_require] section, but not the build one…

So at the moment, the way to build with MPI is to use the old method:

$ MPI=1 python setup.py install

which now triggers a deprecation warning.
I do not know how the build requirements can be adjusted depending on an environment variable, or any other selection method…

[swhite2401]

@lfarv, I see that you have removed requirements.txt, why so? It is very convenient to have a file storing all dependencies required to run the code that you can use to build your venv with a single pip command

[lfarv]

In the new packaging system, the requirements are handled by pyproject.toml, so requirements.txt is now deprecated. Then pip install handles both build requirements when building from source, and install requirements (from setup.cfg) when building from a wheel file. So still a single command to install from scratch!

[swhite2401]

What about pip install -e . , I think it calling develop, will it work in this case?

[lfarv]

@swhite2401: yes, it works. It's the "editable" mode, you have to build either from the source distribution (called "sdist", a tar.gz file), or from a local directory.
See this.

[swhite2401]

btw I still think having the requirements file is useful as you can call it with pip -r, also many people use conda environments (default at ESRF) for which you need the requirements file -> conda install --file requirements.txt

Why not keep it? It doesn't arm no?

[lfarv]

@swhite2401: the problem with this new system is that it privileges a "declarative" approach, in config files like pyproject.toml and setup.cfg, trying to remove all "executable" files (*.py files) for safety reasons: avoid potentally malicious installation procedures. setup.py still exists because it's still necessary to build extensions (compiled C and C++) until something better is defined. But its functionality is more restricted.

[swhite2401]

Ok , but why remove the requirements.txt file? As I said I remains useful in some case and does no arm

[lfarv]

@swhite2401: Can't you use pip under Conda? I thought it was working, and it takes care of everything. Otherwise we should envisage a conda distribution for AT, but that's another story.

It's not recommended to keep requirement.txt because of the difficulty of handling double definitions…

[lfarv]

Anyway requirements.txt would not solve the MPI problem… It'a also declarative!

[swhite2401]

It was un-related to MPI, just a general question.
I have had bad experience with pip in conda, I ended up with conflicts between the conda packages and pip packages... but maybe I did wrong manipulations

[lfarv]

@swhite2401: I see you point, but I just tried to follow as much as I could the new packaging method, which is now mature, which is more flexible, allowing different build back-ends (setuptools is one of those, and I stick with it), and normally simpler, with only 2 commands:

• python -m build to build both source and binary distributions,
• pip install to install from either source or binary distribution

It's also clearly indicated that python setup.py install is deprecated… Hence this problem with MPI.

Concerning conflicts between packages, the best solution is usually to separate different projects in different virtual environments, but if you need everything together, it may be a problem.

[swhite2401]

Yes I understand, this was just for my culture, for now the solution is to add a .pth file in the conda environement. As you said we may explore what it would take for at to be installed with conda, very low priority though.

[swhite2401]

Hello @lfarv, any news on this front? We are starting to use MPI more extensively and this issue needs to be resolve.
Adding mpi4py to the build equirements is really excluded? Do you see any problem with that?

[lfarv]

Hi @swhite2401 : Yes, I am again working on the updated packaging method (removing setup.cfg and setup.py) which ends up in a single configuration file,pyproject.toml. There are still things I don't understand, but it's progressing. At least, there is the possibility to modify the build requirements at run time, using what is called "in-tree backend". In the mean time, you can still use

$ MPI=1  python setup.py install

Things are moving slowly with setuptools, and I'm away for 10 days!

[swhite2401]

Ah very good news! No rush and enjoy vacations! (I am also away for a couple weeks)

[swhite2401]

Solved in #483