Phase install_lib fails using Spack if system's Python is used

[ajaust]

i tried to install the bindings using Spack on a system where external packages were set up with spack external find. This command also found the pre-installed Python of the system. Now, building the package py-pyprecice fails with the following error message:

==> Installing py-pyprecice-2.2.0.1-z4ojdrz6aqadiqidezkkubiifvwjr4uk
==> No binary for py-pyprecice-2.2.0.1-z4ojdrz6aqadiqidezkkubiifvwjr4uk found: installing from source
==> Fetching https://github.com/precice/python-bindings/archive/v2.2.0.1.tar.gz
##################################################################################################################################### 100.0%#-=#=- #     #                                                                                                                             
==> No patches needed for py-pyprecice
==> py-pyprecice: Executing phase: 'install_lib'
==> Error: ProcessError: Command exited with status 1:
    '/usr/bin/python3.9' '-s' 'setup.py' '--no-user-cfg' 'install_lib'

1 error found in build log:
     63     8611 |   __pyx_v_self->thisptr->writeBlockScalarData(__pyx_t_4, __pyx_v_size, ((int const *)__pyx_v__vertex_ids->data), ((doubl
           e const *)__pyx_v__values->data));
     64          |   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     65    /home/alex/software/spack/lib/spack/env/gcc/g++ -shared -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -fno-semantic-interpos
           ition -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now build/temp.linux-x86_64-3.9/tmp/alex/spack-stage/spack-stage-py-pyprecic
           e-2.2.0.1-z4ojdrz6aqadiqidezkkubiifvwjr4uk/spack-src/cyprecice/cyprecice.o -L/usr/lib -o build/lib.linux-x86_64-3.9/cyprecice.cp
           ython-39-x86_64-linux-gnu.so -lprecice
     66    UPDATING build/lib.linux-x86_64-3.9/precice/_version.py
     67    set build/lib.linux-x86_64-3.9/precice/_version.py to 'v2.2.0.1'
     68    copying build/lib.linux-x86_64-3.9/cyprecice.cpython-39-x86_64-linux-gnu.so -> /usr/lib/python3.9/site-packages
  >> 69    error: could not create '/usr/lib/python3.9/site-packages/cyprecice.cpython-39-x86_64-linux-gnu.so': Permission denied

The install_lib phase creates the necessary _version.py file , but it also wants to copy files into the system path

   68    copying build/lib.linux-x86_64-3.9/cyprecice.cpython-39-x86_64-linux-gnu.so -> /usr/lib/python3.9/site-packages
>> 69    error: could not create '/usr/lib/python3.9/site-packages/cyprecice.cpython-39-x86_64-linux-gnu.so': Permission denied

which is not possible without super user rights. Thus, the phase fails.

Why does the build phase want to copy data over and is it necessary? The answer should be no as all py-pyprecice related files should end up in the corresponding Spack directory that is user-writable.

This might be related to the problem observed during the workshop which led to #84.

[BenjaminRodenberg]

Building with dependencies from external adds another possible root of failure that is difficult to test. I started developing a test for the problem you mentioned above in #93. However, it is not done yet and I can also not estimate how much more time it will take. Therefore, I will put it on hold for now. If anybody else wants to continue on this: I would appreciate and help.

[ajaust]

I ran into an issue that I think is related to this problem.

I used a Spack environment that:

• had FEniCS installed, but not preCICE.
• used a Python version installed by Spack.

Then I installed preCICE and the python bindings manually using this environment. That was needed for some tests. When running some Python code it would not pick up the changes I made to preCICE. When running C++ it was fine. It turned out that I would use another (already installed) preCICE version when running Python code.

I was digging a bit further and found the following: The Python-bindings in version 2.2.0.1 do not install into the given prefix of Spack, but instead install into $PATH_TO_SPACK_INSTALLS/python-3.8.8-HASH/site-packages. Therefore, the bindings will always be in the path when the corresponding Python is loaded/used. This can conflict with other installed Python bindings. I would also assume that it creates some issues when one has the bindings installed more than once.

Observations:

• Observed for Python 3.8.8 so far
• Only observer for newese bindings of version 2.2.0.1.
• $PATH_TO_SPACK_INSTALLS/py-pyprecice-2.2.0.1-HASH/lib/python3.8/site-packages/ is empty and $PATH_TO_SPACK_INSTALLS/python-3.8.8-HASH/site-packages is not empty. (bad)
• $PATH_TO_SPACK_INSTALLS/py-pyprecice-2.1.1.1-HASH/lib/python3.8/site-packages/ is not empty and $PATH_TO_SPACK_INSTALLS/python-3.8.8-HASH/site-packages is empty (good)

I think in the latest release something in the installation procedure changed that we need to roll back.

Summary:
The latest release of Python bindings gets installed into to root path of the Python installation. However, it should be installed in the directory prescribed by Spack and then Spack will create suitable symlinks. Otherwise one ends up with problems when installing the bindings several times, using the Python environment with another preCICE version or the systems' Python which would require root right for installing the package.

[BenjaminRodenberg]

This sounds like a nasty bug... Do you think it's possible to reproduce this behavior with a test? I would propose the following strategy:

• Try to set something up similar to Test spack build using external dependencies #93 to reproduce the error.
• Find out whether the error comes from something we introduced in one of the previous releases.

I cannot really imagine what we might have introduced recently that breaks your setup. The history of setup.py looks ok (but this does not say much). A good candidate is in my opinion #71, since this changed quite a lot of things.

[ajaust]

I have debugged this a bit and found the following:

• The main problem seems to be the install_lib phase. As far as I can tell it is only there to trigger the creation of a _version.py file in the build directory. I personally do not like that and would like to get rid of this stage. At the moment the stages and also the order is weird/wrong in the Spack package(install_lib, build_ext, install).
• The install phase with the check if self.version <= Version("2.1.1.1"): seems to have broken the install stage for versions newer than 2.1.1.1. At least it seemed like it did nothing on my system. Link to code https://github.com/spack/spack/blob/1ed44f20f255f63b9e75e739a8bbe6911f4e1664/var/spack/repos/builtin/packages/py-pyprecice/package.py#L61
• The section

  my_cmdclass = {
    'test': my_test,
    'build_ext': my_build_ext,
    'install': my_install}

  seems to mess something up for versioneer. The call

  python-bindings/setup.py

  Line 139 in 8c66cc3

  cmdclass=versioneer.get_cmdclass(my_cmdclass),

  will check the used command classes and do some versioneer magic depending on that. It seems that versioneer does not detect our own classes correctly. For me it seems some of the checks for build_ext were not run, for example.
• I can remove all stages of the Spack package except for install_lib and I will not get any error. This means our Spack package seems to be only installed by that phase. All other phases do nothing (useful)?! install_lib in turn will install the bindings into the Python installation. The phase install_lib calls build_py and build_ext internally.

Goals/Questions:

• Could we have reasonable installation stages (e.g. build_ext and install)?
• Could be a sole install_lib enough?
• What are common/reasonable stages for Python?
• Could we create a setting where the install stage is enough?
• That one sounds most reasonable to me at least?!

[ajaust]

Fixed by spack/spack#25077