Unable to import cellrank. Install Petsc Troubles

[mhulke]

I am working in python3 on Terra (current environment Python: python 3.7.9). I was able to successfully install cellrank with pip install cellrank. When I go to import cellrank, however, the notebook cell gets stuck on running and never successfully loads the package. I do not have this problem when importing any other packages. Restarting the kernel and the environment does not help. I tried installing cellrank[krylov], but always received the fail building wheel error for Petsc and the packages that follow. I followed the Dependencies instructions, but always receive the following error when trying to install Petsc:
RuntimeError: 256

ERROR: Command errored out with exit status 1: /usr/local/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4ne7q01k/petsc/setup.py'"'"'; file='"'"'/tmp/pip-install-4ne7q01k/petsc/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-j294rpcq/install-record.txt --single-version-externally-managed --home /tmp/pip-target-2x0__hm2 --compile --install-headers /tmp/pip-target-2x0__hm2/include/python/petsc Check the logs for full command output.

Any input on what might be causing the problem and how to successfully get cellrank running would be greatly appreciated.

[michalk8]

Hi @mhulke ,
did you also try installing with conda? That usually doesn't give PETSc/SLEPc issues.
If conda is not an option, could you please provide me here with the full log? The above doesn't really contain any info as why it had failed.

[mhulke]

I am not able to use conda. After additional environment resets, I now can no longer install mpi4py, though the Petsc install code does make a bit more progress than previously. I've attached files with the error output for cellrank when using 'pip install 'cellrank[krylov]'', ' pip install mpi4py', and 'pip install petsc'
cellrank_install_error.txt
mpi4py_install_error.txt
petsc_install_error.txt

[michalk8]

The cellrank and petsc logs are "fine", as building the petsc4py and slepc4py as pip packages is quite tricky, but at the end, it says Successfully installed petsc-3.14.1, so that should be fine.
As for mpi4py, it seems to me like it can't find some header files (perhaps there's no mpi system installed?). Here's an unresolved similar issue to yours: https://groups.google.com/g/mpi4py/c/8991ItatjG4
Maybe try what they suggest, i.e. running:

which mpicc

I also had a look at the CI of mpi4py, maybe you're just missing some dependencies or they are in some directories not in PATH.

I hope this helps a bit.

[mhulke]

It seems like not having a MPI system installed may be the problem. When I run 'which mpicc', I now get the output '/usr/bin/mpicc', which I didn't get before. After an environment restart, I was able to successfully install mpi4py. Unfortunately, I now get the following error when trying to install cellrank or cellrank[krylov]. I am also no longer able to successfully install petsc.
cellrank_install_error2.txt

[michalk8]

This seems related: pypa/pip#8438
You could try either (per above's suggestions):

1. pip install cellrank[krylov]==1.1.0 Assertion Error: assert not (home and prefix) pypa/pip#8438 (comment)
2. setting the target in cmdline pip install cellrank[krylov] --target site-packages Assertion Error: assert not (home and prefix) pypa/pip#8438 (comment)
3. unsetting the PIP_TARGET variable as unset PIP_TARGET Assertion Error: assert not (home and prefix) pypa/pip#8438 (comment)

[mhulke]

I am still getting the same error

[michalk8]

That's a really bummer, thought the above might solve it. Looking once again at the log, the issue happens with installing setuptools, setuptools_scm and wheel, since these are required to build the package.
I've tried the following command, works for me just fine:

pip install --target /tmp/ setuptools_scm

However, if I do export PIP_PREFIX=/tmp, it fails, similarly to yours. Try checking your env env | grep PIP and see if PIP_PREFIX is present and then unset it (I). I don't know why the prefix is in the log above, unless it comes from the environment.
Similarly, when I set export PIP_TARGET=/tmp and ran

pip install --prefix foo setuptools_scm

it also failed with:

ERROR: Exception:
Traceback (most recent call last):
  File "/home/michal/.miniconda3/envs/cr3.7/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/home/michal/.miniconda3/envs/cr3.7/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/home/michal/.miniconda3/envs/cr3.7/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 406, in run
    pycompile=options.compile,
  File "/home/michal/.miniconda3/envs/cr3.7/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 90, in install_given_reqs
    pycompile=pycompile,
  File "/home/michal/.miniconda3/envs/cr3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 785, in install
    prefix=prefix,
  File "/home/michal/.miniconda3/envs/cr3.7/lib/python3.7/site-packages/pip/_internal/locations.py", line 186, in get_scheme
    dist_name, user, home, root, isolated, prefix
  File "/home/michal/.miniconda3/envs/cr3.7/lib/python3.7/site-packages/pip/_internal/locations.py", line 118, in distutils_scheme
    assert not (home and prefix), "home={} prefix={}".format(home, prefix)
AssertionError: home=/tmp/pip-target-xmpxmfgr prefix=foo

[mhulke]

Ok, so maybe this is where the problem lies. I am able to successfully install setuptools, setuptools_scm, and wheel, BUT I get the following ouput (pip install setuptools --use-feature=2020-resolver --upgrade):

Collecting setuptools
Using cached setuptools-50.3.2-py3-none-any.whl (785 kB)
Installing collected packages: setuptools
ERROR: google-cloud-bigquery 1.23.1 requires google-resumable-media!=0.4.0,<0.6.0dev,>=0.3.1, but you'll have google-resumable-media 1.1.0 which is incompatible.
Successfully installed setuptools-50.3.2
Note: you may need to restart the kernel to use updated packages.

When I try to modify google-resumable-media, I get the following output (pip install 'google-resumable-media!=0.4.0,<0.6.0dev,>=0.3.1' --use-feature=2020-resolver --upgrade):

Collecting google-resumable-media!=0.4.0,<0.6.0dev,>=0.3.1
Using cached google_resumable_media-0.5.1-py2.py3-none-any.whl (38 kB)
Collecting six
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Installing collected packages: six, google-resumable-media
ERROR: google-cloud-storage 1.32.0 requires google-resumable-media<2.0dev,>=1.0.0, but you'll have google-resumable-media 0.5.1 which is incompatible.
Successfully installed google-resumable-media-0.5.1 six-1.15.0
Note: you may need to restart the kernel to use updated packages.

If I update google-resumable-media to >=1.0.0, I get the first message again. When I look in my packages folder, I have google_resumable_media-0.5.1.dist-info and google_resumable_media-1.1.0.dist-info. Could this be causing problems?

In response to your previous suggestions, env | grep PIP gives this:
PIP_USER=false
PIP_TARGET=/home/jupyter-user/notebooks/packages

Trying to set PIP_TARGET or unset PIP_TARGET says ERROR: No such key - PIP_TARGET.

I don't see PIP_PREFIX, and trying to unset it also throws ERROR: No such key - PIP_PREFIX

[michalk8]

The google related stuff shouldn't cause any problem, at least not in this case (it's not a dependency of setuptools or of cellrank).
Since you can't unset PIP_TARGET as unset PIP_TARGET, you can try following:

PIP_TARGET= pip install cellrank[krylov]

That should override the environment variable for just this command.

[mhulke]

I'm sorry to be such a challenge. I am still getting the same error.

Collecting cellrank[krylov]
Downloading cellrank-1.1.0.tar.gz (21.8 MB)
|████████████████████████████████| 21.8 MB 14.5 MB/s eta 0:00:01
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python3 /usr/local/lib/python3.7/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ha8unuwu/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools setuptools_scm wheel
cwd: None
Complete output (14 lines):
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.7/dist-packages/pip/main.py", line 23, in
from pip._internal.cli.main import main as _main # isort:skip # noqa
File "/usr/local/lib/python3.7/dist-packages/pip/_internal/cli/main.py", line 5, in
import locale
File "/usr/lib/python3.7/locale.py", line 16, in
import re
File "/usr/lib/python3.7/re.py", line 145, in
class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'

ERROR: Command errored out with exit status 1: /usr/local/bin/python3 /usr/local/lib/python3.7/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-ha8unuwu/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools setuptools_scm wheel Check the logs for full command output.

[michalk8]

No worries, I'm happy to help. enum34 is a backport for Enums from later versions and uninstalling it should solve this.
Here are the steps: iterative/dvc#1995 (comment)

[mhulke]

I tried to uninstall enum34, but keep being block due to permissions. It seems like I can't touch it. Another comment in the discussion you linked suggested I download enum34==1.1.8, but that didn't help. Is there any way to get around permission issues other than chmod, or silence the package in some way?

[michalk8]

You can try manually (re)moving the files that belong to this package, though I think the permissions will still be an issue.
If this doesn't work and since conda is not an option, I'd try creating a new virtual environment using virtualenv.

[Marius1311]

Hi @mhulke, any progress on this?

[mhulke]

Yes and no. I wasn't able to create a virtual environment in Terra, and couldn't remove the enum34 files manually. Ultimately got fed up and downloaded cellrank on a local server. Download was painless and everything seems to be working. Thank you for all the help with this. I'm sorry it was so painful fighting with my setup.

[Marius1311]

Great, thanks for the update! Closing this.