Unexpected issue with package import, using cython and numpy

[bdice]

Details

• Read the Docs project URL: https://freud.readthedocs.io/
• Build URL (if applicable): https://readthedocs.org/projects/freud/builds/8431508/ and https://readthedocs.org/projects/freud/builds/8452348/
• Read the Docs username (if applicable): bdice

Expected Result

Build passes, documentation shows module info.

Actual Result

Build passes but doesn't actually contain module documentation. Builds should show as failing.

Expected result is on stable: https://freud.readthedocs.io/en/stable/box.html
Failing result is on latest: https://freud.readthedocs.io/en/latest/box.html

The package appears to build correctly, but we think there is something going wrong with Cython or NumPy and we can't get enough info to debug it ourselves. When RTD installs on top of our environment.yml file, it changes some package versions, specifically numpy from the openblas to mkl variants. I don't think that openblas/mkl is the problem, but changing numpy versions may very well invalidate something we rely on. We've seen two different error messages but it's not clear what causes either one.

This failure began appearing about simultaneously with our switch from Bitbucket (https://bitbucket.org/glotzer/freud.git) to GitHub (https://github.com/glotzerlab/freud.git) so we're not sure what to blame. However, I think I ruled out the repo URL change as a cause. I tried changing the URL back to Bitbucket and the issue can be reproduced. Also I created a fresh RTD project with the GitHub URL and it fails in the same way.

Most recent error (https://readthedocs.org/projects/freud/builds/8452348/):

/home/docs/checkouts/readthedocs.org/user_builds/freud/checkouts/awesome-readme/doc/source/box.rst:7: WARNING: failed to import freud.box.Box
/home/docs/checkouts/readthedocs.org/user_builds/freud/checkouts/awesome-readme/doc/source/box.rst:7: WARNING: failed to import freud.box.ParticleBuffer
WARNING: autodoc: failed to import module 'box' from module 'freud'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/awesome-readme/lib/python3.7/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/awesome-readme/lib/python3.7/site-packages/freud/__init__.py", line 5, in <module>
    from . import box
  File "freud/box.pyx", line 37, in init freud.box
AttributeError: type object 'freud.box.Box' has no attribute '__reduce_cython__'

Previous error from earlier in the month (https://readthedocs.org/projects/freud/builds/8431508/):

/home/docs/checkouts/readthedocs.org/user_builds/freud/checkouts/latest/doc/source/box.rst:7: WARNING: failed to import freud.box.Box
/home/docs/checkouts/readthedocs.org/user_builds/freud/checkouts/latest/doc/source/box.rst:7: WARNING: failed to import freud.box.ParticleBuffer
WARNING: autodoc: failed to import module 'box' from module 'freud'; the following exception was raised:
Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/freud/__init__.py", line 7, in <module>
    from . import common
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/freud/common.py", line 7, in <module>
    import numpy as np
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/numpy/core/__init__.py", line 59, in <module>
    from . import numeric
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/numpy/core/numeric.py", line 3093, in <module>
    from . import fromnumeric
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 17, in <module>
    from . import _methods
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/latest/lib/python3.6/site-packages/numpy/core/_methods.py", line 158, in <module>
    _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__
AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'

[bdice]

This issue may be related, it could be a problem with numpy 1.16: numpy/numpy#12736

I tried to build our maint branch, whose documentation successfully built when that tag (v0.11.4) was originally released, and this happens:

1. My environment.yml specifies conda package numpy=1.15.1=py36_blas_openblashd3ea46f_0
2. The command conda install --yes --quiet --name maint mock pillow sphinx sphinx_rtd_theme upgrades numpy to numpy=1.15.4-py36h1d66e8a_0
3. The command /home/docs/checkouts/readthedocs.org/user_builds/freud/conda/maint/bin/python -m pip install --ignore-installed --cache-dir /home/docs/checkouts/readthedocs.org/user_builds/freud/.cache/pip . notices that our package depends on numpy and installs numpy=1.16 via pip.

This seems to reproduce one sequence of events that the issue linked above describes as causing the bug.

This may resolve itself with a bugfix release of numpy, but I don't know what I can do as an RTD user to run my own commands that would prevent this problem. I can't forcibly uninstall packages, and I can't get pip to detect the conda-installed numpy as far as I know. Any suggestions?

[stsewd]

What I can see is conda installing a new version of python now (3.7 instead of 3.6), maybe some incompatible packages?

[stsewd]

Also, looks like the same problem as #3829

[bdice]

@stsewd Good idea, I'll try rebuilding a new environment.yml using Python 3.6.

[bdice]

I am still seeing these issues. As before, all of these builds report to ReadTheDocs as "passing." Here is what I've tried.

1. Made new environment.yml

Following @stsewd's idea, I built a new environment.yml file that uses Python 3.6 with the defaults conda channel, instead of conda-forge (except for nbsphinx, which I had to get from conda-forge). I created environment.yml using these commands on the Docker image continuumio/miniconda3:

conda create --yes -n freud-rtd python=3.6 cython tbb tbb-devel numpy scipy sphinx sphinx_rtd_theme mock pillow
conda activate freud-rtd
conda install --yes -c conda-forge nbsphinx
conda env export > environment.yml

• environment.yml
• numpy 1.15.4 (conda)
• cython 0.29.2 (conda)
• pip installs numpy 1.16 on top of the conda environment (I didn't realize this until later 🤦‍♂️)

Result: https://readthedocs.org/projects/freud/builds/8482593/

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/freud/__init__.py", line 5, in <module>
    from . import box
  File "freud/box.pyx", line 36, in init freud.box
AttributeError: type object 'freud.box.Box' has no attribute '__reduce_cython__'

2. Tried downgrading Cython to 0.28.5 (known to work previously)

• environment.yml
• numpy 1.15.4 (conda)
• cython 0.28.5 (conda)
• pip installs numpy 1.16 on top of the conda environment, as before

Result: https://readthedocs.org/projects/freud/builds/8482819/

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/freud/__init__.py", line 7, in <module>
    from . import common
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/freud/common.py", line 7, in <module>
    import numpy as np
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/numpy/core/__init__.py", line 59, in <module>
    from . import numeric
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/numpy/core/numeric.py", line 3093, in <module>
    from . import fromnumeric
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 17, in <module>
    from . import _methods
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/numpy/core/_methods.py", line 158, in <module>
    _NDARRAY_ARRAY_FUNCTION = mu.ndarray.__array_function__
AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'

This seems to be the bug from numpy 1.16 (pip) failing to coexist with numpy 1.15.4 (conda).

3. Forced setup.py to use 1.15.4

I edited setup.py to force it to use numpy 1.15.4.

• same environment.yml as the last attempt
• setup.py diff
• numpy 1.15.4 (conda)
• cython 0.28.5 (conda)
• pip installs numpy 1.15.4 on top of the conda environment

Result: https://readthedocs.org/projects/freud/builds/8483033/

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/freud/__init__.py", line 14, in <module>
    from . import order
  File "__init__.pxd", line 872, in init freud.order
ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216

4. Remove install_requires from setup.py

I removed numpy from the install_requires field in setup.py.

• same environment.yml as the last two attempts
• setup.py diff
• numpy 1.15.4 (conda)
• cython 0.28.5 (conda)
• pip does not install numpy on top of the conda environment

Result: https://readthedocs.org/projects/freud/builds/8483144/

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/sphinx/ext/autodoc/importer.py", line 154, in import_module
    __import__(modname)
  File "/home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/freud/__init__.py", line 14, in <module>
    from . import order
  File "__init__.pxd", line 872, in init freud.order
ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216

Seems like this is still using a conflicting numpy installation, but from where?

I have a requirements.txt file specifying numpy, scipy, six. I tried removing it from the ReadTheDocs config but nothing changed (https://readthedocs.org/projects/freud/builds/8483223/).

Currently, I have "Use system packages: Give the virtual environment access to the global site-packages dir." disabled in the Advanced Settings.

I will keep this issue updated if I am able to find a different error message or a working configuration.

Other ideas to try:

• install only tbb/tbb-devel through conda and everything else via pip

[bdice]

Continuing to test...

5. Minimal (tbb + cython) conda environment (conda-forge)

It might be possible to use only TBB and Cython from conda-forge and rely on pip to install numpy correctly. I made a new Python 3.6 environment with only tbb tbb-devel cython installed from conda-forge.

• environment.yml
• cython 0.29.4 (conda-forge)
• pip install . fails to find Cython?! I thought this part was working before, and I'm not sure what changed. 🤷‍♂️

Result: https://readthedocs.org/projects/freud/builds/8519670/

Processing /home/docs/checkouts/readthedocs.org/user_builds/freud/checkouts/fix-rtd
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  Complete output from command /home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/bin/python /home/docs/checkouts/readthedocs.org/user_builds/freud/conda/fix-rtd/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpipwqys9e:
  Traceback (most recent call last):
    File "setup.py", line 168, in <module>
      from Cython.Build import cythonize
  ModuleNotFoundError: No module named 'Cython'

... # and then our setup script catches this and raises a user-friendly exception

[bdice]

@stsewd I think I need to know more about how the actual ReadTheDocs image works to fix this problem. Using a Docker container (docker run -it readthedocs/build) and the following script adapted from the RTD build output, I am able to build my package successfully. I'm not sure what RTD does differently than this script. Can you give any insight?

Specifically, when does the conda environment get activated? It doesn't show that step in the build output, but surely it has to happen.

cd ~
git clone --no-single-branch --depth 50 https://github.com/glotzerlab/freud.git
cd freud
git checkout --force origin/fix-rtd
git clean -d -f -f
git submodule update --init
conda env create --name fix-rtd --file environment.yml
conda install --yes --name fix-rtd mock pillow sphinx sphinx_rtd_theme
source activate fix-rtd
python -m pip install -U recommonmark readthedocs-sphinx-ext
python -m pip install --ignore-installed .
cd ~  # so that we don't import the local, un-built package
python -c "import freud"  # succeeds

[stsewd]

We don't call activate, we just execute the commands with the python binary from conda.

https://github.com/rtfd/readthedocs.org/blob/a472cb939c65b8765ea3eaaa710fd26d22afdd45/readthedocs/doc_builder/python_environments.py#L386-L387

https://github.com/rtfd/readthedocs.org/blob/a472cb939c65b8765ea3eaaa710fd26d22afdd45/readthedocs/doc_builder/python_environments.py#L122-L132

Maybe we are losing something when skipping the activate step?

[bdice]

@stsewd I tried installing pip into the environment, but that didn't help. Previously, the environment.yml I provided worked fine and Cython was found from the conda installation.

The first failing build was Jan. 22: https://readthedocs.org/projects/freud/builds/8431508/
The last working build was Jan. 9: https://readthedocs.org/projects/freud/builds/8368585/

I see some commits (by you) over that time span that change the handling of pip, I wonder if that is related to the problem? The build outputs show that the change from pip to python -m pip coincides with the introduction of this failure.

50035a4#diff-31f8391d8643c47c6caedf53930c0b7e

[bdice]

I tried enabling the option to use setup.py, but it didn't seem to change the build output commands. That feature appears to be disabled for conda builds, but that isn't documented on the settings page...

I also tried creating a new doc/requirements-rtd.txt file that includes cython for ReadTheDocs, but this is unfortunately disabled for conda environments (also not documented on the settings page). https://github.com/rtfd/readthedocs.org/blob/50035a40adcb111b98581a58cc466363bfa8be00/readthedocs/doc_builder/python_environments.py#L418-L421

[stsewd]

We have this issue #2776, isn't clear for me why we don't install requirements in conda

[bdice]

How should we proceed with testing the pip vs python -m pip issue? Do I need to install RTD locally and test RTD before/after that change was made? I looked at the installation guide and could probably do this if you aren't able. We're holding on the 1.0 release of our package until docs can be fixed, so fixing this is pretty high on our team's priority list. Your help is sincerely appreciated, @stsewd!

[stsewd]

Sorry, wrong button, I'll try to replicate this locally in the next days. If it was working before, I have some guess

[bdice]

Awesome, @stsewd. I got pretty far with installing RTD locally. I got it to start running Docker containers to build the software but the containers fail when they get to conda commands for some reason. I have not had time to debug that yet. I look forward to your tests, hopefully we can identify the cause and get this resolved soon. Thanks again for your help.

[humitos]

I didn't read the whole thread. Why do you need to install conda dependencies from environment.yml and requirements.txt file?

If you want to install things with pip inside your conda env, you can use dependencies.pip in your YAML file: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually

[bdice]

@humitos Good idea, I hadn't considered using that approach in the conda environment. My current understanding of the problem(s) is that conflicting versions of numpy were being installed (>=1.16 and <1.16 can't coexist, it seems). This may have been due to a change in the way packages are installed and detected in the environment (RTD switched from pip to python -m pip). I have been experimenting with different ways to install packages, using pip or install_requires in setup.py instead of the conda environment, hoping to find a configuration that works as a short-term solution. But if this is due to the changes in RTD (which coincide with the problem's introduction, but we haven't yet definitively shown), it would be good to address that root cause instead of combining various environment/package management hacks.

[bdice]

@stsewd @humitos It seems that I've been able to solve the problem by circumventing pip entirely. I discovered that I was running v1 of the readthedocs.yml schema, so I upgraded that to v2. (I'm not sure if upgrading this was necessary.) I created a fresh conda environment with all requirements and installed the package with setuptools, and it seems to build. I may have been able to fix this by just using setuptools with config v1, but I'm not sure. I had tried using setuptools via the web interface earlier but I think I must not have realized that the web interface buttons didn't affect my build at all (the config file is the primary source of truth, in my understanding).

I think there is still an underlying bug with how pip was being used by RTD, since using pip instead of setuptools should still detect all the packages I installed (namely cython and the proper version of numpy). This used to work fine with pip until the recent changes mentioned above. I'll close this issue for now - thanks again to both of you for the helpful suggestions and support. 🎉🎉🎉

Working build: https://readthedocs.org/projects/freud/builds/8531307/

[humitos]

@bdice thanks for the extensive testing and the final update on this! Appreciated!