add instructions when verdi import fails #2420

ltalirz · 2019-01-24T19:50:13Z

when verdi import fails because of an old/new export file version, add a hint on what to do
fix issue with pip 19.x
fix issue with prospector failing to install
fix nasty travis issue with build of pymatgen (built using incompatible numpy version provided by travis)

when verdi import fails because of an old/new export file version, add a hint on what to do

trying cython>=0.29 from here: numpy/numpy#12785

ltalirz · 2019-01-25T08:19:13Z

@giovannipizzi I've already fixed two issues that prevented release_v0.12.3 from building but now I'm stuck with this numpy/pymatgen issue

I'm a bit puzzled because, originally, both the numpy version and the pymatgen version were the same as in provenance_redesign (also the cython build requirement I only added afterwards, trying to fix the issue), so I don't understand how provenance_redesign can build successfully but this one doesn't.
Any ideas what could be causing this?

giovannipizzi · 2019-01-25T11:33:29Z

did you try pip<19 instead of pip 19 with --no-use-pep517? Maybe the way things are done in pip 19 are different

ltalirz · 2019-01-25T12:29:18Z

Was worth a try, but the problem remains

From looking at the code, it seems to me it should be either numpy, cython or libc
https://github.com/materialsproject/pymatgen/blob/master/pymatgen/util/coord_cython.pyx#L19-L24

Could probably be worked around by adding --no-binary to pip install but this would increase install times and we don't really want to do that...

Btw, here are the dependencies for pymatgen 2018.12.12

ltalirz · 2019-01-25T14:01:58Z

I'm unable to reproduce the problem locally in a conda environment with python 2.7

$ pip install --upgrade pip
$ pip install pymatgen==2018.12.12 numpy==1.15.4 scipy==1.1.0
$ python
Python 2.7.15 |Anaconda, Inc.| (default, Dec 14 2018, 13:10:39)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pymatgen.util import coord_cython
>>>

Here are the relevant installed wheels on macOS:

Here the same info from the travis log:

it does not print the wheel used for pymatgen - does this mean it is building it from source?
https://files.pythonhosted.org/packages/de/37/fe7db552f4507f379d81dcb78e58e05030a8941757b1f664517d581b5553/numpy-1.15.4-cp27-cp27mu-manylinux1_x86_64.whl

Perhaps, it is building pymatgen from source using a different numpy version, and then it doesn't work together with the one we are installing.
Will try adding numpy==1.15.4 to the build requirements.

giovannipizzi · 2019-01-25T14:10:44Z

Try to delete the travis cache, it is possible that it's using a cached version? From the travis GUI you should have a button to clear the cache for this specific PR/branch

ltalirz · 2019-01-25T14:22:30Z

Ok, just for the record: adding numpy==1.15.4 as build dependency did not solve the problem.
Now deleted the cache for the PR and restarted the job...

ltalirz · 2019-01-25T15:26:00Z

Clearing the cache also didn't help - out of ideas for the moment.

giovannipizzi · 2019-01-27T18:24:40Z

It says at some point:

  Found existing installation: numpy 1.13.3
    Uninstalling numpy-1.13.3:
      Successfully uninstalled numpy-1.13.3

and then in the same stage it compiles numpy 1.15 and pymatgen. So it is possible that it's using the old numpy to compile pymatgen? I would have thought, though, that the addition of the numpy version to the toml file as well would have fixed this... maybe the syntax is not correct and something is getting ignored? Or maybe you can try uninstalling numpy before doing the pip install of everything? (at least to see if this is the source of the problem).

ltalirz · 2019-01-30T13:26:15Z

From the extremely verbose output of pip with -vvv one can see that it is now using a cached wheel pymatgen-2018.12.12-cp27-cp27mu-linux_x86_64.whl from travis (i.e. pymatgen is not built from source)

  Using version 2018.12.12 (newest of versions: 2018.12.12)^M
  Using cached wheel link: file:///home/travis/.cache/pip/wheels/4e/8c/1e/41c73592a64ed7e1f7bf5e52f9009eb9987b753d0a128226b3/pymatgen-2018.12.12-cp27-cp27mu-linux_x86_64.whl^M
  Added pymatgen==2018.12.12 from file:///home/travis/.cache/pip/wheels/4e/8c/1e/41c73592a64ed7e1f7bf5e52f9009eb9987b753d0a128226b3/pymatgen-2018.12.12-cp27-cp27mu-linux_x86_64.whl (from aiida-core==0.12.2) to build tracker '/tmp/pip-req-tracker-t5thnv'^M
  Removed pymatgen==2018.12.12 from file:///home/travis/.cache/pip/wheels/4e/8c/1e/41c73592a64ed7e1f7bf5e52f9009eb9987b753d0a128226b3/pymatgen-2018.12.12-cp27-cp27mu-linux_x86_64.whl (from aiida-core==0.12.2) from build tracker '/tmp/pip-req-tracker-t5thnv'^M

However, I tried clearing the cache before and it did not help

giovannipizzi · 2019-01-30T13:32:38Z

Did you clear it for the correct branch/PR? Also, if the cache is missing, I don't know if it reuses the cache of the source branch (in your fork) and/or the destination branch, you might want to try cleaning up those as well and rebuilding with -vvv?

ltalirz · 2019-01-30T13:35:14Z

yeah, I was wondering that - looking back at the travis job I thought I restarted, it looks to me like it was still using the cache.
Restarting this job now: https://travis-ci.org/aiidateam/aiida_core/jobs/485255224 let's see...

Ok: clearing the cache worked now - it is now downloading from pypi:

  Using version 2018.12.12 (newest of versions: 2018.12.12)^M
  Created temporary directory: /tmp/pip-unpack-11bmzz^M
  Looking up "https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz" in the cache^M
  No cache entry available^M
  https://files.pythonhosted.org:443 "GET /packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz HTTP/1.1" 200 2026362^M
^[[?25l  Downloading https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz (2.0MB)^M
  Downloading from URL https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz#sha256=520cdf326f25dd62cb99590e081e32a0b38588740c71f4432c2d13713be64fc7 (from https://pypi.org/simple/pymatgen/)^M
  Ignoring unknown cache-control directive: immutable^M
  Updating cache with response from "https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz"^M
  Caching due to etag^M
^M
^[[?25h  Added pymatgen==2018.12.12 from https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz#sha256=520cdf326f25dd62cb99590e081e32a0b38588740c71f4432c2d13713be64fc7 (from aiida-core==0.12.2) to build tracker '/tmp/pip-req-tracker-Zt8hrk'^M
  Running setup.py (path:/tmp/pip-install-rKXN5b/pymatgen/setup.py) egg_info for package pymatgen^M
    Running command python setup.py egg_info^M
    running egg_info^M
    creating pip-egg-info/pymatgen.egg-info^M
    writing requirements to pip-egg-info/pymatgen.egg-info/requires.txt^M
    writing pip-egg-info/pymatgen.egg-info/PKG-INFO^M
    writing top-level names to pip-egg-info/pymatgen.egg-info/top_level.txt^M
    writing dependency_links to pip-egg-info/pymatgen.egg-info/dependency_links.txt^M
    writing entry points to pip-egg-info/pymatgen.egg-info/entry_points.txt^M
    writing manifest file 'pip-egg-info/pymatgen.egg-info/SOURCES.txt'^M
    reading manifest file 'pip-egg-info/pymatgen.egg-info/SOURCES.txt'^M
    reading manifest template 'MANIFEST.in'^M
    warning: no previously-included files matching 'tests/*.*' found under directory 'pymatgen'^M
    writing manifest file 'pip-egg-info/pymatgen.egg-info/SOURCES.txt'^M
  Source in /tmp/pip-install-rKXN5b/pymatgen has version 2018.12.12, which satisfies requirement pymatgen==2018.12.12 from https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz#sha256=520cdf326f25dd62cb99590e081e32a0b38588740c71f4432c2d13713be64fc7 (from aiida-core==0.12.2)^M
  Removed pymatgen==2018.12.12 from https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz#sha256=520cdf326f25dd62cb99590e081e32a0b38588740c71f4432c2d13713be64fc7 (from aiida-core==0.12.2) from build tracker '/tmp/pip-req-tracker-Zt8hrk'^M

However, all jobs fail because pip -vvv is too verbose for travis

The job exceeded the maximum log length, and has been terminated.

So let's deactivate it again...

ltalirz · 2019-01-30T14:00:38Z

It seems since the previous job was terminated too early, pymatgen was downloaded again now (this differs from the usual output before):

Collecting pymatgen==2018.12.12 (from aiida-core==0.12.2)
  Downloading https://files.pythonhosted.org/packages/d6/b9/f386300f232c8523f562e39bf07bae0d9c4e6379f423f71df7cd622ddae7/pymatgen-2018.12.12.tar.gz (2.0MB)
...
 Running setup.py bdist_wheel for pymatgen ... \|/-\||done
  Stored in directory: /home/travis/.cache/pip/wheels/4e/8c/1e/41c73592a64ed7e1f7bf5e52f9009eb9987b753d0a128226b3

I uncommented the uninstallation of numpy, so the upgrade from numpy 1.13 to 1.15 happens only at the very end (the numpy 1.15.4 specified in the pyproject.toml should mean, however, that this numpy is seen during build).

The problem persists - ~~but now that we know a wheel is being downloaded I believe it doesn't matter at all what version of numpy we have - it matters what they had when they compiled the binary for the wheel~~.
Ah, it seems like it is not a wheel...
Then I'll just try to put the recent numpy version there, clear the cache and try again.

coveralls · 2019-01-30T14:32:03Z

Coverage decreased (-0.002%) to 48.005% when pulling 5cbce6c on ltalirz:import-instructions into 3e96d16 on aiidateam:release_v0.12.3.

ltalirz · 2019-01-30T14:38:57Z

Looks like this indeed fixed the issue - yay!

To summarize:

travis is downloading a source package of pymatgen (file ending in .tar.gz), then runs python setup.py bdist_wheel and stores the wheel in the travis cache, which is reused from the next build onwards
by default, the travis python environment has numpy 1.13.3 installed. Building pymatgen in this environment causes the error

numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216

when running the aiida test suite with the correct numpy version 1.15.4 for pymatgen 2018.12.12
updating numpy to 1.15.4 before installing aiida (and making sure the cache is cleared) solves the problem

This remains a mystery to me:

for some reason, adding numpy==1.15.4 to the build-system in pyproject.toml does not have the same effect (I believe it should). I haven't been able to spot any typos.
I still don't understand how builds on PRs to provenance_redesign are working - I've checked the .travis.yml and could not identify at which point it might lead to a different environment/numpy version for python 2.7

sphuber · 2019-01-31T11:51:59Z

Regarding the question why builds are working for provenance_redesign, I think I encountered a similar/related issue at some point with builds failing, due to a mismatch between numpy and scipy at the time. I also tracked it down to a wrong version of numpy being built from the cache on Travis, so I added a --no-cache-dir flag which seemed to work. Note that it is still there now and is probably sub optimal and we should maybe remove it

ltalirz · 2019-01-31T14:04:13Z

Thanks @sphuber - that does indeed explain it.

Note that it is still there now and is probably sub optimal and we should maybe remove it

Hm... I guess --no-cache-dir will make builds longer. On the other hand, perhaps it is the only way to make the pyproject.toml work as intended on travis?

it seems that, for whatever reason, the issue described here has resurfaced: aiidateam#2420 (comment) It was thought that this could be prevented by adding the `--no-cache-dir` argument to pip here: aiidateam#2799 but apparently this fix was fortuituous?

@kjappelbaum

* docs: add note on increasing work_mem + restructure troubleshooting section a bit + incorporate suggestions by @kjappelbaum + add a vacuum Note on failing travis build it seems that, for whatever reason, the issue described here has resurfaced: #2420 (comment) It was thought that this could be prevented by adding the `--no-cache-dir` argument to pip here: #2799 but apparently this fix was fortuitous...

ltalirz · 2019-07-03T09:09:47Z

Just to add another chapter to the story - I had a documentation build fail because pymatgen was installing a 1.17.0rc1 version of numpy (no idea how pip got the idea that it should be allowed to install release candidates):
https://travis-ci.org/ltalirz/aiida_core/jobs/553536605#L998

Apparently, this version of numpy requires python >= 3.5, so the build failed there.

Note: I guess some of the issue comes from the fact that pymatgen specifies numpy>=1.14 in the setup_requires, which may make the build non-deterministic when new numpy versions come out (and I don't know whether there is any way of controlling from the "outside" which version of numpy is installed inside the build environment).

sphuber · 2019-07-03T09:26:39Z

I have the same problem now on all my branches on my fork. All my py2.7 tests are failing. The weird thing is that we explicitly specify numpy==1.16.1 in our setup.json but apparently this gets overridden by the requirement of pymatgen. I think this should be considered a bug in pip. The requirements numpy==1.16.1 and numpy>=1.14 are not in conflict and what's more, pip should not be installing 1.17.0rc1 to begin with, since it is a pre-release

sphuber · 2019-07-03T12:32:40Z

The problem seems to be that setup_requires is not actually handled by pip, but rather by easy_install so our requirements are ignored. I have looked around but so far cannot find anyway to force a limit on the setup_requires dependencies of a dependency. What a mes...

ltalirz added 10 commits January 24, 2019 20:49

add instructions when verdi import fails

eb7a5b5

when verdi import fails because of an old/new export file version, add a hint on what to do

add fastentrypoints to build requirements

6b5c8ff

try fixing pip issue

06e97ca

missing flag for regular install

c4d9724

update pre-commit to fix ruby issue

65e3728

discussed in aiidateam#2362

try bumping numpy version

ca81f9c

downgrade numpy because of 1.16 issue

636cd25

see numpy/numpy#12749

try upgrading pymatgen

6a38419

go back to 2018.12.12 for py2 support

262e418

trying cython version

5e8d411

trying cython>=0.29 from here: numpy/numpy#12785

revert back to pip 18.1

381e9d0

ltalirz added 2 commits January 25, 2019 13:31

try reverting other packages as well

e7d6630

try fixing scipy version

d7e2953

try adding numpy to build requirements

7a667b7

ltalirz added 3 commits January 27, 2019 22:17

try uninstalling numpy

5fa4594

add --yes

7df750d

travis: add pip freeze + verbose flag

3705e5e

tone down verbosity for travis

37c83f1

new numpy version

88d9126

ltalirz added 2 commits January 30, 2019 15:39

cleanup

bcf2b4c

add missing exception

5cbce6c

ltalirz requested a review from giovannipizzi January 30, 2019 14:56

giovannipizzi approved these changes Jan 30, 2019

View reviewed changes

ltalirz merged commit 040b34e into aiidateam:release_v0.12.3 Jan 30, 2019

ltalirz mentioned this pull request Jun 15, 2019

docs: add note on increasing work_mem #2952

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add instructions when verdi import fails #2420

add instructions when verdi import fails #2420

ltalirz commented Jan 24, 2019 •

edited

Loading

ltalirz commented Jan 25, 2019 •

edited

Loading

giovannipizzi commented Jan 25, 2019

ltalirz commented Jan 25, 2019 •

edited

Loading

ltalirz commented Jan 25, 2019 •

edited

Loading

giovannipizzi commented Jan 25, 2019

ltalirz commented Jan 25, 2019

ltalirz commented Jan 25, 2019

giovannipizzi commented Jan 27, 2019

ltalirz commented Jan 30, 2019 •

edited

Loading

giovannipizzi commented Jan 30, 2019

ltalirz commented Jan 30, 2019 •

edited

Loading

ltalirz commented Jan 30, 2019 •

edited

Loading

coveralls commented Jan 30, 2019 •

edited

Loading

ltalirz commented Jan 30, 2019 •

edited

Loading

sphuber commented Jan 31, 2019

ltalirz commented Jan 31, 2019 •

edited

Loading

ltalirz commented Jul 3, 2019 •

edited

Loading

sphuber commented Jul 3, 2019

sphuber commented Jul 3, 2019

add instructions when verdi import fails #2420

add instructions when verdi import fails #2420

Conversation

ltalirz commented Jan 24, 2019 • edited Loading

ltalirz commented Jan 25, 2019 • edited Loading

giovannipizzi commented Jan 25, 2019

ltalirz commented Jan 25, 2019 • edited Loading

ltalirz commented Jan 25, 2019 • edited Loading

giovannipizzi commented Jan 25, 2019

ltalirz commented Jan 25, 2019

ltalirz commented Jan 25, 2019

giovannipizzi commented Jan 27, 2019

ltalirz commented Jan 30, 2019 • edited Loading

giovannipizzi commented Jan 30, 2019

ltalirz commented Jan 30, 2019 • edited Loading

ltalirz commented Jan 30, 2019 • edited Loading

coveralls commented Jan 30, 2019 • edited Loading

ltalirz commented Jan 30, 2019 • edited Loading

sphuber commented Jan 31, 2019

ltalirz commented Jan 31, 2019 • edited Loading

ltalirz commented Jul 3, 2019 • edited Loading

sphuber commented Jul 3, 2019

sphuber commented Jul 3, 2019

ltalirz commented Jan 24, 2019 •

edited

Loading

ltalirz commented Jan 25, 2019 •

edited

Loading

ltalirz commented Jan 25, 2019 •

edited

Loading

ltalirz commented Jan 25, 2019 •

edited

Loading

ltalirz commented Jan 30, 2019 •

edited

Loading

ltalirz commented Jan 30, 2019 •

edited

Loading

ltalirz commented Jan 30, 2019 •

edited

Loading

coveralls commented Jan 30, 2019 •

edited

Loading

ltalirz commented Jan 30, 2019 •

edited

Loading

ltalirz commented Jan 31, 2019 •

edited

Loading

ltalirz commented Jul 3, 2019 •

edited

Loading